[ᚬmaster] Rc/v0.15.0

[doitian]

Target Release: v0.15.0 (rylai v4)
Release Date: Jun 29, 2019

The code hash of default secp256k1 changed in this version, which is

0x94334bdda40b69bae067d84937aa6bbccf8acd0df6626d4b9ac70d4612a11933

Change log for secp256k1

• feat: use recoverable signature to reduce tx size by jjyr · Pull Request #15 · nervosnetwork/ckb-system-scripts

Features

• feat: initial windows support #937: Initial windows support (@xxuejie)

  This PR also updates CKB-VM with the following changes:

  • #64 windows support adjustments

• feat: add CI files for Windows environments #942: Add CI files for Windows environments (@xxuejie)

  This PR adds Windows CI configurations for both appveyor and Azure Pipeline, we can then decide which one to use(or if we want to keep both).

• feat: add a function to select all tx-hashes from storage for a block #931: Add a function to select all tx-hashes from storage for a block (@yangby-cryptape)

• feat: network alert #910: Network alert (@jjyr)

  This is a breaking change: b:p2p

  Introduce:
  https://en.bitcoin.it/wiki/Alert_system

  Purpose:
  Implement the alert system in CKB for urgent situation,
  In CKB early stage we may meet the same crisis bugs that Bitcoin meet,
  in urgent case, CKB core team can send an alert message across CKB P2P network,
  the client will show the alert message, the other behaviors of CKB node will not change.

  The Network Alert will be removed soon once the CKB network is considered mature.

  Implement:

  1. Add a new network protocol alt.
  2. Add a default disabled RPC send_alert.

• feat: simplify git describe info in version #967: Simplify git describe info in version (@doitian)

  The full git describe output easily cause confusion, this change will only
  show the 7 hex digests commit sha and the dirty flag, such as

    ckb 0.15.0-pre (rylai-v4 d7d3baca-dirty 2019-06-06)
  

• feat: explicitly specify bundled or file system #939: Explicitly specify bundled or file system (@doitian)

  This is a breaking change: b:cli

  Eliminate ambiguity and simplify code.

  BREAKING CHANGE:

  config ckb.toml and ckb-miner.toml

  - spec = "specs/dev.toml"
  + spec = { file = "specs/dev.toml" }

  chain spec

  [genesis.system_cells]
  files = [
  -   "specs/cells/secp256k1_blake160_sighash_all"
  +   { bundled = "specs/cells/secp256k1_blake160_sighash_all" }
  ]

• feat: Add load_code syscall #972: Add load_code syscall (@xxuejie)

  This newly added syscall can be used to load cell data into VM as
  executable memory. This can be used to dynamically load code into VM
  on demand.

• feat: Upgrade p2p #977: Upgrade p2p (@driftluo)

  1. upgrade p2p dependence
  2. support upnp optional

  changelog: https://github.com/nervosnetwork/p2p/blob/master/CHANGELOG.md

• feat: abstract data loader layer to decouple ckb-script and ckb-store #971: Abstract data loader layer to decouple ckb-script and ckb-store (@jjyr)

  Intention

  The needs to run ckb-script outside the ckb is wide, such as test contract scripts, verify tx in layer 2 program or in development tools. etc.

  This PR tries to decouple ckb-store dependency from ckb-script via abstract a data loader trait, make ckb-script easier to be used standalone.

  Changes

  1. abstract a trait DataLoader. TransactionScriptVerifier should only load data from this trait.
  2. remove the ckb-store dependency from ckb-script and dao-utils

• feat: use new identify protocol #978: Use new identify protocol (@driftluo)

  This is a breaking change: b:p2p

  BREAKING CHANGE: identify protocol break

  The current identify protocol does not play a role in identifying the capabilities of both parties, and the message structure is not reasonable.

  So, I rewrote it and added the capability ID and network ID.

  Moreover, I changed the process of network connection from random protocol opening to the first one, which must be the identify protocol, and then decide whether to continue to open the remaining protocols.

  This can slightly alleviate address pollution problems between different networks.

• feat: Wrap lock methods #994: Wrap lock methods (@keroro520)

  • feat: Avoid locking a long time

  ---

  [ᚬpr-mirror/976] feat: wrap lock methods #979 has closed by bot automatically and I cannot find the way to re-open it :(
  So here re-submit [ᚬpr-mirror/976] feat: wrap lock methods #979 here.

• feat: Allow miner add an arbitrary message into the cellbase #1000: Allow miner add an arbitrary message into the cellbase (@driftluo)

  Allow miner add an arbitrary message into the cellbase

  closed Allow miner add an arbitrary message into the cellbase #888

• feat: ckb init use secp256k1 as default ba-code-hash #1023: Ckb init use secp256k1 as default ba-code-hash (@doitian)

• feat: remove script_hash from docs/hashes.toml #1029: Remove script_hash from docs/hashes.toml (@doitian)

  It makes no sense since most system cells require args to work.

• feat: stats uncle rate #1047: Stats uncle rate (@zhangsoledad)

• feat: add indexer related rpc #905: Add indexer related rpc (@quake)

  This PR added 5 indexer related rpc, most lines of code are unit / integration test and document, core logic is in indexer/src/store.rs

• feat: ckb init allow set ba-data #1035: Ckb init allow set ba-data (@driftluo)

  1. ckb init allow set ba-data
  2. Verify that the input is valid hex

• feat: add an extra cost for dynamic length types #1055: Add an extra cost for dynamic length types (@yangby-cryptape)

  This is a breaking change: b:consensus

  BREAKING CHANGE:

  • The occupied capacity for dynamic length types is changed, and some data and hashes have to be updated.
  • The serialized size for dynamic length types is changed.

• feat: revise epoch rpc #1088: Revise epoch rpc (@zhangsoledad)

  This is a breaking change: b:rpc

Bug Fixes

• fix: update code hashes to correct value #969: Update code hashes to correct value (@xxuejie)

• fix: fix confusing JsonBytes deserializing error message #998: Fix confusing JsonBytes deserializing error message (@driftluo)

  fix Confusing JsonBytes deserializing error message #974

  1. Not 0x-prefixed hex string, error message:

  Config Error: Error { inner: ErrorInner { kind: Custom, line: None, col: 0, message: "invalid value: string \"7f52f0fccdd1d11391c441adfb174f87bca612b0\", expected a 0x-prefixed hex string", key: ["block_assembler", "args"] } }
  

  2. The illegal 0x-prefixed hex string

  Config Error: Error { inner: ErrorInner { kind: Custom, line: None, col: 0, message: "invalid length 43, expected even length", key: ["block_assembler", "args"] } }
  

• fix: witnesses_root cal including cellbase #1011: Witnesses_root cal including cellbase (@u2)

  https://github.com/cryptape/ckb-internal/issues/1063

• fix: avoid dummy worker re-solve the same works #1022: Avoid dummy worker re-solve the same works (@keroro520)

  • fix: Avoid dummy worker re-solve the same works. Resolving the same works and submit to chain, will increase the useless uncle blocks.

• fix: use Rust VM runner in bundled config file #1028: Use Rust VM runner in bundled config file (@doitian)

  Some command still uses the bundled config file, for example, following
  command will fail in Windows when ckb.toml not found in current directory:

    ckb cli hashes
  

• fix: peer_store time calculation overflow #1044: Peer_store time calculation overflow (@jjyr)

  Fixes panic: attempt to subtract with overflow #1026

• fix: RPC doc for deindex_lock_hash #1065: RPC doc for deindex_lock_hash (@ashchan)

  It always returns null, instead of empty array.

• fix: Check new block based on orphan block pool #1066: Check new block based on orphan block pool (@keroro520)

  • orphan_block_pool use ParentHash rather than BlockHash as key. So when we want to check whether we have received the same block as the arrived one, based orphan_block_pool, we should check deeply.

• fix: resolve ChainSpec script deserialize issue #1077: Resolve ChainSpec script deserialize issue (@quake)

• fix: cellset consistency #1084: Cellset consistency (@zhangsoledad)

  sync: [ᚬrc/v0.14.0] fix: cellset consistency #1076

Improvements

• refactor: remove low S check from util-crypto #938: Remove low S check from util-crypto (@jjyr)

  Cause we already use witness, the low S rule for signature is unnecessary.

  See https://github.com/bitcoin/bips/blob/master/bip-0062.mediawiki#new-rules

• refactor: Remove redundant interface from ChainProvider #959: Remove redundant interface from ChainProvider (@zhangsoledad)

  pub trait ChainProvider: Sync + Send {
      type Store: ChainStore;
  
      fn store(&self) -> &Arc<Self::Store>;
      ...
  -    fn block_body(&self, hash: &H256) -> Option<Vec<Transaction>>;
  -    fn block_header(&self, hash: &H256) -> Option<Header>;
  -    fn block_proposal_txs_ids(&self, hash: &H256) -> Option<Vec<ProposalShortId>>;
  -    fn uncles(&self, hash: &H256) -> Option<Vec<UncleBlock>>;
  -    fn block_hash(&self, number: BlockNumber) -> Option<H256>;
  -    fn block_ext(&self, hash: &H256) -> Option<BlockExt>;
  -    fn block_number(&self, hash: &H256) -> Option<BlockNumber>;
  -    fn block(&self, hash: &H256) -> Option<Block>;
  -    fn get_transaction(&self, hash: &H256) -> Option<(Transaction, H256)>;
      ...
  }

• perf(sync): Fix get ancestor performance issue #970 sync: Fix get ancestor performance issue (@TheWaWaR)

  • Use skip list to search ancestor of a header
  • Adjust block fetch interval IBD/not-IBD
  • Add test for skip list

  NOTE: The implementation and test is almost copy from bitcoin.

• refactor: flatten fields into shared #976: Flatten fields into shared (@keroro520)

  • Flatten some fields originally in Synchronizer, Relayer and Peers into SyncSharedState

  ---

  This PR intends to make Synchronizer and Relayer share the same global state. So I try to move their fields into SyncSharedState.
  Here only refactor around the "fields", and will refactor the "methods" in another PR.

• refactor: get tip_header from store instead of from chain_state #1051: Get tip_header from store instead of from chain_state (@jjyr)

• refactor: remove extra cost from capacity and simplify the occupied capacity #1062: Remove extra cost from capacity and simplify the occupied capacity (@yangby-cryptape)

  BREAKING CHANGE: Revert the commit 3ecf446.

Misc

• feat: proposer reward #922: Feat: proposer reward (@zhangsoledad)

  This is a breaking change: b:consensus

  Break Changes

  consensus

  1. earliest transaction proposer get 40% of the transaction fee as a reward.
  2. block reward finalized after proposal window close.
  3. enforce one-input one-output one-witness on cellbase.

  database

  1. block_ext.txs_verified -> block_ext.verified
  2. txs_fees persistence

  Enhancement

  1. lazy static always success script for test
  2. introduce AsCapacity make Capacity interface more flexible

• Replace system cell #1054: Replace system cell (@driftluo)

  This is a breaking change: b:consensus

  1. Replace system cell to another repo
  2. cherry pick [HOLD] feat: default lock script use recoverable signing #1012 to this pr

  BREAKING CHANGE: It changes the default secp256k1 script, which now uses recoverable signature.

[nervos-bot]

@driftluo is assigned as the chief reviewer

[doitian]

Config files changes:

ckb-miner.toml

chain.spec

 [chain]
-spec = "specs/testnet.toml"
+spec = { file = "specs/testnet.toml" }

ckb.toml

chain.spec

 [chain]
-spec = "specs/testnet.toml"
+spec = { file = "specs/testnet.toml" }

New options network.upnp and network.discovery_local_address

 [network]
+# If set to true, try to register upnp
+upnp = false
+# If set to true, network service will add discovered local address to peer store, it's helpful for private net development
+discovery_local_address = false

New option block_assembler.data

+# Also, ckb allows the miners to add any data to the cellbase that they have dug out.
+# The data must be A 0x-prefixed hex string.
+#
+# note: The data field is optional.
+#
+# **WARNING**: if data is larger than the capacity value of the current cellbase,
+# it will be truncated
+#
+# [block_assembler]
+# code_hash = "0xf1951123466e4479842387a66fabfd6b65fc87fd84ae8e6cd3053edb27fff2fd"
+# args = [ "ckb cli blake160 <compressed-pubkey>" ]
+# data = "0x prefix hex string"

[doitian]

RPC changes:

• Add new RPC module Indexer.
• Response schema of get_current_epoch, get_epoch_by_number has changed.

     "result": {
-        "block_reward": "100000000000",
         "difficulty": "0x3e8",
-        "last_block_hash_in_previous_epoch": "0x0000000000000000000000000000000000000000000000000000000000000000",
+        "epoch_reward": "125000000000000",
         "length": "1250",
         "number": "0",
-        "remainder_reward": "0",
         "start_number": "0"
     }

• Response schema of get_blockchain_info has changed

     "result": {
+        "alerts": [],
         "chain": "main",
         "difficulty": "0x3e8",
         "epoch": "0",
         "is_initial_block_download": true,
-        "median_time": "1557311762",
-        "warnings": ""
+        "median_time": "1557311762"
     }

[doitian]

Wrong head branch