Moved ancestor hashes to state, changed to batched accumulator (#269)

* Replaced ancestors hashes in blocks with the previous block root
* Add a block root batched accumulator
* Cleanup processing presentation

specs/core/0_beacon-chain.md

@@ -67,7 +67,7 @@
             - [`get_shard_committees_at_slot`](#get_shard_committees_at_slot)
             - [`get_block_root`](#get_block_root)
             - [`get_beacon_proposer_index`](#get_beacon_proposer_index)
-            - [`get_updated_ancestor_hashes`](#get_updated_ancestor_hashes)
+            - [`merkle_root`](#merkle_root)
             - [`get_attestation_participants`](#get_attestation_participants)
             - [`bytes1`, `bytes2`, ...](#bytes1-bytes2-)
             - [`get_effective_balance`](#get_effective_balance)
@@ -77,16 +77,20 @@
             - [`hash_tree_root`](#hash_tree_root)
             - [`verify_casper_votes`](#verify_casper_votes)
             - [`integer_squareroot`](#integer_squareroot)
-            - [`bls_verify`](#blsverify)
-            - [`bls_verify_multiple`](#blsverifymultiple)
+            - [`bls_verify`](#bls_verify)
+            - [`bls_verify_multiple`](#bls_verify_multiple)
         - [On startup](#on-startup)
         - [Routine for processing deposits](#routine-for-processing-deposits)
         - [Routine for updating validator status](#routine-for-updating-validator-status)
     - [Per-slot processing](#per-slot-processing)
+        - [Misc counters](#misc-counters)
+        - [Block roots](#block-roots)
+    - [Per-block processing](#per-block-processing)  
+        - [Slot](#slot)  
         - [Proposer signature](#proposer-signature)
         - [RANDAO](#randao)
         - [PoW receipt root](#pow-receipt-root)
-        - [Block operations](#block-operations)
+        - [Operations](#operations)
             - [Proposer slashings](#proposer-slashings-1)
             - [Casper slashings](#casper-slashings-1)
             - [Attestations](#attestations-1)
@@ -154,6 +158,7 @@ Unless otherwise indicated, code appearing in `this style` is to be interpreted
 | `BEACON_CHAIN_SHARD_NUMBER` | `2**64 - 1` | - |
 | `BLS_WITHDRAWAL_PREFIX_BYTE` | `0x00` | - |
 | `MAX_CASPER_VOTES` | `2**10` (= 1,024) | votes |
+| `LATEST_BLOCK_ROOTS_COUNT` | `2**13` (= 8,192) | block roots |
 
 * For the safety of crosslinks a minimum committee size of 111 is [recommended](https://vitalik.ca/files/Ithaca201807_Sharding.pdf). (Unbiasable randomness with a Verifiable Delay Function (VDF) will improve committee robustness and lower the safe minimum committee size.) The shuffling algorithm generally ensures (assuming sufficient validators) committee sizes at least `TARGET_COMMITTEE_SIZE // 2`.
 
@@ -388,7 +393,7 @@ Unless otherwise indicated, code appearing in `this style` is to be interpreted
     'slot': 'uint64',
     # Skip list of ancestor beacon block hashes
     # i'th item is the most recent ancestor whose slot is a multiple of 2**i for i = 0, ..., 31
-    'ancestor_hashes': ['hash32'],
+    'parent_hash': 'hash32',
     'state_root': 'hash32',
     'randao_reveal': 'hash32',
     'candidate_pow_receipt_root': 'hash32',
@@ -459,6 +464,7 @@ Unless otherwise indicated, code appearing in `this style` is to be interpreted
     'latest_block_roots': ['hash32'],  # Needed to process attestations, older to newer
     'latest_penalized_exit_balances': ['uint64'],  # Balances penalized at every withdrawal period
     'latest_attestations': [PendingAttestationRecord],
+    'batched_block_roots': ['hash32'],
 
     # PoW receipt root
     'processed_pow_receipt_root': 'hash32',
@@ -477,7 +483,7 @@ Unless otherwise indicated, code appearing in `this style` is to be interpreted
     # RANDAO commitment
     'randao_commitment': 'hash32',
     # Slots the proposer has skipped (i.e. layers of RANDAO expected)
-    'randao_skips': 'uint64',
+    'randao_layers': 'uint64',
     # Balance in Gwei
     'balance': 'uint64',
     # Status code
@@ -655,7 +661,7 @@ Processing the beacon chain is similar to processing the Ethereum 1.0 chain. Cli
 
 For a beacon chain block, `block`, to be processed by a node, the following conditions must be met:
 
-* The parent block with `hash_tree_root` `block.ancestor_hashes[0]` has been processed and accepted.
+* The parent block with root `block.parent_root` has been processed and accepted.
 * The node has processed its `state` up to slot, `block.slot - 1`.
 * The Ethereum 1.0 block pointed to by the `state.processed_pow_receipt_root` has been processed and accepted.
 * The node's local clock time is greater than or equal to `state.genesis_time + block.slot * SLOT_DURATION`.
@@ -902,16 +908,14 @@ def get_beacon_proposer_index(state: BeaconState,
     return first_committee[slot % len(first_committee)]
 ```
 
-#### `get_updated_ancestor_hashes`
+#### `merkle_root`
 
 ```python
-def get_updated_ancestor_hashes(latest_block: BeaconBlock,
-                                latest_root: Hash32) -> List[Hash32]:
-    new_ancestor_hashes = copy.deepcopy(latest_block.ancestor_hashes)
-    for i in range(32):
-        if latest_block.slot % 2**i == 0:
-            new_ancestor_hashes[i] = latest_root
-    return new_ancestor_hashes
+def merkle_root(values):
+    o = [0] * len(values) + values
+    for i in range(len(values)-1, 0, -1):
+        o[i] = hash(o[i*2] + o[i*2+1])
+    return o[1]
 ```
 
 #### `get_attestation_participants`
@@ -1006,7 +1010,7 @@ def verify_casper_votes(state: BeaconState, votes: SlashableVoteData) -> bool:
 
     pubs = [aggregate_pubkey([state.validators[i].pubkey for i in votes.aggregate_signature_poc_0_indices]),
             aggregate_pubkey([state.validators[i].pubkey for i in votes.aggregate_signature_poc_1_indices])]
-    return bls_verify_multiple(pubkeys=pubs, msgs=[hash_tree_root(votes)+bytes1(0), hash_tree_root(votes)+bytes1(1), sig=aggregate_signature)
+    return bls_verify_multiple(pubkeys=pubs, messages=[hash_tree_root(votes)+bytes1(0), hash_tree_root(votes)+bytes1(1), signature=aggregate_signature)
 ```
 
 #### `integer_squareroot`
@@ -1040,7 +1044,7 @@ A valid block with slot `INITIAL_SLOT_NUMBER` (a "genesis block") has the follow
 {
     'header': BeaconBlockHeader(
         slot=INITIAL_SLOT_NUMBER,
-        ancestor_hashes=[ZERO_HASH for i in range(32)],
+        parent_hash=ZERO_HASH,
         state_root=STARTUP_STATE_ROOT,
         randao_reveal=ZERO_HASH,
         candidate_pow_receipt_root=ZERO_HASH,
@@ -1093,10 +1097,10 @@ def on_startup(initial_validator_deposits: List[Deposit],
 
         # Recent state
         latest_crosslinks=[CrosslinkRecord(slot=INITIAL_SLOT_NUMBER, shard_block_root=ZERO_HASH) for _ in range(SHARD_COUNT)],
-        latest_block_roots=[ZERO_HASH for _ in range(EPOCH_LENGTH * 2)],
+        latest_block_roots=[ZERO_HASH for _ in range(LATEST_BLOCK_ROOTS_COUNT)],
         latest_penalized_exit_balances=[],
         latest_attestations=[],
-
+        batched_block_roots=[]
         # PoW receipt root
         processed_pow_receipt_root=processed_pow_receipt_root,
         candidate_pow_receipt_roots=[],
@@ -1153,9 +1157,9 @@ def process_deposit(state: BeaconState,
     Note that this function mutates ``state``.
     """
     assert bls_verify(
-        pub=pubkey,
-        msg=hash(bytes32(pubkey) + withdrawal_credentials + randao_commitment),
-        sig=proof_of_possession,
+        pubkey=pubkey,
+        message=hash(bytes32(pubkey) + withdrawal_credentials + randao_commitment),
+        signature=proof_of_possession,
         domain=get_domain(
             state.fork_data,
             state.slot,
@@ -1170,7 +1174,7 @@ def process_deposit(state: BeaconState,
             pubkey=pubkey,
             withdrawal_credentials=withdrawal_credentials,
             randao_commitment=randao_commitment,
-            randao_skips=0,
+            randao_layers=0,
             balance=deposit,
             status=PENDING_ACTIVATION,
             latest_status_change_slot=state.slot,
@@ -1301,43 +1305,46 @@ def exit_validator(state: BeaconState,
 
 Below are the processing steps that happen at every slot.
 
-* Let `latest_block` be the latest `BeaconBlock` that was processed in the chain.
-* Let `latest_root` be the `hash_tree_root` of `latest_block`.
-* Set `state.slot += 1`
-* Set `state.latest_block_roots = state.latest_block_roots + [latest_root]`. (The output of `get_block_root` should not change, except that it will no longer throw for `state.slot - 1`).
+### Misc counters
+
+* Set `state.slot += 1`.
+* Set `state.validator_registry[get_beacon_proposer_index(state, state.slot)].randao_layers += 1`.
+
+### Block roots
+
+* Let `previous_block_root` be the `tree_hash_root` of the previous beacon block processed in the chain.
+* Set `state.latest_block_roots = state.latest_block_roots[1:] + [previous_block_root]`.
+* If `state.slot % LATEST_BLOCK_ROOTS_COUNT == 0` append `merkle_root(state.latest_block_roots)` to `state.batched_block_roots`.
 
-If there is a block from the proposer for `state.slot`, we process that incoming block:
+## Per-block processing
 
-* Let `block` be that associated incoming block.
-* Verify that `block.slot == state.slot`
-* Verify that `block.ancestor_hashes` equals `get_updated_ancestor_hashes(latest_block, latest_root)`.
+Below are the processing steps that happen at every `block`.
 
-If there is no block from the proposer at state.slot:
+### Slot
 
-* Set `state.validator_registry[get_beacon_proposer_index(state, state.slot)].randao_skips += 1`.
-* Skip all other per-slot processing. Move directly to [per-epoch processing](#per-epoch-processing).
+* Verify that `block.slot == state.slot`.
 
 ### Proposer signature
 
-* Let `block_root_without_sig` be the `hash_tree_root` of `block` where `block.signature` is set to `[0, 0]`.
-* Let `proposal_root = hash_tree_root(ProposalSignedData(state.slot, BEACON_CHAIN_SHARD_NUMBER, block_root_without_sig))`.
-* Verify that `bls_verify(pubkey=state.validator_registry[get_beacon_proposer_index(state, state.slot)].pubkey, data=proposal_root, sig=block.signature, domain=get_domain(state.fork_data, state.slot, DOMAIN_PROPOSAL))`.
+* Let `block_without_signature_root` be the `hash_tree_root` of `block` where `block.signature` is set to `[0, 0]`.
+* Let `proposal_root = hash_tree_root(ProposalSignedData(state.slot, BEACON_CHAIN_SHARD_NUMBER, block_without_signature_root))`.
+* Verify that `bls_verify(pubkey=state.validator_registry[get_beacon_proposer_index(state, state.slot)].pubkey, data=proposal_root, signature=block.signature, domain=get_domain(state.fork_data, state.slot, DOMAIN_PROPOSAL))`.
 
 ### RANDAO
 
 * Let `repeat_hash(x, n) = x if n == 0 else repeat_hash(hash(x), n-1)`.
 * Let `proposer = state.validator_registry[get_beacon_proposer_index(state, state.slot)]`.
-* Verify that `repeat_hash(block.randao_reveal, proposer.randao_skips + 1) == proposer.randao_commitment`.
+* Verify that `repeat_hash(block.randao_reveal, proposer.randao_layers) == proposer.randao_commitment`.
 * Set `state.randao_mix = xor(state.randao_mix, block.randao_reveal)`.
 * Set `proposer.randao_commitment = block.randao_reveal`.
-* Set `proposer.randao_skips = 0`.
+* Set `proposer.randao_layers = 0`.
 
 ### PoW receipt root
 
 * If `block.candidate_pow_receipt_root` is `x.candidate_pow_receipt_root` for some `x` in `state.candidate_pow_receipt_roots`, set `x.votes += 1`.
 * Otherwise, append to `state.candidate_pow_receipt_roots` a new `CandidatePoWReceiptRootRecord(candidate_pow_receipt_root=block.candidate_pow_receipt_root, votes=1)`.
 
-### Block operations
+### Operations
 
 #### Proposer slashings
 
@@ -1346,8 +1353,8 @@ Verify that `len(block.body.proposer_slashings) <= MAX_PROPOSER_SLASHINGS`.
 For each `proposer_slashing` in `block.body.proposer_slashings`:
 
 * Let `proposer = state.validator_registry[proposer_slashing.proposer_index]`.
-* Verify that `bls_verify(pubkey=proposer.pubkey, msg=hash_tree_root(proposer_slashing.proposal_data_1), sig=proposer_slashing.proposal_signature_1, domain=get_domain(state.fork_data, proposer_slashing.proposal_data_1.slot, DOMAIN_PROPOSAL))`.
-* Verify that `bls_verify(pubkey=proposer.pubkey, msg=hash_tree_root(proposer_slashing.proposal_data_2), sig=proposer_slashing.proposal_signature_2, domain=get_domain(state.fork_data, proposer_slashing.proposal_data_2.slot, DOMAIN_PROPOSAL))`.
+* Verify that `bls_verify(pubkey=proposer.pubkey, message=hash_tree_root(proposer_slashing.proposal_data_1), signature=proposer_slashing.proposal_signature_1, domain=get_domain(state.fork_data, proposer_slashing.proposal_data_1.slot, DOMAIN_PROPOSAL))`.
+* Verify that `bls_verify(pubkey=proposer.pubkey, message=hash_tree_root(proposer_slashing.proposal_data_2), signature=proposer_slashing.proposal_signature_2, domain=get_domain(state.fork_data, proposer_slashing.proposal_data_2.slot, DOMAIN_PROPOSAL))`.
 * Verify that `proposer_slashing.proposal_data_1.slot == proposer_slashing.proposal_data_2.slot`.
 * Verify that `proposer_slashing.proposal_data_1.shard == proposer_slashing.proposal_data_2.shard`.
 * Verify that `proposer_slashing.proposal_data_1.block_root != proposer_slashing.proposal_data_2.block_root`.
@@ -1383,7 +1390,7 @@ For each `attestation` in `block.body.attestations`:
 * `aggregate_signature` verification:
     * Let `participants = get_attestation_participants(state, attestation.data, attestation.participation_bitfield)`.
     * Let `group_public_key = BLSAddPubkeys([state.validator_registry[v].pubkey for v in participants])`.
-    * Verify that `bls_verify(pubkey=group_public_key, msg=hash_tree_root(attestation.data) + bytes1(0), sig=attestation.aggregate_signature, domain=get_domain(state.fork_data, attestation.data.slot, DOMAIN_ATTESTATION))`.
+    * Verify that `bls_verify(pubkey=group_public_key, message=hash_tree_root(attestation.data) + bytes1(0), signature=attestation.aggregate_signature, domain=get_domain(state.fork_data, attestation.data.slot, DOMAIN_ATTESTATION))`.
 * [TO BE REMOVED IN PHASE 1] Verify that `attestation.data.shard_block_hash == ZERO_HASH`.
 * Append `PendingAttestationRecord(data=attestation.data, participation_bitfield=attestation.participation_bitfield, custody_bitfield=attestation.custody_bitfield, slot_included=state.slot)` to `state.latest_attestations`.
 
@@ -1430,7 +1437,7 @@ Verify that `len(block.body.exits) <= MAX_EXITS`.
 For each `exit` in `block.body.exits`:
 
 * Let `validator = state.validator_registry[exit.validator_index]`.
-* Verify that `bls_verify(pubkey=validator.pubkey, msg=ZERO_HASH, sig=exit.signature, domain=get_domain(state.fork_data, exit.slot, DOMAIN_EXIT))`.
+* Verify that `bls_verify(pubkey=validator.pubkey, message=ZERO_HASH, signature=exit.signature, domain=get_domain(state.fork_data, exit.slot, DOMAIN_EXIT))`.
 * Verify that `validator.status == ACTIVE`.
 * Verify that `state.slot >= exit.slot`.
 * Verify that `state.slot >= validator.latest_status_change_slot + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD`.
@@ -1666,7 +1673,6 @@ while len(state.persistent_committee_reassignments) > 0 and state.persistent_com
 ### Final updates
 
 * Remove any `attestation` in `state.latest_attestations` such that `attestation.data.slot < state.slot - EPOCH_LENGTH`.
-* Set `state.latest_block_roots = state.latest_block_roots[EPOCH_LENGTH:]`.
 
 ## State root processing