Rework crosslink processing

[mkalinin]

What was wrong?

Addresses a number of issues in crosslink processing encountered during update to v0.5.1:

1. Attestation created in one epoch but not included into any block of that epoch had a chance to eventually become invalid. It happened because of crosslink update on the closest epoch transition which made previous_crosslink in the attestation outdated. This PR, also, adds a test to a spec that covers this case. Described in this issue Previous crosslink becomes outdated after epoch transition #845.
2. Penalties and rewards for crosslinks were calculated after crosslinks have been processed, therefore get_winning_root_and_participants returned empty participation lists for crosslinks of updated shards as this update made previous_crosslink in attestations out of date.
3. There is a case when two attestations for a same shard, i.e. for previous and current epoch, are included into a state during current epoch. In this case attestation from previous epoch had a chance to win cause epoch was not counted in the comparison of crosslink data roots. Mentioned here: Previous crosslink becomes outdated after epoch transition #845 (comment).
4. Shard was not counted in attestation filtering. Supplants Missed filter condition in get_winning_root_and_participants #841.
5. An edge case with GENESIS_EPOCH caused incorrect crosslink processing. Mentioned here: Missed filter condition in get_winning_root_and_participants #841 (comment).

How it was fixed.

1. Outdated previous_crosslink was fixed by breaking crosslinks down to previous_epoch_crosslinks and current_epoch_crosslinks. Now attestation is checked against both lists which prevents it from being spoiled. There was another solution of this issue, to process crosslinks only for previous epoch. But this approach adds a delay to a shard crosslinking, IMO, a speed of crosslinking process is invariant that is not desired to be broken.
2. To work around rewards issue crosslinks are calculated and stored in memory before rewards calculation and got applied after apply_reward has been called during the epoch processing. It preserves an invariant of calculating voting balances before validator balances get changed.
3. Other issues have been fixed by adding a strict order to crosslink processing: current epoch is processed after previous one.

Crosslinks processing flow

This is how this PR affects epoch processing:

def process_epoch_transition(state: BeaconState) -> None:
    spec.update_justification_and_finalization(state)
    
    # calculating crosslinks:
    latest_crosslinks = spec.get_latest_crosslinks(state)
    
    spec.maybe_reset_eth1_period(state)

    # applying rewards and crosslinks after them
    spec.apply_rewards(state)
    spec.apply_crosslinks(state, latest_crosslinks)
    ...

Complexity

This PR adds a certain degree of complexity to crosslink and epoch processing. This complexity has been added in order to preserve invariants that from my perspective are important for the beacon chain processing. It may be there are other ways of solving these issues, would be happy to discuss them here.

Naming and types

Disclaimer: I am not pythonist. In this PR you can meet bad namings, not that idiomatic function decomposition alongside with inappropriate type usage (esp. Vector).

Testing

These changes have been tested in our simulator. It works well for ideal case when all attestations are included in the next coming block, but I believe those are good for sophisticated cases as well. Corresponding commit harmony-dev/beacon-chain-java@a1ee101

Misc

This PR a bit affects Phase 1 spec. There is also renaming of previous_crosslink in attestation data to source_crosslink; this is done because previous_crosslink name started to clash with previous_epoch_crosslinks. Not sure that I like source_crosslink but gave up on picking something else.

Related issues and PRs

Addresses #845, supplants #841.

[djrtwo]

Agreed that there are a few things that will be better served with dual crosslink arrays, but I'm going to take a stab at simplifying this before we merge.

Thanks!

[mkalinin]

but I'm going to take a stab at simplifying this before we merge.

Happy to implement your simplification when it's done, would be helpful to check it in a simulation before the merge.

[mkalinin]

Superseded by #874