New repository structure

[paulhauner]

Problem

Whenever we need to bump up a spec version, it requires a single PR to update the entire repository, including eth2 libraries, the beacon node binary and the validator client binary.

The present all-in-one repository structure means that updates to the spec:

• Create giant PRs that are impractical to review.
• Almost always create merge conflicts as it touches a lot of files.
• Create an unreasonable workload for whoever needs to do the update.

Solution

We (@AgeManning, @spble and I) propose to spawn three new repos from the present lighthouse repository. Here are their names and purposes:

• sigp/lighthouse (the existing one, here): contains all other repositories as git submodules, provides integration testing and likely some form orchestration scripts (docker, etc). It will also serve as the "landing page" for the other repos -- it will provide help/support and direct users to the other repos. If people choose to raise issues there that belong on another repo, we can just transfer the issue there using the Github beta transfer feature.
• sigp/lighthouse-beacon: contains the beacon node binary.
• sigp/lighthouse-validator: contains the validator client binary.
• sigp/lighthouse-common: contains the present eth2 dir and will contain crates that are common to the other crates (e.g., state-less spec logic, ssz, protos, ??)

Downsides

• More places to look for issues and PRs; easy to miss or lose things and more overhead.
• More overhead for new developers.
• More version management (e.g., "is the beacon_node on spec v0.1.0 or 0.2.0?")
• It'll look like we have less stars because they won't be collected into the same place.

Upsides

• We can move faster and more freely when it comes to the spec -- we can keep on the cutting edge of the pure spec logic without also needing to keep the binaries at the cutting edge too.
• We can progress different components at different times. E.g., we can choose to lag the validator client behind the beacon node, or vice-versa.
• Smaller tasks and PRs. E.g., you can update the spec functions in on PR and ripple those updates across to the beacon node in another.
• Seems like best-practice.
• More places for people to put stars! They can even star us 4 times if they want!!

[protolambda]

A few days ago I requested prysmatic labs to move their SSZ lib to its own repo, and I would like to see the same here. Not only as an effort to prevent bigger change-sets for releases to either library, but also to invite community projects to contribute back to the common libraries.

See Prysm issue: prysmaticlabs/prysm#1549

[AgeManning]

Our current thinking is that there are quite a few such libraries that will be cross-project such that it would be infeasible to maintain a separate repo for each individually.
Our plan is to separate these into their own crates in the lighthouse-common repo.
Each crate can be maintained and upgraded independently and pulled individually from crates.io.

[paulhauner]

Accidental close -- I subconsciously clicked "close" because I wanted to abandon the message I was typing. Oops!