Added light client related files #766

vbuterin · 2019-03-13T07:55:50Z

No description provided.

zah · 2019-03-15T14:00:44Z

This new proposal is really good and easy to follow, but I do have some questions:

Consider the situation where the light client receives a BlockValidityProof from a valid committee majority, but the referenced block did not become part of the canonical chain. As I see it, this may happen if the proposer for the next slot didn't notice the referenced block in time and some alternative history was created and got eventually selected due to the fork choice rules.

How will the light client handle this situation? Perhaps it has to keep requesting BlockValidityProofs from other peers until it gets one at a better height? The situation is somewhat bad, because the light client will trust a crosslink that was not officially confirmed.
The sync procedure allows you to keep monitoring the changes in a particular shard's committee (and a state root the validators agreed on), but how do you get the very latest crosslink of a different shard? Presumably, you can use the known state root to verify a merkle proof for the members of another committee, but perhaps the spec needs to be refactored a bit to facilitate this (we need a function that computes the committee for a particular shard).

specs/light_client/sync_protocol.md

vbuterin · 2019-03-16T03:11:03Z

Consider the situation where the light client receives a BlockValidityProof from a valid committee majority, but the referenced block did not become part of the canonical chain. As I see it, this may happen if the proposer for the next slot didn't notice the referenced block in time and some alternative history was created and got eventually selected due to the fork choice rules. How will the light client handle this situation?

I'm inclined to say that light clients can just keep asking for headers, and internally apply LMD GHOST based on the committee they're following. It is possible for this fork choice to diverge from the "official" one, but only in cases where close to 1/2 of validators or the committee are byzantine.

The sync procedure allows you to keep monitoring the changes in a particular shard's committee (and a state root the validators agreed on), but how do you get the very latest crosslink of a different shard?

You authenticate the Merkle path shard block header -> beacon block header -> state -> latest_crosslink_data.

zah · 2019-03-16T09:28:11Z

So, the light client has a slightly weaker notion of finality. Isn't this a problem for the use case of authenticating ETH2 data in other blockchains? You cannot rely on heuristics like requesting data from other peers in such contexts.

You authenticate the Merkle path shard block header -> beacon block header -> state -> latest_crosslink_data.

But the latest_crosslink_data is not necessarily the very latest crosslink. Two scenarios can happen:

The monitored shard did not receive block proposal in the most recent epochs. The best BlockValidityProof proof will then sync the client with a slightly older version of the state. There might even be some new crosslinks for the other shards that are finalized at this point.
The client is interested in the non-finalized crosslinks appearing in the slots after the block in the BlockValidityProof.

Both problems are solvable if the client is able to reconstruct another committee and then use that to authenticate a recent block header.

vbuterin · 2019-03-17T11:33:56Z

So if you want to learn about finalized blocks, then you can authenticate the Merkle path shard block header -> beacon block header -> state -> finalized_epoch -> historical block root[finalized_epoch], and there's your finalized block. Even if that shard block gets reverted later, if you assume honest majority of the persistent committee, the fact that that block exists at all implies that the finalized block actually is finalized.

zah · 2019-03-17T13:15:10Z

Fair enough, thanks for the clarification.

Am I right to assume that one of the next required steps would be to develop an automatic calculator that, given an arbitrary SSZ expression/function, can compute the size of the required Merkle proof and perhaps the minimal amount of time (gas) to verify it in a certain environment? Such a tool can also try to produce a zk-snark circuit to test the feasibility of compressing the proof. I'll be very happy to work on something like this.

Having such a tool would allow us to compare in more concrete terms the various possible ways to arrange the signed data. I think there might be some reasonable optimisations brought by denormalizing the data a little bit (i.e. including an easier to reach copy of something, that can be otherwise obtained through a longer chain of links). This affects the size of the Merkle proofs, the required verification time and the size of the zk-snark circuits.

As an example, even here, you are suggesting that the BlockValidityProof will include a signed block header, but in reality this will have to be the AttestationData that the validators sign and broadcast. By adding additional fields or signatures in the attestations, we make certain trade-offs in the required computation time and network traffic for both light clients and full nodes.

vbuterin · 2019-03-18T03:12:16Z

Am I right to assume that one of the next required steps would be to develop an automatic calculator that, given an arbitrary SSZ expression/function, can compute the size of the required Merkle proof and perhaps the minimal amount of time (gas) to verify it in a certain environment?

It depends on what your goals are :)

If you just want to build a light client, then none of that is needed, you just need a tool to create and verify multiproofs of arbitrary SSZ objects (that happens to be on my own todolist as well). If you want to build an eth2-in-eth1 light client,then sure, though the code for that is easy; it's just verifying a Merkle multiproof.

As an example, even here, you are suggesting that the BlockValidityProof will include a signed block header, but in reality this will have to be the AttestationData that the validators sign and broadcast

Don't think this has anything to do with AttestationData. This particular light client design is based off of persistent committee shard attestations, not crosslink committee block attestations (it's done that way to cut costs, as you would only need to download a new committee every ~9 days).

* shuffled committee -> period committee * Reduced code redundancy

djrtwo · 2019-03-29T17:36:10Z

What's the status of these vs the files already in dev branch?
It seemed that some were accidentally direct committed to master at some point

vbuterin · 2019-03-30T07:45:07Z

These files are newer. I feel like we should just merge this PR in soon; it's not final but it's definitely closer to final and more updated than what exists there now.

djrtwo · 2019-04-02T03:15:01Z

specs/light_client/merkle_proofs.md

+    return MerklePartial(
+        root=hash_tree_root(obj),
+        indices=indices,
+        values= mk_multi_proof


mk_multi_proof is undefined, and this function does not set proof

Removed that section entirely for now. I think the right time to bring it back would be after we have executable SSZ serialization, as then we would be able to actually generate the proofs as part of the spec (right now I'm not sure what the "final" interface in the executable spec would be...)

djrtwo

lgtm for now. merging

vbuterin added 2 commits March 13, 2019 02:54

Added light client related files

3371dcc

Added links to light client docs in the readme

154eec0

hwwhww added the scope:light-clients label Mar 14, 2019

vbuterin and others added 3 commits March 14, 2019 08:29

Updated Merkle proof file

b65601a

Update merkle_proofs.md

22be212

Update sync_protocol.md

b566722

zah reviewed Mar 15, 2019

View reviewed changes

specs/light_client/sync_protocol.md Outdated Show resolved Hide resolved

specs/light_client/sync_protocol.md Outdated Show resolved Hide resolved

Fixed some variable names

1967a89

vbuterin added 2 commits March 20, 2019 11:49

Edits

ced6208

* shuffled committee -> period committee * Reduced code redundancy

Add merkle_parial_from_paths

b91dfd6

Merge branch 'dev' into vitalik81

2418bab

djrtwo reviewed Apr 2, 2019

View reviewed changes

Removed merkle partial from paths for now

bee740e

djrtwo approved these changes Apr 3, 2019

View reviewed changes

djrtwo merged commit afdfb2a into dev Apr 3, 2019

djrtwo deleted the vitalik81 branch April 3, 2019 06:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added light client related files #766

Added light client related files #766

vbuterin commented Mar 13, 2019

zah commented Mar 15, 2019 •

edited

Loading

vbuterin commented Mar 16, 2019

zah commented Mar 16, 2019 •

edited

Loading

vbuterin commented Mar 17, 2019

zah commented Mar 17, 2019

vbuterin commented Mar 18, 2019 •

edited

Loading

djrtwo commented Mar 29, 2019

vbuterin commented Mar 30, 2019 •

edited

Loading

djrtwo Apr 2, 2019

vbuterin Apr 3, 2019

djrtwo left a comment

Added light client related files #766

Added light client related files #766

Conversation

vbuterin commented Mar 13, 2019

zah commented Mar 15, 2019 • edited Loading

vbuterin commented Mar 16, 2019

zah commented Mar 16, 2019 • edited Loading

vbuterin commented Mar 17, 2019

zah commented Mar 17, 2019

vbuterin commented Mar 18, 2019 • edited Loading

djrtwo commented Mar 29, 2019

vbuterin commented Mar 30, 2019 • edited Loading

djrtwo Apr 2, 2019

Choose a reason for hiding this comment

vbuterin Apr 3, 2019

Choose a reason for hiding this comment

djrtwo left a comment

Choose a reason for hiding this comment

zah commented Mar 15, 2019 •

edited

Loading

zah commented Mar 16, 2019 •

edited

Loading

vbuterin commented Mar 18, 2019 •

edited

Loading

vbuterin commented Mar 30, 2019 •

edited

Loading