[WIP] Turboproof

[s1na]

Port of @gballet's multiproof-rs which is itself an implementation of @ledgerwatch's multiproof algorithm (with minor differences).

[zmitton]

I'm not sure what this solves that can't be solved with a the concept of a sparse (proof) tree.

If the goal is to have a root-only client, you would just have the full-client dump the "touched" tree-nodes during the state-transition in question.

The root-only client would build a proof tree from the nodes and try the operation.

[s1na]

If the goal is to have a root-only client, you would just have the full-client dump the "touched" tree-nodes during the state-transition in question.

I agree that it is possible. In fact I have a prototype of this here.

I think the original design has been for the turbo-geth client (I don't remember exactly for what purpose). For most use-cases I agree a partial proof trie is perfectly sufficient. But for our use-case this is not about a feature, but efficiency.

For a stateless contract, if you take the partial proof trie approach, you'd have to first store every proof node in a kv-db (a lot of memory copy), and then to verify each key, the trie has to be walked down once (with every node fetched from kv-db), and for updating each key, the trie has to be walked down once to find value, update it, and then hashed up to the root. Doing all this costs more gas than necessary.

The hope is that with this opcode-based approach, you could verify multiple keys in one pass. You could potentially even do update in the same pass (there's some difficulty if new nodes have to be added/deleted though).

[zmitton]

Thank you for the explanation. With the partial proof trie approach, you don't have to do the validation step for any of the leaves. It is only once the tree is being used that validation occurs automatically. Same with updates. I'm not sure if there are any extra steps for your specific use case?

I don't like this rigid approach, but admit that I don't fully exactly follow all the details (read the link a few times tho). Given how much complexity the tree itself already has, this is going to be hard to get right (lots of edge cases).

[coveralls]

Coverage increased (+0.04%) to 93.006% when pulling d125f36 on turboproof into d48a38c on master.

[s1na]

For now I'm developing this proof format here, because it was partly tied to the EE we were developing using the proof format. For this reason I'm not sure about merging this code here. I can see arguments for both sides and have no strong preference. If there's no appetite for it right now we can close this PR.

[holgerd77]

Maybe we just keep this open for some more time without merging, than people have a greater chance to discover. If there is 3rd-party demand here, we can then re-take on a decision.

[s1na]

It'll make sense to include this PR in the library when things get more serious with the research being done on stateless clients for Eth1, e.g. see here. Although it's unlikely that the multiproof format stays exactly the same (it has already changed).