Support for storage proof queries

[skunert]

In the old API there was state_getReadProof to which one would supply a list of keys and receive a storage proof including those keys. Support for something like this in the new API would be something I would like to see. Any thoughts on this?

[tomaka]

What would be the use case for these read proofs?

[skunert]

Every parachain block needs to include the set_validation_data inherent which along other things includes a read proof of some well-known keys from the relay chain. I integrated smoldot as a relay chain client into cumulus a while ago, but only for full-nodes. To actually build blocks I would need a way to fetch the storage proof so I can include it in the inherent.

[tomaka]

I don't really have a definite opinion on this at the moment, but for what it's worth you can almost construct a read proof using two consecutive chainHead_storage calls: one to get the value of all the necessary keys and for every single of their potential ancestors, and one to get the Merkle value of the children of these ancestors.
EDIT: actually you can do all of them in one call, I don't know why I thought two

It might be complicated to do for someone not familiar with the format of read proofs, but someone who's familiar with them should be able to do it.

I say "almost" because chainHead_storage doesn't let you query keys with an uneven number of nibbles, but that could easily be fixed.

[tomaka]

To give an example, if you want to generate a proof for the key 0xf005, you would ask for the:

• The storage value of 0xf0
• The storage value of 0xf005
• The closest descendant merkle value of 0xf1, 0xf2, 0xf3, 0xf4, ..., 0xfd, 0xfe, 0xff
• The closest descendant merkle value of 0xf01, 0xf02, 0xf03, 0xf04, ..., 0xf0d, 0xf0e, 0xf0f
• The closest descendant merkle value of 0xf000, 0xf001, 0xf002, 0xf003, 0xf004, 0xf006, ..., 0xf00d, 0xf00e, 0xf00f
• The closest descendant merkle value of 0xf0050, 0xf0051, 0xf0052, 0xf0053, ..., 0xf005d, 0xf005e, 0xf005f

Then based on all this information you can build the proof.

There's no need for the storage values of 0xf and 0xf00 because you know they're always empty as the number of nibbles is uneven and thus it's not possible for the runtime to write a value there.

Beyond the possibility to ask for the closest descendant Merkle value of trie nodes with an uneven number of nibbles (which isn't possible yet), we would also need a way to know the state version of each storage value (v0 or v1), which is also not possible yet but can be easily added with an extra field.

The pros of this approach are:

• More simplicity in the JSON-RPC API, as there are fewer functions and there's no need to introduce the concept of "Merkle proof" in the API.
• No need for extra code paths in the light client. Contrary to what one might think, these code paths aren't easy because you need to handle situations where the proof requested by the JSON-RPC client might be too big to fit in a single networking request, meaning that you would need to split the request and recombine the responses.
• No need to define what constitutes a valid proof or not. For example, would the proof still be valid if it includes duplicate items or unnecessary items?
• Maybe parachains might want to use a different proof format than the relay chain? Returning directly a proof from the JSON-RPC server kind of locks you to that format. However that's just a random thought and I'm not sure if it's even possible to use a different format.

The pros of adding a separate function that directly returns a proof are:

• It's way easier to write the JSON-RPC client code.
• A light client could directly send back the proof sent by the full node after verifying it. It's not that big of an advantage since the proof needs to be verified.