Solana geyser plugin for proof generation (inclusion and non-inclusion) per block

[dubbelosix]

Summary

The major component in this PR is a geyser plugin that can be plugged into a solana validator and stream merkle inclusion and non-inclusion proofs for the accounts that we want to monitor. If an account is modified in a specific block, we get an inclusion proof. If the account is not modified we get a non-inclusion merkle proof.

Note

The following is a WIP and has been halted at a stable point. We will be gradually committing to it on a need basis and encouraging open source contributions, but it is incomplete as of now

Changes

• Geyser plugin for monitoring and accumulating accounts, blocks to generate the proof
• Simple TCP server as part of the geyser plugin that streams the proofs to a listening client
• Simple TCP client that listens to server, gets the inclusion proofs and verifies them
• Merkle proof generation for account delta hash
• Types for comprehensive proof generation (including data)
• Simple on chain program for proving the value of a specific account
• GEYSER.md readme file added with steps remaining. will be updated as more outstanding items are found

[vovkman]

There is a big push right now to pull RPC functionality out of the validator. The backbone of this is a geyser plugin that runs a gRPC server with subscriptions mirroring the geyser interface (can see some more info here and here, the component you would implement in this model would be an Adapter Sidecar)

Wondering if you would consider writing this plugin as a consumer of the gRPC stream instead of as a dedicated plugin? Some of the benefits are

• dynamically update subscription
• upgrade your code without having to restart geyser node
• consume data from multiple sources
• ability to add N additional consumers as requirements change without adding a ton of logic to existing geyser plugin or developing a new plugin dedicated to that purpose

We are using this plugin successfully in production now and lots of resources are being put towards it

[dubbelosix]

hi @vovkman , one of the reasons to run it inside the validator is to push the responsibility of running extra infra into the validator itself and to not keep track of missed updates etc on the client side - in case something goes wrong or older updates are needed, its easier to start the validator with a previous snapshot and the plugin would process the blocks from that point o

the original client was actually one that calculated the proofs etc on the client side based on raw geyser updated (as you suggested), but would it would require more responsibility on the client side (and the server as well, since the intermediate account updates need to be stored to be served). if you check https://github.com/Sovereign-Labs/sovereign-sdk/pull/1087/files#diff-f14178dd29b7be7ec21b4b270e14f8b0d4a75aac7ffad6e4d8d928bfbf186cb9 its actually client side implementation for the block/account processing.

also, to clarify, this just requires an RPC (or a full node), and doesn't need to run inside a voting validator

overall, I believe whatever solution is easier infra-wise for application layer :)

[codecov]

Codecov Report

Merging #1087 (3b959f1) into nightly (7cf2e64) will increase coverage by 0.0%.
The diff coverage is n/a.

see 2 files with indirect coverage changes