Tracking: Update lightwalletd RPCs for "spend before sync" algorithm

[teor2345]

Motivation

The zcashd and lightwalletd teams have been working on a "spend before sync" algorithm, targeted for zcashd 5.6.0 in June 2023.

As part of that change, the JSON-RPC interface used by lightwalletd will need extra methods or fields.

Preparation:

completed preparation

• Make the first stable release forward-compatible with planned state changes #6746

• Create an empty database format update task and format update tests #6955

• Tracking: Upgrade shared ECC dependencies and zcash_script for zcashd 5.6.0 #6859 (Update incrementalmerkletree crate to 0.4.0)

  • Add round-trip note commitment tree serialization format tests #7143

• Upgrade lightwalletd used by Zebra #7292

  • fix(ci): Show correct lightwalletd cache heights in CI cached state #7390
  • change(tests): update tests for new lightwalletd version used in Zebra #7349

Zebra will need to implement some of those changes to remain compatible with the latest lightwalletd versions:

• Add note tree size fields to the getblock RPC #6952
• Add note subtree indexes to the finalized and non-finalized state #6953
  • bug: some subtrees have incorrect end heights after a full sync #7532
  • Make subtree index upgrade compatible with lightwalletd back-tracking algorithm #7407
  • Test that "sync from empty" and "upgrade from existing" state code produces the same data #7570
• Implement z_getsubtreesbyindex RPC #6954
  • Test the z_getsubtreesbyindex RPC #7449
    • Add format checks with fixed subtree values from zcashd #7446
    • Manually test z_getsubtreesbyindex using zcash-rpc-diff #7450
    • Test z_getsubtreesbyindex using a lightwalletd gRPC request #7451
    • Add snapshot tests for the z_getsubtreesbyindex RPC #7513

Extra Dependencies & Tests:

completed

• Check that the note commitment tree RPC serialization format hasn't changed in zcashd 5.6.0 #7201
• Remove duplicate dependencies from deny.toml once ed25519-zebra and zcashd are released #6638
• Fix lightwalletd sync finish detection code in tests #7335

Lower Priority

Minor bug or test fixes:

• Fix proptest coverage of note commitment subtree serialization #7443
• Compare subtrees in non-finalized chains when evaluating chain equality #7447
• Check that there are no missing note commitment trees after deduplication #7461
• Correctly populate subtree values in ZebraDb::note_commitment_trees() #7444
• Add Zebra and zcashd versions to the output of zcash-rpc-diff #7351
• Rename subtree methods and types to "subtree root" #7463

completed performance & documentation

To get reasonable lookup performance:

• Store only the first tree state in each series of identical tree states #4784

Documentation:

• Document how to do in-place upgrades of the state database format #7215
• Update lightwalletd fork in zebra book and code docs #7319
  • Missing test harness logs in Zebra CI Docker tests due to RUST_LOG env var #7370

Optional Cleanups & Fixes:

these tickets are optional, ask Pili before starting them

Bug fixes:

• fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots #7392
• Return errors instead of panicking in methods for Heights #7263

Technical debt:

• (De)Serialize NoteCommitmentTrees using supported serialization format instead of bincode #7383
• change(state): Move format upgrades to separate modules #7377

Diagnostics:

• Missing diagnostic spans on state logs #7369
• diagnostic: Log column family and database size on startup and shutdown #7416

Needs decision/triage:

• Decide if we need to speed up adding subtrees for existing database blocks #7441

Timing

zcashd aims to finalize the RPC design by the end of May, and then release it in the first two weeks of June.

lightwalletd will start calling the new RPCs around the same time, but they aim to be compatible with old RPC versions using an off-by-default flag.

Implementation Notes

Zebra already stores the data we need for these lookups. We might need extra indexes for good performance, but that's not a breaking state change. (So we don't need to upgrade the state version, we can just automatically add/update a new column family with the new index at startup.)

We will need to implement #4784 to have acceptable note commitment tree scanning performance.

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @dconnolly @oxarbitrage @teor2345 @upbqdn

[teor2345]

@mpguerra I can't estimate this ticket until the zcashd team has decided what the changes to the RPCs will be.

[str4d]

The current draft of the JSON-RPC changes is here (last few commits, the PR is currently based on another one and will be rebased as the backend changes are landed): zcash/zcash#6677.

[teor2345]

I've added a draft list of RPC changes to the ticket, just trying to work out how to calculate subtree roots:
https://discord.com/channels/809218587167293450/809251012610752513/1110391483367960646

[teor2345]

Note from the engineering sync: This seems risky to do between the last release candidate and the first stable release. There is no specific deadline for this change.

[teor2345]

@mpguerra I'd like to talk through the design for this change with the engineers who will be working on it. We need to work out the tree API changes, and the database format changes.

Did you want to have a large group meeting? Or we could do it in two smaller groups: tree APIs and database format.

[teor2345]

Also I just split this task into tickets, can we schedule them in this sprint or next sprint?
I haven't set up the dependencies yet, but they are in the right order in the list in the PR.

[mpguerra]

Unscheduling as this will be the overall "tracking" issue and we'll schedule individual issues in sprints instead

[mpguerra]

@mpguerra I'd like to talk through the design for this change with the engineers who will be working on it. We need to work out the tree API changes, and the database format changes.

Did you want to have a large group meeting? Or we could do it in two smaller groups: tree APIs and database format.

I think it's probably best to have smaller group meetings and involve only the people that will need to be involved

[mpguerra]

We are done here for now 🎉