Optimization: Avoid duplicating proofs during txn verification

[tizoc]

Explain your changes:

• When a batch of transactions are verified using the verifier helper subprocess, they are serialized and sent to the verifier subprocess. The verifier subprocess will then write the transactions as Valid.t back to the node, which in will decode them from a stream creating duplicates of the already existing transactions. This is an issue for zkApp transactions because those contain proofs which are quite memory-heavy.

This patch removes the duplicates from the response, and converts the Verifiable.t values to their Valid.t counterparts through the unsafe User_command.to_valid_unsafe function.

Explain how you tested your changes:

• Running patched nodes in a mixed-enviroment (patched and unpatched nodes) in a private cluster.

From our testing, we noted this saves quite a bit of memory (both these nodes were launched at the same time):

Patched	Unpatched

Checklist:

• Dependency versions are unchanged
  • Notify Velocity team if dependencies must change in CI
• Modified the current draft of release notes with details on what is completed or incomplete within this project
• Document code purpose, how to use it
  • Mention expected invariants, implicit constraints
• Tests were added for the new behavior
  • Document test purpose, significance of failures
  • Test names should reflect their purpose
• All tests pass (CI will check this if you didn't)
• Serialized types are in stable-versioned modules
• Does this close issues? List them

• Closes #0000

[bkase]

I'm slightly sad that we're introducing some potential future error if the command/result ordering ever gets decoupled; ideally we'd add a unit test to catch this kind of regression, but I'm not sure it's worth refactoring the code to do that and risk making this a larger change than it needs to be given where we are with the testnet.

Curious if anyone disagrees with me? cc @nholland94

[tizoc]

One way to make it more explicit is to tag (in the caller) each command with an id that then must be matched in the response, something like this:

(* Each command is tagged with a unique identifier in the caller. *)
let tag_commands_with_ids commands =
  List.mapi commands ~f:(fun id command -> (id, command))

(* The verify function in the subprocess now also returns the identifier with the result. *)
let verify_commands (tagged_commands : (int * User_command.Verifiable.t With_status.t) list) :
    (int * verification_result) list Deferred.t =
  ...
  
let reinject_valid_user_command_into_valid_result command result =
  match result with
  | #invalid as invalid ->
    invalid (* Directly return invalid results *)
  | `Valid_assuming x ->
    `Valid_assuming x
  | `Valid -> ... (* reinject command into [`Valid command] *)

(* In the parent process, use the IDs to match results to commands. *)
let reassociate_commands_with_results tagged_commands results =
  let result_map = Int.Map.of_alist_exn results in
  List.map tagged_commands ~f:(fun (id, command) ->
    match Map.find result_map id with
    | Some result -> reinject_valid_user_command_into_valid_result command result
    | None -> failwith "Verification result missing for command"
  ) ...

[tizoc]

Pushed a new commit that implements tagging in a similar way to what was described in the previous comment. If you don't think it is necessary or that there is a better solution I will revert it.

[tizoc]

!ci-build-me

[nholland94]

LGTM. The id tagging isn't a perfect guarantee, but it's better for nothing, and we can consider a more robust solution in the future if we end up having more complicated verifier control flow.

[deepthiskumar]

!ci-build-me

[deepthiskumar]

!ci-nightly-me

[deepthiskumar]

!approved-for-mainnet

[tizoc]

!ci-build-me

[tizoc]

@dkijania just rebased and triggered a new CI run

[dkijania]

We tested changes on our tailored pipeline for forks (unfortunately status is not reported here):

https://buildkite.com/o-1-labs-2/open-mina-pipeline/builds/54

There are 4 failures:

• Check merges cleanly to berkeley - due to conflicts between rampup and berkeley branch
• Check merges cleanly to develop - due to conflicts between rampup and develop branch
• Delegation backed unit - test was disabled on berkeley as it test obsoleted components, but this change was not merged to rampup
• Version type linter - problem with git, kind of mistery for me. I'm still debugging the problem, but i ran tests manually on buildkite agent and it passed

TL;DR:

After we clean up rampup branch this is good to be merged from CI perspective. No action needed from openmina team

[deepthiskumar]

!ci-build-me

[deepthiskumar]

!approved-for-mainnet