feat: witness-merging

[dexterslabor]

Description

This pull request introduces a significant feature for the Contract API and Virtual Storage called "Witness Merging." The primary goal of this feature is to eliminate the need for light clients to fetch the entire MerkleMap state, in order to generate necessary witnesses on-demand for executing zkApp transactions/computations. This implementation is finalized for a local testing mode. Future work will integrate with the ZKFS node.

Background and Context

Previously, clients had to fetch the entire Merkle Map state to create necessary witnesses for zkApp transactions/computations. This process was resource-intensive and inefficient, particularly for clients with limited bandwidth.

It is important to understand how Merkle witnesses work in this context. Suppose we have two values and their corresponding witnesses:

• value1 with witness1
• value2 with witness2

These witnesses allow us to compute the same Merkle root, R.

Now, let's suppose value1 is updated to value1'.

• With value1' and witness1, we can compute a new root R'.

However, witness2 becomes outdated because some values (aka siblings) required to compute the root R' depend on the updated value1'. To address this issue, it is necessary to retrieve the updated witness from an updated MerkleMap that reflects the change in value1'. The issue is that a light client would need to fetch the whole MerkleMap, which is especially resource intensive if multiple maps are fetched (nested maps).

Initial State:
--------------
value1 --witness1--> R
value2 --witness2--> R

Update value1 to value1':
-------------------------
value1' --witness1--> R'
value2  --witness2--> R (outdated)

Retrieve updated witness for value2:
------------------------------------
value1' --witness1--> R'
value2  --updated_witness2'--> R'

Solution: Witness Merging

Witness Merging is a novel technique that allows clients to update witnesses on-demand without the need to fetch the entire MerkleMap state. For this, it is necessary to keep track of last updated values and their witnesses. The Witness Merging algorithm takes witness1, witness2, and value1' as inputs and generates an updated_witness2', which computes the new root R' with value2. This eliminates the need for clients to access the whole Merkle Map to generate updated witnesses.

Changes

This PR includes the following major changes:

1. Virtual Storage: The Witness Merging algorithm is integrated into the Virtual Storage module. This algorithm is responsible for merging witnesses on-demand, enabling clients to generate updated witnesses without needing to access the entire MerkleMap state.
2. Contract API: The lastUpdatedState is tracked within the Contract API to determine which updated values and their corresponding witnesses need to be used for updating the witness currently in use. In addition to this, the last updated state is also backed up across transactions, mirroring the approach taken for virtual storage backups.

Performance Impact

A Merkle witness is significantly smaller and most certainly negligible in size compared to a Merkle tree of height 255. This translates to less network activity and reduced resource consumption for ZKFS light clients and ZKFS writer nodes.

Future Work

While this PR implements the core functionality of Witness Merging, there needs to be an integration with the ZKFS node/light client in order to manage off-chain state fetching and any configuration between testing locally or with a live Mina network.

Reviewers

Please review the changes in this PR, focusing on the following aspects:

1. Correctness and security of the new Witness Merging algorithm.
2. Quality and completeness of the tests.