Experimental offchain storage pt 1 #1630

mitschabaude · 2024-05-02T06:57:06Z

MVP design of an offchain state implementation.

This PR doesn't release the API yet. TBD if we want to implement update() and make the Merkle updates more efficient before release
Adds a new interface: OffchainState
See this working example for how to use it: https://github.com/o1-labs/o1js/blob/cda0e1f2489123e5acde0bdcd0ec64f6baddfa83/src/lib/mina/actions/offchain-contract.unit-test.ts

Design

There are currently two types of offchain state: OffchainState.Field (a single field of state) and OffchainState.Map (a key-value map).

All offchain state is stored in a single Merkle map (Merkle tree of size 256)
There are no practical limits to the number of state fields and maps
you can use (pure) provable types of size up to ~100 field elements (~size of an action) for field and map values. (Map keys have unlimited size, since they don't need to be part of the action.)
Fields support field.get() and field.set(value) in a contract
Maps support map.get(key) and map.set(key, value)

To use offchain state, a smart contract developer must

declare an OffchainState
call offchainState.compile() and offchainState.setContractInstance() in the setup phase
add a specific onchain state field manually. This might seem brittle, but is somewhat type-safe because setContractInstance() requires the contract to have that state field.
manually add a settle() method and call it periodically to settle state
- state is only available for get() after it was settled
- the settle() implementation is trivial using the tools OffchainState provides
- settling also involves calling createSettlementProof() outside the contract first, which is also simple from the user point of view

Some design decisions:

All info required to use offchain state is available from actions. No extra events or other external data store.
Currently, the entire merkle tree is recovered on the fly by each user, from fetched actions.
- Pro: there is no extra service to set up to distribute the merkle tree
- Con: This scales badly, especially with how naive the MVP implementation is
- Con: The slowness of the archive node is a major bottleneck to being able to settle and use state
- To me this seems like a good baseline design, to which more advanced storage and distribution mechanisms can be added later

Caveats / Future work

While writing the example, I realized that the pure set() API, which instructs offchain settlement to blindly overwrite a current value with the new one, is a major footgun and probably only useful for specific scenarios (e.g. each user of a contract is empowered to manage their own slice of state and overwrite it without taking into account old state. example: registration as a voter), but fails as a solution for the majority of use cases.

Q: Should we rename set() to overwrite() to be more honest about its limitations?

An extension that can be made easily is to add an update() method which not only takes the new state, but also the old state to replace. It only succeeds if the old state is correct, i.e. it acts exactly the same as a precondition for onchain state.

update() can be implemented such that it fails atomically on an entire account update. E.g., you make two state updates, and both fail if one of the old states is invalid.
- This lets you properly implement a transfer() method, for example. The method which dispatches the transfer will only dispatch valid pairs of to/from balance changes. If one of them is outdated, both fail, so there's no way to cause an invalid balance change.
- A variant of this, which reuses the implementation, would also enable map entries that can only be modified once. E.g. nullifiers. We just hard-code the "old state" hash to the initial 0 value

With update(), what we arrive at is a clean extension of the onchain state API to offchain state of arbitrary size. It completely solves the size problem. However, it doesn't fully solve the concurrency problem:

for state that is per-user, like individual account balances, concurrency is not a dealbreaker. A user has to sequence their own individual interactions - fine.
however, the current design does NOT solve the problem of shared state that is modified concurrently. A good example is the token supply in our example: The current design can only change supply (= mint or burn) one block at a time.

One idea to improve shared state handling is to introduce user-defined (or predefined) commutative actions. For example, an action could say "add this number to the state" instead of "change the state to this value".

This would solve the token supply use case example
I think this can be shipped as an extension to the current API, where commutative actions are additional inputs to the OffchainState.Field / OffchainState.Map declarations
the implementation is more involved and comes with an overhead because it means the reducer has to interact with state values, not just value hashes. it also has to disambiguate different state fields. (Efficiency-wise, both of those factors should be dominated by the Merkle hashing per action that is already done)

The final design should probably be more similar to protokit's runtime modules: A client/server like model where arbitrary computation is run in an offchain reducer to go from the previous to the next set of states, in one atomic update.

mitschabaude

Some other changes I forgot to mention in the description!

mitschabaude · 2024-05-14T09:56:37Z

src/index.ts

-  Reducer,
-} from './lib/mina/zkapp.js';
+export { SmartContract, method, declareMethods } from './lib/mina/zkapp.js';
+export { Reducer } from './lib/mina/actions/reducer.js';


moved Reducer code to a separate file, zkapp.ts is unreadable enough without it

mitschabaude · 2024-05-14T09:57:12Z

src/lib/mina/actions/reducer.ts

no new code here, just moved from zkapp.ts

mitschabaude · 2024-05-14T09:58:54Z

src/lib/mina/state.ts

+function State<A>(defaultValue?: A): State<A> {
+  return createState<A>(defaultValue);


State() now accepts passing in a default value, which is used to initialize it by init()!

this is very handy when there's no other reason to override init(), like in our offchain state example

mitschabaude · 2024-05-14T10:00:05Z

src/lib/mina/zkapp.ts

+    // for all explicitly declared states, set them to their default value
+    let stateKeys = getLayout(this.constructor as typeof SmartContract).keys();
+    for (let key of stateKeys) {
+      let state = this[key as keyof this] as InternalStateType<any> | undefined;
+      if (state !== undefined && state.defaultValue !== undefined) {
+        state.set(state.defaultValue);
+      }
+    }


new logic in init() that uses default values for all state fields that have one

mitschabaude · 2024-05-14T10:01:45Z

src/lib/provable/merkle-list.ts

+  /**
+   * Iterate through the list in a fixed number of steps any apply a given callback on each element.
+   *
+   * Proves that the iteration traverses the entire list.
+   * Once past the last element, dummy elements will be passed to the callback.
+   *
+   * Note: There are no guarantees about the contents of dummy elements, so the callback is expected
+   * to handle the `isDummy` flag separately.
+   */
+  forEach(
+    length: number,
+    callback: (element: T, isDummy: Bool, i: number) => void


API for iterating a MerkleList without having to explicitly instantiate an iterator

this is safer than .next() since it does assertAtEnd() for you

mitschabaude · 2024-05-14T10:02:49Z

src/lib/provable/merkle-tree.ts

This fixes two pain points I always had when dealing with MerkleTrees:

can't clone them

can't just get a leaf, need to figure out how to use getNode()

mitschabaude · 2024-05-14T10:03:12Z

src/lib/provable/option.ts

A nice new Option type!

mitschabaude · 2024-05-14T10:04:06Z

src/lib/provable/types/struct.ts

-  provable,
-  provablePure,


I always find re-exports confusing, removed them here

mitschabaude · 2024-05-14T10:06:38Z

src/lib/provable/types/unconstrained.ts

using Unconstrained as part of a MerkleList Struct element (ex: MerkleLeaf in this PR) used to fail in a very mysterious way, because the Merkle list attempted to use .empty() and Struct deeply walked the Unconstrained without finding anything.

These changes add a clear error message, and an easier way to declare the empty type alongside an Unconstrained type

Trivo25 · 2024-05-14T14:23:18Z

src/examples/simple-zkapp.ts

@@ -18,14 +18,13 @@ const doProofs = true;
 const beforeGenesis = UInt64.from(Date.now());

 class SimpleZkapp extends SmartContract {
-  @state(Field) x = State<Field>();
+  @state(Field) x = State(initialState);

  events = { update: Field, payout: UInt64, payoutReceiver: PublicKey };

  @method
  async init() {


We should be able to remove this method entirely now, I think?

no because this example uses the fact that it's a @method and affects provedState

Trivo25 · 2024-05-14T14:37:40Z

src/lib/mina/actions/offchain-state-rollup.ts

+      }
+
+      // base proof
+      console.time('batch 0');


note to remove these before we merge

@Trivo25 I removed it in the next PR, ok? #1652

mitschabaude added 30 commits May 2, 2024 08:56

move existing reducer code into a new folder

1d0bdde

remove provable reexport

941639d

for each method on merkle list

693f06b

scaffold out merkle map rollup

bf24822

some code golf

1f2bf0d

cloneable merkle tree

bd676ce

get leaf of merkle tree

d29d490

finish v0 of merkle map rollup

d9c3824

add outside proof logic

c5700ee

some comments with efficiency ideas

a1439c3

serialization and custom efficient hashing

8fc54e0

finish merkle leaf serialization

f5256c7

rename

5a6604c

start typing out stuff

23d926d

nice option type

f39e013

more api scaffolding

ca2dc19

example offchain state interaction

6021181

bindings

d442abf

minor

cd1e402

start implementation

98fb766

minor

65645b3

fix

d19e3d0

small fix

ecbce96

small fixes

246d2d6

probably fixes

9018c93

+ update API

ebf3e47

implement set()

7bc958d

don't use update() for now

80d66ab

change serialization and implement get()

bc2893d

settling

e00fb47

mitschabaude added 11 commits May 13, 2024 16:25

support onchain state default values

bd966c6

set state fields explicitly

4f525ba

Merge branch 'main' into feature/experimental-offchain-state

7a10b49

fix how we get contract

5ead7a5

improve unconstrained + empty

0c51655

refactor to two separate program methods for now

75c6dd2

disable proofs

1e5b5a3

suprisingly non mutating api?

e9df987

use merkle tree, not map (set leaf is different)

fc7e633

make unit test more test-like

d7502aa

fixup contracts without state

cda0e1f

mitschabaude marked this pull request as ready for review May 14, 2024 06:17

mitschabaude mentioned this pull request May 14, 2024

Experimental offchain storage pt 1 o1-labs/o1js-bindings#272

Merged

mitschabaude commented May 14, 2024

View reviewed changes

mitschabaude added 4 commits May 14, 2024 12:12

some boyscouting

44d5616

export option type

0df6f4e

option docs, initial state example

744251e

changelog

28a4e69

Trivo25 reviewed May 14, 2024

View reviewed changes

mitschabaude mentioned this pull request May 14, 2024

Experimental offchain state, pt 2 #1652

Merged

1 task

Trivo25 approved these changes May 15, 2024

View reviewed changes

mitschabaude merged commit 89c59c7 into main May 15, 2024
14 checks passed

mitschabaude deleted the feature/experimental-offchain-state branch May 15, 2024 08:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental offchain storage pt 1 #1630

Experimental offchain storage pt 1 #1630

mitschabaude commented May 2, 2024 •

edited

mitschabaude left a comment

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

mitschabaude May 14, 2024

Trivo25 May 14, 2024

mitschabaude May 14, 2024 •

edited

Trivo25 May 14, 2024

mitschabaude May 14, 2024

		function State<A>(defaultValue?: A): State<A> {
		return createState<A>(defaultValue);

Experimental offchain storage pt 1 #1630

Experimental offchain storage pt 1 #1630

Conversation

mitschabaude commented May 2, 2024 • edited

Design

Caveats / Future work

mitschabaude left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude May 14, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitschabaude commented May 2, 2024 •

edited

mitschabaude May 14, 2024 •

edited