Add commit storing

src/app/zeko/sequencer/README.md

@@ -0,0 +1,154 @@
+# Zeko sequencer
+
+## Zeko's Transaction Powerhouse
+
+Think of the sequencer as the conductor of an orchestra in Zeko. It plays a vital role in keeping everything running smoothly. Here's what it does:
+
+1. **Transaction Collector**: The sequencer acts like a tireless collector, gathering these transactions and applying them to the current state.
+2. **Proof Verifier**: The sequencer prooves the validity of these transactions using zero-knowledge proofs. This ensures only legitimate transactions enter the system.
+3. **Batch Processor**: The sequencer doesn't process transactions one by one. Instead, it efficiently groups them into batches for more efficient settling on layer 1.
+4. **Layer 1 Bridge**: Once a batch is ready, the sequencer sends it to the main chain (Layer 1) via a smart contract. This keeps Layer 1 informed about the activity happening on Zeko.
+
+## Build
+
+```bash
+DUNE_PROFILE=devnet dune build
+```
+
+## Tests
+
+```bash
+# Run local network to imitate L1
+DUNE_PROFILE=devnet dune exec ./tests/local_network/run.exe -- --db-dir l1_db
+
+DUNE_PROFILE=devnet dune runtest
+```
+
+## Run
+
+Running the sequencer exposes the Graphql API on the port `-p`. The Graphql schema is a subset of the L1 Graphql API joined with the L1 Graphql API for fetching of actions/events.
+
+```bash
+export DA_PROVIDER="da evm provider"
+export DA_PRIVATE_KEY="da layer private key"
+export MINA_PRIVATE_KEY="base58 signer private key"
+export DUNE_PROFILE=devnet
+dune exec ./run.exe -- \
+    -p <int?> \
+    --rest-server <string> \
+    --zkapp-pk <string> \
+    --max-pool-size <int?> \
+    --commitment-period <float?> \
+    --da-contract-address <string?> \
+    --db-dir <string?> \
+    --test-accounts-path <string?>
+```
+
+Run help to see the options:
+
+```bash
+dune exec ./run.exe -- --help
+```
+
+## Deploy rollup contract to L1
+
+The following script deploys the rollup contract on the L1 with the initial state, which is the genesis ledger of the rollup.
+
+```bash
+export MINA_PRIVATE_KEY="base58 signer private key"
+export DUNE_PROFILE=devnet
+dune exec ./deploy.exe -- \
+    --rest-server <string>
+    --test-accounts-path <string?>
+```
+
+Run help to see the options:
+
+```bash
+dune exec ./deploy.exe -- --help
+```
+
+## Using archive node as indexer
+
+Archive node is used for mina blockchain to index the history of the blockchain. You can optionally run the archive alongside the node's daemon, which dispatches new blocks to the archive. In Zeko rollup the blockcreator is the sequencer, and since currently it's not possible to run multiple sequencers, you need to run the client that subscribes to the sequencer and dispatches the new blocks to the archive.
+
+To use standard mina archive node to index the history of Zeko rollup, you need to run the Zeko archive relay adapter, that can subscribe to Zeko sequencer for new changes and relay them to the archive node.
+You can run the adapter with the following command:
+
+```bash
+export DUNE_PROFILE=devnet
+dune exec ./archive_relay/run.exe -- \
+    --zeko-uri <string> \
+    --archive-host <string> \
+    --archive-port <int> \
+    --bootstrap
+```
+
+To run the adapter from docker see the section below.
+
+## Use with docker
+
+Build:
+
+```bash
+make docker
+```
+
+Run:
+
+```bash
+docker run -p <port>:<port> \
+           -v <local-db-path>:<container-db-path> \
+           -e DA_PROVIDER=<da-evm-provider> \
+           -e DA_PRIVATE_KEY=<da-private-key> \
+           -e MINA_PRIVATE_KEY=<mina-private-key> \
+           dcspark/zeko -p <port> \
+           --zkapp-pk <zkapp-pk> \
+           --rest-server <mina-node-graphql> \
+           --archive-uri <mina-archive-node-graphql> \
+           --commitment-period <int> \
+           --da-contract-address <da-layer-contract> \
+           --db-dir <container-db-path>
+```
+
+### Running archive relay adapter from docker
+
+```bash
+docker run --entrypoint archive_relay \
+           dcspark/zeko
+           --zeko-uri <string> \
+           --archive-host <string> \
+           --archive-port <int> \
+           --bootstrap
+```
+
+## Manual commit
+
+In case automatic commit transaction fails and sequencer gets in the unrecoverable state, `committer.exe` can be used to manually resend commit transactions.
+
+To list all available commit transactions:
+
+```bash
+export DUNE_PROFILE=devnet
+dune exec ./committer.exe -- list
+```
+
+To get json of the commit transaction:
+
+```bash
+export DUNE_PROFILE=devnet
+dune exec ./committer.exe -- get --source <source-ledger-hash> --target <target-ledger-hash>
+```
+
+To resend the commit transaction:
+
+```bash
+export DUNE_PROFILE=devnet
+export MINA_PRIVATE_KEY="base58 signer private key"
+dune exec ./committer.exe -- send
+    --source <source-ledger-hash> \
+    --target <target-ledger-hash> \
+    --l1-uri <l1-uri> \
+    --fee <fee> \
+    --nonce <optional-nonce>
+```

src/app/zeko/sequencer/committer.ml

@@ -0,0 +1,115 @@
+open Core
+open Async
+open Sequencer_lib
+open Mina_base
+
+let printf = Core.printf
+
+let list =
+  ( "list"
+  , Command.basic ~summary:"List all of the transactions in the database"
+      (let%map_open.Command db_dir =
+         flag "--db-dir"
+           (optional_with_default "db" string)
+           ~doc:"string Directory to store the database"
+       in
+       fun () ->
+         let kvdb = Kvdb.of_dir db_dir in
+         let indices = Executor.Commits_store.get_index kvdb in
+         printf "Found %d transactions\n%!" (List.length indices) ;
+         List.iter indices ~f:(fun (source, target) ->
+             printf "Source: %s\nTarget: %s\n\n%!"
+               (Frozen_ledger_hash.to_decimal_string source)
+               (Frozen_ledger_hash.to_decimal_string target) ) ) )
+
+let get =
+  ( "get"
+  , Command.basic ~summary:"Find the command with the given source and target"
+      (let%map_open.Command db_dir =
+         flag "--db-dir"
+           (optional_with_default "db" string)
+           ~doc:"string Directory to store the database"
+       and source =
+         flag "--source" (required string)
+           ~doc:"string The source ledger of the transaction"
+       and target =
+         flag "--target" (required string)
+           ~doc:"string The target ledger of the transaction"
+       in
+       fun () ->
+         let kvdb = Kvdb.of_dir db_dir in
+         match
+           Executor.Commits_store.get_commit kvdb
+             ~source:(Frozen_ledger_hash.of_decimal_string source)
+             ~target:(Frozen_ledger_hash.of_decimal_string target)
+         with
+         | Some commit ->
+             print_endline
+               (Yojson.Safe.pretty_to_string @@ Zkapp_command.to_yojson commit)
+         | None ->
+             printf "No commit found\n%!" ) )
+
+let send =
+  ( "send"
+  , Command.basic ~summary:"Send a transaction to the sequencer"
+      (let%map_open.Command db_dir =
+         flag "--db-dir"
+           (optional_with_default "db" string)
+           ~doc:"string Directory to store the database"
+       and l1_uri = flag "--l1-uri" (required string) ~doc:"string L1 URI"
+       and fee_arg =
+         flag "--fee" (required int) ~doc:"int The fee amount in nanomina"
+       and nonce_opt =
+         flag "--nonce" (optional int)
+           ~doc:"int Will be fetched from the network if not provided"
+       and source =
+         flag "--source" (required string)
+           ~doc:"string The source ledger of the transaction"
+       and target =
+         flag "--target" (required string)
+           ~doc:"string The target ledger of the transaction"
+       in
+       let signer =
+         Signature_lib.(
+           Keypair.of_private_key_exn @@ Private_key.of_base58_check_exn
+           @@ Sys.getenv_exn "MINA_PRIVATE_KEY")
+       in
+       let l1_uri : Uri.t Cli_lib.Flag.Types.with_name =
+         Cli_lib.Flag.Types.{ value = Uri.of_string l1_uri; name = "l1-uri" }
+       in
+       let kvdb = Kvdb.of_dir db_dir in
+       fun () ->
+         Thread_safe.block_on_async_exn (fun () ->
+             let command =
+               Option.value_exn ~message:"No commit found"
+               @@ Executor.Commits_store.get_commit kvdb
+                    ~source:(Frozen_ledger_hash.of_decimal_string source)
+                    ~target:(Frozen_ledger_hash.of_decimal_string target)
+             in
+             let%bind nonce =
+               match nonce_opt with
+               | Some nonce ->
+                   return nonce
+               | None ->
+                   Sequencer_lib.Gql_client.fetch_nonce l1_uri
+                     (Signature_lib.Public_key.compress signer.public_key)
+             in
+             let executor = Executor.create ~nonce ~l1_uri ~signer ~kvdb () in
+             let command =
+               Zkapp_command.
+                 { command with
+                   fee_payer =
+                     { command.fee_payer with
+                       body =
+                         { command.fee_payer.body with
+                           fee = Currency.Fee.of_nanomina_int_exn fee_arg
+                         }
+                     }
+                 }
+             in
+             Executor.send_zkapp_command executor command ) ) )
+
+let () =
+  Command.group ~summary:"Script to manually send commiting transactions to L1"
+    [ list; get; send ]
+  |> Command_unix.run

src/app/zeko/sequencer/dune

@@ -1,4 +1,7 @@
-(env (_ (flags (-w -4-48))))
+(env
+ (_
+  (flags
+   (-w -4-48))))
 
 (executable
  (name run)
@@ -56,3 +59,26 @@
  (preprocess
   (pps ppx_jane))
  (modules deploy))
+
+(executable
+ (name committer)
+ (libraries
+  sequencer_lib
+  ;; mina ;;
+  init
+  signature_lib
+  cli_lib
+  ;; opam libraries ;;
+  cohttp
+  cohttp-async
+  graphql
+  graphql-async
+  core
+  core_kernel
+  async_kernel
+  async
+  async_unix
+  core_unix.command_unix)
+ (preprocess
+  (pps ppx_jane))
+ (modules committer))

src/app/zeko/sequencer/lib/archive.ml

@@ -8,7 +8,6 @@ open Mina_transaction
 open Sexplib.Std
 open Snark_params
 open Key_value_database.Monad.Ident.Let_syntax
-module Db = Mina_ledger.Ledger.Kvdb
 
 let ok_exn x =
   let open Ppx_deriving_yojson_runtime.Result in
@@ -93,7 +92,7 @@ module Account_update_actions = struct
 end
 
 module Archive = struct
-  type t = Db.t
+  type t = Kvdb.t
 
   let create ~kvdb : t = kvdb
 
@@ -105,10 +104,7 @@ module Archive = struct
       ^ Token_id.to_string (Account_id.token_id account_id) )
 
   let query_events t account_id =
-    let%bind events =
-      Db.get t
-        ~key:Bigstring.(concat [ of_string "events"; serialize_key account_id ])
-    in
+    let%bind events = Kvdb.get t ~key:(EVENTS account_id) in
     let events =
       Option.value ~default:"[]" @@ Option.map ~f:Bigstring.to_string events
     in
@@ -117,11 +113,7 @@ module Archive = struct
       Yojson.Safe.(Util.to_list @@ from_string events)
 
   let query_actions t account_id =
-    let%bind actions =
-      Db.get t
-        ~key:
-          Bigstring.(concat [ of_string "actions"; serialize_key account_id ])
-    in
+    let%bind actions = Kvdb.get t ~key:(ACTIONS account_id) in
     let actions =
       Option.value ~default:"[]" @@ Option.map ~f:Bigstring.to_string actions
     in
@@ -135,9 +127,7 @@ module Archive = struct
       @@ `List (List.map ~f:Account_update_events.to_yojson events)
     in
     let%bind () =
-      Db.set t
-        ~key:Bigstring.(concat [ of_string "events"; serialize_key account_id ])
-        ~data:(Bigstring.of_string events)
+      Kvdb.set t ~key:(EVENTS account_id) ~data:(Bigstring.of_string events)
     in
     ()
 
@@ -147,10 +137,7 @@ module Archive = struct
       @@ `List (List.map ~f:Account_update_actions.to_yojson actions)
     in
     let%bind () =
-      Db.set t
-        ~key:
-          Bigstring.(concat [ of_string "actions"; serialize_key account_id ])
-        ~data:(Bigstring.of_string actions)
+      Kvdb.set t ~key:(ACTIONS account_id) ~data:(Bigstring.of_string actions)
     in
     ()