Address user-reported secret leak, RPC proxy retry, and watchdog container issues

README.md

@@ -98,7 +98,7 @@ nonce on demand (keyed operator tool).
 `setup` requires: `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`, `CARTESI_SEQUENCER_BLOCKCHAIN_ID`, `CARTESI_SEQUENCER_APP_ADDRESS`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS`.
 `run` requires: `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`, `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` (or `_FILE`); it refuses to boot until `setup` has completed.
 
-Optional: `CARTESI_SEQUENCER_HTTP_ADDR` (default `127.0.0.1:3000`, `run`), `CARTESI_SEQUENCER_DATA_DIR` (default `sequencer-data` — SQLite file is `sequencer.db` inside; created if missing), `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS` (default `300`), `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` (default `12`), `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS` (default `600`), `CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES` (default `-32005,-32600,-32602,-32616`), `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE` (alternative to `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY`; first line of the file is the key), `CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH`.
+Optional: `CARTESI_SEQUENCER_HTTP_ADDR` (default `127.0.0.1:3000`, `run`), `CARTESI_SEQUENCER_DATA_DIR` (default `sequencer-data` — SQLite file is `sequencer.db` inside; created if missing), `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS` (default `300`), `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` (default `12`), `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS` (default `600`), `CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES` (default `-32005,-32012,-32600,-32602,-32616`), `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE` (alternative to `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY`; first line of the file is the key), `CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH`.
 
 By default the blockchain endpoint must be `https://` unless its host is loopback (`localhost`, `127.0.0.0/8`, `::1`) — a guard against accidentally sending L1 traffic to a public RPC in the clear. Set `CARTESI_SEQUENCER_ALLOW_INSECURE_RPC=true` (or `--allow-insecure-rpc`) to permit plaintext `http://` to a non-loopback host on a **trusted private network** — e.g. a Docker Compose / Kubernetes service name (`http://anvil:8545`), `host.docker.internal`, or a private-VPC IP.
 
@@ -146,6 +146,7 @@ Notes:
 - payload size is bounded at ingress; oversized requests are rejected before entering the hot path.
 - overload is enforced at queue admission: if the inclusion-lane queue is full, `POST /tx` returns HTTP `429` with code `OVERLOADED` and message `queue full`.
 - queue capacity is an internal runtime constant tuned alongside inclusion-lane chunking to absorb short bursts; if this starts triggering persistently, it is a signal to revisit runtime sizing or throughput rather than add another admission layer.
+- CORS is currently **permissive** (`Access-Control-Allow-Origin: *`, all methods/headers) so browser wallets can call `POST /tx`. Tighten once the planned ingress/egress port split lands.
 
 ### `GET /ws/subscribe?from_offset=<u64>`
 

docs/watchdog/README.md

@@ -182,7 +182,7 @@ the chain id explicitly).
 The watchdog has two subcommands:
 
 ```bash
-sequencer-watchdog init   # one-time setup: writes config.json + head.json
+sequencer-watchdog init   # setup: writes config.json + head.json (idempotent if complete)
 sequencer-watchdog tick   # one compare cycle; schedule this
 ```
 

docs/watchdog/operator-deployment.md

@@ -160,6 +160,7 @@ Today `WalletApp::default()` / `WalletConfig::sepolia()` align with Sepolia stag
 | `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | Chain id label for `status.prom` metrics (prefer set at `init`; optional auto-detect via `eth_chainId` when L1 endpoint is present at `init`) |
 | `CARTESI_WATCHDOG_METRICS_FILE` | Override path for the Prometheus textfile written by each `tick` |
 | `CARTESI_WATCHDOG_LUA_DEPS` | `.deps/lua` |
+| `CARTESI_WATCHDOG_LONG_BLOCK_RANGE_ERROR_CODES` | Optional CSV of RPC error codes that trigger `eth_getLogs` partition retry. **Evaluated only at `init` and persisted in `config.json`** — not a tick-time override; re-running idempotent `init` does not refresh it. Wipe state and re-init (or edit `config.json`) to change. Default matches the sequencer: `-32005,-32012,-32600,-32602,-32616`. |
 
 The sequencer discovers and pins `input_box_address` at startup; use the same values as `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT` / `CARTESI_SEQUENCER_APP_ADDRESS` configuration.
 
@@ -174,7 +175,12 @@ Pick one:
 3. **Replay from genesis** (only for new rollups / low block height — slow).
 
 Run `init` once to store the bootstrap CM snapshot into the watchdog state
-layout. The L1 RPC URL is not persisted — each `tick` reads
+layout. Re-running `init` on a **complete** already-initialized state directory
+is a no-op success (exit `0`), matching `sequencer setup` — safe for process
+supervisors that always invoke init before tick. If `head.json` exists but
+`config.json` or the selected snapshot is missing/corrupt, `init` fails (exit
+`1`) and asks you to wipe `state_dir` and re-run — it will not certify an
+unusable state. The L1 RPC URL is not persisted — each `tick` reads
 `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` so it can rotate without editing
 state. If `CARTESI_WATCHDOG_BLOCKCHAIN_ID` is unset at `init`, auto-detect also
 needs that endpoint present then (prefer setting the chain id explicitly):

sequencer/Cargo.toml

@@ -18,7 +18,7 @@ serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 toml = "0.8"
 tracing = "0.1"
-tower-http = { version = "0.6.8", features = ["trace"] }
+tower-http = { version = "0.6.8", features = ["trace", "cors"] }
 rusqlite = { version = "0.38.0", features = ["bundled"] }
 rusqlite_migration = "2.3.0"
 alloy-primitives = { version = "1.4.1", features = ["serde", "k256"] }

sequencer/src/http.rs

@@ -21,6 +21,7 @@ use axum::response::{IntoResponse, Response};
 use serde::Serialize;
 use thiserror::Error;
 use tokio::sync::mpsc;
+use tower_http::cors::CorsLayer;
 use tower_http::trace::TraceLayer;
 
 pub use crate::egress::api::SnapshotState;
@@ -228,7 +229,10 @@ pub fn start_on_listener(
         ))
         // Enforces a raw request-body cap before JSON deserialization, including whitespace.
         .layer(DefaultBodyLimit::max(config.max_body_bytes))
-        .layer(TraceLayer::new_for_http());
+        .layer(TraceLayer::new_for_http())
+        // Permissive CORS so browser wallets can POST /tx (and preflight OPTIONS).
+        // Tighten when the ingress/egress port split lands and public exposure is narrower.
+        .layer(CorsLayer::permissive());
 
     tokio::spawn(async move {
         axum::serve(listener, app)

sequencer/src/l1/partition.rs

@@ -8,8 +8,22 @@
 //! This module is stateless: callers pass the retry error codes explicitly. There is no
 //! global mutable state; `RunConfig` owns the codes and passes them down via configs.
 
-/// Default RPC error codes that trigger partition retry (e.g. Infura -32005, Alchemy -32600/-32602, QuickNode -32616).
-pub const DEFAULT_LONG_BLOCK_RANGE_ERROR_CODES: &[&str] = &["-32005", "-32600", "-32602", "-32616"];
+/// Default RPC error codes that trigger partition retry (e.g. Infura -32005,
+/// Alchemy -32600/-32602, QuickNode -32616, Goldsky / some JSON-RPC proxies
+/// -32012).
+///
+/// **Overloaded codes (accepted).** `-32005` and `-32012` are not range-only
+/// across providers: Alloy's transport also treats bare `-32005` as Infura
+/// rate-limit, and `-32012` + `"credits"` as QuickNode credit rate-limit.
+/// We still match on the bare code — the same policy already used for
+/// `-32005` — because range-error messages are not stable across proxies
+/// (Goldsky vs custom gateways). A persistent QuickNode credits limit that
+/// survives Alloy's own retries can therefore fan an N-block scan into
+/// `2N−1` partition queries before failing; operators on QuickNode can drop
+/// `-32012` from `CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES` if that
+/// matters. We deliberately do **not** add message-marker heuristics here.
+pub const DEFAULT_LONG_BLOCK_RANGE_ERROR_CODES: &[&str] =
+    &["-32005", "-32012", "-32600", "-32602", "-32616"];
 
 use alloy::contract::Error as ContractError;
 use alloy::contract::Event;
@@ -373,6 +387,27 @@ mod tests {
         assert!(!error_message_matches_retry_codes("ok", &[]));
     }
 
+    #[test]
+    fn default_codes_include_proxy_range_exceeded() {
+        assert!(
+            DEFAULT_LONG_BLOCK_RANGE_ERROR_CODES.contains(&"-32012"),
+            "default codes must include -32012 (JSON-RPC proxy range limit)"
+        );
+    }
+
+    #[test]
+    fn proxy_range_exceeded_error_triggers_retry() {
+        let msg = r#"-32012: getLogs request exceeded max allowed range, data: {"code":"ErrGetLogsExceededMaxAllowedRange"}"#;
+        let codes: Vec<String> = DEFAULT_LONG_BLOCK_RANGE_ERROR_CODES
+            .iter()
+            .map(|c| (*c).to_string())
+            .collect();
+        assert!(
+            error_message_matches_retry_codes(msg, &codes),
+            "proxy -32012 error must match retry codes"
+        );
+    }
+
     #[test]
     fn decode_evm_advance_input_round_trips() {
         let encoded = EvmAdvanceCall {

sequencer/src/runtime/config.rs

@@ -116,13 +116,15 @@ pub struct KeyArgs {
     #[arg(
         long,
         env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY",
+        hide_env_values = true,
         group = "batch_submitter_key_source"
     )]
     batch_submitter_private_key: Option<String>,
     /// Path to a file whose first line contains the batch submitter private key.
     #[arg(
         long,
         env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE",
+        hide_env_values = true,
         group = "batch_submitter_key_source"
     )]
     batch_submitter_private_key_file: Option<String>,
@@ -167,10 +169,18 @@ fn resolve_key_source(
 #[derive(Debug, Clone, Args)]
 pub struct OptionalKeyArgs {
     /// Hex-encoded batch-submitter private key.
-    #[arg(long, env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY")]
+    #[arg(
+        long,
+        env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY",
+        hide_env_values = true
+    )]
     batch_submitter_private_key: Option<String>,
     /// Path to a file whose first line is the batch-submitter private key.
-    #[arg(long, env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE")]
+    #[arg(
+        long,
+        env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE",
+        hide_env_values = true
+    )]
     batch_submitter_private_key_file: Option<String>,
 }
 
@@ -204,7 +214,7 @@ impl OptionalKeyArgs {
 pub struct SetupConfig {
     #[arg(long, env = "CARTESI_SEQUENCER_DATA_DIR", default_value = DEFAULT_DATA_DIR, value_parser = parse_non_empty_string)]
     pub data_dir: String,
-    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", value_parser = parse_non_empty_string)]
+    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", hide_env_values = true, value_parser = parse_non_empty_string)]
     pub eth_rpc_url: String,
     /// Allow plaintext (`http://`) RPC to a non-loopback host — a trusted
     /// private network (Docker/K8s service name, `host.docker.internal`, a
@@ -330,7 +340,7 @@ pub struct RunConfig {
     pub http_addr: String,
     #[arg(long, env = "CARTESI_SEQUENCER_DATA_DIR", default_value = DEFAULT_DATA_DIR, value_parser = parse_non_empty_string)]
     pub data_dir: String,
-    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", value_parser = parse_non_empty_string)]
+    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", hide_env_values = true, value_parser = parse_non_empty_string)]
     pub eth_rpc_url: String,
     /// Allow plaintext (`http://`) RPC to a non-loopback host — a trusted
     /// private network (Docker/K8s service name, `host.docker.internal`, a
@@ -411,7 +421,7 @@ impl RunConfig {
 pub struct FlushConfig {
     #[arg(long, env = "CARTESI_SEQUENCER_DATA_DIR", default_value = DEFAULT_DATA_DIR, value_parser = parse_non_empty_string)]
     pub data_dir: String,
-    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", value_parser = parse_non_empty_string)]
+    #[arg(long, env = "CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", hide_env_values = true, value_parser = parse_non_empty_string)]
     pub eth_rpc_url: String,
     /// Allow plaintext (`http://`) RPC to a non-loopback host — a trusted
     /// private network (Docker/K8s service name, `host.docker.internal`, a
@@ -664,6 +674,7 @@ mod tests {
             config.long_block_range_error_codes,
             vec![
                 "-32005".to_string(),
+                "-32012".to_string(),
                 "-32600".to_string(),
                 "-32602".to_string(),
                 "-32616".to_string()
@@ -727,6 +738,91 @@ mod tests {
         );
     }
 
+    /// Render `--help` for `subcmd` inside a child process whose env carries
+    /// the given secret values. Mutating this process's env is unsound under
+    /// parallel `cargo test` (and neighbouring Clap parses race on the same
+    /// vars), so the probe runs via `Command::env` on a re-exec of this
+    /// test binary.
+    fn help_text_with_secret_env(subcmd: &str, secrets: &[(&str, &str)]) -> String {
+        let out_path = std::env::temp_dir().join(format!(
+            "sequencer-help-leak-{}-{}-{}.txt",
+            subcmd,
+            std::process::id(),
+            std::time::SystemTime::now()
+                .duration_since(std::time::UNIX_EPOCH)
+                .expect("clock")
+                .as_nanos()
+        ));
+
+        let exe = std::env::current_exe().expect("current_exe");
+        let mut cmd = std::process::Command::new(&exe);
+        cmd.args([
+            "--exact",
+            "runtime::config::tests::help_does_not_leak_private_key_value",
+            "--quiet",
+        ])
+        .env("SEQUENCER_HELP_LEAK_OUT", &out_path)
+        .env("SEQUENCER_HELP_LEAK_SUBCMD", subcmd);
+        for (key, value) in secrets {
+            cmd.env(key, value);
+        }
+
+        let status = cmd.status().expect("spawn help-leak child");
+        assert!(
+            status.success(),
+            "help-leak child failed for subcommand {subcmd}: {status}"
+        );
+        let help = std::fs::read_to_string(&out_path)
+            .unwrap_or_else(|err| panic!("read help output {}: {err}", out_path.display()));
+        let _ = std::fs::remove_file(&out_path);
+        help
+    }
+
+    #[test]
+    fn help_does_not_leak_private_key_value() {
+        // Child arm: render help under the caller's env and write it out.
+        // Exits the process so we never nest another spawn.
+        if let Ok(out_path) = std::env::var("SEQUENCER_HELP_LEAK_OUT") {
+            let subcmd = std::env::var("SEQUENCER_HELP_LEAK_SUBCMD")
+                .expect("SEQUENCER_HELP_LEAK_SUBCMD must be set in the child");
+            let help = Cli::try_parse_from(["sequencer", subcmd.as_str(), "--help"])
+                .expect_err("--help must short-circuit as a clap error")
+                .to_string();
+            std::fs::write(&out_path, help).expect("write help output");
+            std::process::exit(0);
+        }
+
+        let sentinel_key = "0xSENTINEL_PRIVATE_KEY_DEADBEEF1234567890abcdef";
+        let sentinel_file = "/secret/path/key.pem";
+        let sentinel_rpc = "https://eth-mainnet.g.alchemy.com/v2/SECRET_TOKEN_XYZ";
+        let secrets = [
+            ("CARTESI_SEQUENCER_AUTH_PRIVATE_KEY", sentinel_key),
+            ("CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE", sentinel_file),
+            ("CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT", sentinel_rpc),
+        ];
+
+        for subcmd in ["run", "setup", "flush-mempool"] {
+            let help = help_text_with_secret_env(subcmd, &secrets);
+
+            assert!(
+                !help.contains(sentinel_key),
+                "{subcmd} --help must not print the private key value"
+            );
+            assert!(
+                !help.contains(sentinel_file),
+                "{subcmd} --help must not print the key-file path"
+            );
+            assert!(
+                !help.contains("SECRET_TOKEN_XYZ"),
+                "{subcmd} --help must not print the RPC URL (may contain API tokens)"
+            );
+            assert!(
+                help.contains("CARTESI_SEQUENCER_AUTH_PRIVATE_KEY"),
+                "{subcmd} --help should still mention the env var name"
+            );
+        }
+    }
+
     #[test]
     fn domain_uses_fixed_name_and_version() {
         use alloy_primitives::U256;

sequencer/tests/snapshot_endpoints.rs

@@ -522,3 +522,33 @@ async fn latest_snapshot_falls_back_to_finalized_when_no_pending() {
 
     assert!(wait_for_lease(db.path.as_str(), fin_id, 0).await);
 }
+
+#[tokio::test]
+async fn cors_permits_browser_preflight_on_tx() {
+    let db = temp_db("cors-preflight");
+    let Some(server) = start_server(db.path.as_str()).await else {
+        return;
+    };
+
+    let resp = reqwest::Client::new()
+        .request(reqwest::Method::OPTIONS, server.url("/tx"))
+        .header("Origin", "https://wallet.example")
+        .header("Access-Control-Request-Method", "POST")
+        .header("Access-Control-Request-Headers", "content-type")
+        .send()
+        .await
+        .expect("OPTIONS /tx");
+
+    assert!(
+        resp.status().is_success(),
+        "preflight status: {}",
+        resp.status()
+    );
+    let allow_origin = resp
+        .headers()
+        .get("access-control-allow-origin")
+        .expect("Access-Control-Allow-Origin")
+        .to_str()
+        .expect("header utf8");
+    assert_eq!(allow_origin, "*");
+}

tests/fixtures/l1_partition_vector.json

@@ -1,5 +1,5 @@
 {
-  "long_block_range_error_codes": ["-32005", "-32600", "-32602", "-32616"],
+  "long_block_range_error_codes": ["-32005", "-32012", "-32600", "-32602", "-32616"],
   "scenarios": [
     {
       "name": "bisect_on_default_infura_code",
@@ -65,6 +65,20 @@
       ],
       "expect_ok": true
     },
+    {
+      "name": "bisect_on_proxy_range_exceeded_code",
+      "start_block": 100,
+      "end_block": 200,
+      "fail_ranges": [
+        {
+          "from": 100,
+          "to": 200,
+          "message": "-32012: getLogs request exceeded max allowed range"
+        }
+      ],
+      "expect_calls": [[100, 200], [100, 150], [151, 200]],
+      "expect_ok": true
+    },
     {
       "name": "single_block_fail_no_split",
       "start_block": 7,

watchdog/config.lua

@@ -86,7 +86,7 @@ function config.load_init(env)
         retry_attempts = optional_number("CARTESI_WATCHDOG_RETRY_ATTEMPTS", 3, env),
         retry_delay_sec = optional_number("CARTESI_WATCHDOG_RETRY_DELAY_SEC", 5, env),
         long_block_range_error_codes = split_csv(
-            env.CARTESI_WATCHDOG_LONG_BLOCK_RANGE_ERROR_CODES or "-32005,-32600,-32602,-32616"
+            env.CARTESI_WATCHDOG_LONG_BLOCK_RANGE_ERROR_CODES or "-32005,-32012,-32600,-32602,-32616"
         ),
     }
 end
@@ -125,6 +125,21 @@ function config.persisted(cfg)
     }
 end
 
+--- Validate a persisted config.json object (no tick-time env required).
+--- Raises on missing/invalid fields; used by idempotent init before exit 0.
+function config.validate_persisted(data)
+    if type(data) ~= "table" then
+        error("config.json is not an object")
+    end
+    if data.version ~= config.VERSION then
+        error("unsupported config.json version: " .. tostring(data.version))
+    end
+    required_field(data, "sequencer_url")
+    required_field(data, "input_box_address")
+    required_field(data, "app_address")
+    return true
+end
+
 function config.from_persisted(state_dir, data, env)
     env = normalize_env(env)
     if data.version ~= config.VERSION then