test(cketh): end-to-end balance scan against a live anvil node

[gregorydemay]

Adds a single end-to-end test proving the ckERC20 balance scan works against a real EVM, complementing the existing mock-based integration tests.

The test drives the full production path — minter → real EVM RPC canister → Ethereum — with no JSON-RPC mocking:

• A new PocketIC live harness in the cketh test_utils crate owns a local anvil node and installs the minter and the EVM RPC canister, configuring the latter with an overrideProvider that routes every provider to that node. In live mode the EVM RPC canister issues genuine HTTPS outcalls that reach anvil for real. The new harness is required because current integration tests infrastructure uses the StateMachine that cannot issue real HTTP request. Migrating the integration test infrastructure to PocketIC is deferred to DEFI-2262 (see test(ledger-suite-orchestrator): migrate integration tests to PocketIC #10949).
• Supported tokens (ckUSDC, ckUSDT) are registered directly through the minter's add_ckerc20_token endpoint by pointing its orchestrator id at a principal the harness controls, so no real orchestrator or spawned ledgers are needed.
• The harness places the two tokens at their real mainnet addresses on anvil (anvil_setCode) and credits the minter's derived deposit address above the candidate threshold (anvil_setStorageAt). The test then asserts the minter's periodic scan reads those real balances and flags the address as a deposit candidate for both tokens.

Stacked on top of #10873.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to b508bce.

Security Overview

• 🔎 Scanned files: 8 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/783a9bd4-fb4e-4e10-ab3c-c2fb2ddfb92f?scanId=b8e91870-6a48-43d9-9363-de1be6a26f2e&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/ethereum/cketh/minter/BUILD.bazel     Rename deposit_from_cex_test to deposit_from_cex and increase test timeout; add new test dependencies and environment variables ► rs/ethereum/cketh/minter/tests/deposit_from_cex.rs     Update test to reflect new live-end-to-end balance-scan approach and interface changes
Enhancement	► rs/ethereum/cketh/test_utils/BUILD.bazel     Add pocket-ic dependency and expose reqwest in test utils ► rs/ethereum/cketh/test_utils/Cargo.toml     Add pocket-ic and reqwest dependencies
Enhancement	► rs/ethereum/cketh/test_utils/src/anvil.rs (new file)     Introduce anvil-based test utilities with RPC, deployment, and balance helpers
Enhancement	► rs/ethereum/cketh/test_utils/src/lib.rs     Expose new modules including anvil and live_scan
Enhancement	► rs/ethereum/cketh/test_utils/src/live_scan.rs (new file)     Add live scan harness for ckERC20 balance scanning using PocketIC and real Anvil
Enhancement	► rs/ethereum/cketh/test_utils/src/ (various files within new live_scan harness)

[mbjorkqvist]

Thanks @gregorydemay!

[Copilot]

🟡 Not ready to approve

The new anvil JSON-RPC client uses blocking HTTP requests without timeouts, which can hang CI indefinitely if the local node becomes unresponsive.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (5)

rs/ethereum/cketh/test_utils/src/anvil.rs:194

• Like rpc_result, this readiness probe has no HTTP timeout and can hang forever on a stuck TCP connect/read, preventing the 30s deadline from being enforced. Add a short request timeout so the loop can retry/fail fast.

        let ready = reqwest::blocking::Client::new()
            .post(url)
            .json(&serde_json::json!({"jsonrpc": "2.0", "id": 1, "method": "eth_blockNumber", "params": []}))
            .send()
            .map(|r| r.status().is_success())
            .unwrap_or(false);

rs/ethereum/cketh/test_utils/src/live_scan.rs:10

• This docs block says the EVM RPC canister issues "genuine HTTPS outcalls", but the configured overrideProvider replacement is the anvil URL (currently http://127.0.0.1:...). Updating the wording avoids confusion when reading the test harness.

//! harness runs PocketIC in *live* mode so the EVM RPC canister issues genuine HTTPS outcalls, and
//! installs it with an `overrideProvider` that rewrites every provider URL to the harness' anvil
//! node (mirroring the `evm_rpc_local` configuration of the EVM RPC canister). The minter therefore
//! reads real Ethereum state from anvil, exercising the balance scan end to end: minter → EVM RPC
//! canister → anvil.

rs/ethereum/cketh/test_utils/src/anvil.rs:64

• reqwest::blocking::Client defaults to no request timeout; if anvil becomes unresponsive, this JSON-RPC call can hang indefinitely and stall CI. Add an explicit per-request timeout.

This issue also appears on line 189 of the same file.

        let body: Value = reqwest::blocking::Client::new()
            .post(&self.url)
            .json(
                &serde_json::json!({"jsonrpc": "2.0", "id": 1, "method": method, "params": params}),
            )
            .send()
            .unwrap()
            .json()
            .unwrap();

rs/ethereum/cketh/test_utils/src/live_scan.rs:2

• The harness routes providers to Anvil::url(), which is an http://127.0.0.1:... URL; calling these "HTTPS outcalls" in the module docs is misleading. Consider describing this as real HTTP (or HTTP(S)) outcalls instead.

This issue also appears on line 6 of the same file.

//! A live [`PocketIc`] harness for the ckERC20 balance scan, driving *real* HTTPS outcalls against
//! a local anvil node that the harness owns (see [`crate::anvil`]).

rs/ethereum/cketh/minter/tests/deposit_from_cex.rs:182

• This comment says the scan uses "genuine HTTPS outcalls", but the live harness routes providers to an http://127.0.0.1:... anvil URL. Consider changing this to "HTTP" (or "HTTP(S)") to match what is actually configured.

/// End-to-end balance scan against a real EVM: a live PocketIC runs the minter and the *real* EVM
/// RPC canister (configured to route every provider to the harness' anvil node), so the minter's
/// periodic balance scan issues genuine HTTPS outcalls and reads real ERC-20 balances from anvil.

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[gregorydemay]

🤖 Thanks — addressed both points from the Copilot review in b0e768e:

• No HTTP timeout on the blocking anvil client (anvil.rs): added an explicit per-request timeout via a shared rpc_client() builder, used by both rpc_result and wait_until_ready. A wedged node now fails fast instead of hanging a send() past the readiness deadline.
• "HTTPS outcalls" wording (live_scan.rs, deposit_from_cex.rs): reworded to say the EVM RPC canister makes genuine outcalls through the IC's HTTPS-outcalls feature, reaching the local anvil node over HTTP — dropping the misleading "HTTPS" wire description.

Live deposit_from_cex test still passes locally.

[Copilot]

🟡 Not ready to approve

The new Anvil JSON-RPC helper currently panics on transport/JSON failures despite returning Result, which can reduce test reliability and debuggability for the added live end-to-end test.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

rs/ethereum/cketh/test_utils/src/anvil.rs:33

• rpc_client() builds a new reqwest::blocking::Client for every JSON-RPC call. Creating clients is relatively expensive and prevents connection reuse; this test helper makes many RPCs (including receipt polling), so caching the client reduces overhead and improves test runtime stability.

fn rpc_client() -> reqwest::blocking::Client {
    reqwest::blocking::Client::builder()
        .timeout(RPC_TIMEOUT)
        .build()
        .expect("failed to build the anvil RPC client")
}

rs/ethereum/cketh/test_utils/src/anvil.rs:81

• rpc_result() is documented as returning a Result, but transport / decode failures currently unwrap() and panic. That defeats the caller’s ability to handle transient failures (e.g., timeouts) and also loses context about which RPC failed.

    fn rpc_result(&self, method: &str, params: Value) -> Result<Value, String> {
        let body: Value = rpc_client()
            .post(&self.url)
            .json(
                &serde_json::json!({"jsonrpc": "2.0", "id": 1, "method": method, "params": params}),
            )
            .send()
            .unwrap()
            .json()
            .unwrap();

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[gregorydemay]

🤖 Addressed both points from the latest Copilot review in b508bce:

• Client reuse — the blocking JSON-RPC client is now built once and stored on Anvil, so RPCs share a connection pool instead of constructing a fresh client (and TCP connection) per call.
• rpc_result honoring its Result — transport and decode failures (including an RPC_TIMEOUT timeout) now return Err tagged with the failing method instead of unwrap()-panicking, so callers like eth_call_create can distinguish a transient failure. Existing behavior is unchanged: a revert still surfaces via the JSON-RPC error body.

Live deposit_from_cex test still passes locally.

[Copilot]

🟡 Human review recommended

It introduces a live PocketIC + anvil end-to-end test with wall-clock polling and real HTTP outcalls/process management that warrants human validation for CI reliability/flakiness.

Review details

Comments suppressed due to low confidence (1)

rs/ethereum/cketh/test_utils/src/anvil.rs:88

• rpc_result is documented as returning Err values “tagged with the method”, but JSON-RPC errors currently return error.to_string() without any method context. This makes failures harder to diagnose and contradicts the doc comment; wrap the JSON-RPC error with the method name (or update the docs to match).

        match body.get("error") {
            Some(error) if !error.is_null() => Err(error.to_string()),
            _ => Ok(body["result"].clone()),
        }

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.