Fail over fork RPC endpoints with an archive-aware preflight - #289
Fail over fork RPC endpoints with an archive-aware preflight#289thedavidmeister wants to merge 5 commits into
Conversation
Foundry maps one [rpc_endpoints] alias to exactly one URL and --fork-retries only retries that same URL, so a deterministic upstream failure — an exhausted plan quota, a pruning node, a dead host — cannot be recovered inside forge. rainix-static rpc-preflight builds a merged candidate pool per network (the RPC_URL_<NET>_FORK secret's newline-separated list, then the variable's, then hardcoded public archive defaults), probes each with chain id plus historical account state and a historical eth_call at the deepest block any repo in the org pins for that network, and exports the first healthy one as <NETWORK>_RPC_URL. Every entry in every source is a candidate; the sources only set the order. Candidate URLs are structurally unprintable: the Url newtype has no Display and a Debug of <redacted>, every log line is a Source label plus a closed Reason enum, and the only sinks are curl's argv and $GITHUB_ENV. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 28 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
rainix has no fork suite of its own, so the reusables' RPC wiring was never exercised here and every bug in it surfaced downstream in a consumer's red CI. rainix's own sources name every <NETWORK>_RPC_URL, so the demand scan selects all of them: the job runs the composite on a real runner and asserts each archive network got bound, without echoing a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured today: rpc.hyperliquid.xyz/evm, rpc.hyperlend.finance and hyperliquid.drpc.org all report chain 999 and answer totalSupply() on WHYPE at 0x5555…5555, so hyperevm no longer has to depend on a configured secret to have any candidate at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
parse_chain_id, probe_blocks, block_tag and check_call_result were inline in probe(), which needs a live endpoint, so nothing covered them: the archive-vs- latest branch, the hex block tag and the 32-byte word check could all be changed without a test noticing. They are pure, so lift them out and test them directly — the block-tag test caught a wrong hex constant on the way in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
thedavidmeister
added
the
ai:ready
|
🤖 ai:vetter |
thedavidmeister
added
ai:design
|
🤖 ai:producer |
1 participant
Closes #288.
Foundry maps one
[rpc_endpoints]alias to exactly one URL and--fork-retriesonly retries that same URL, so a deterministic upstream failure — the exhausted drpc plan quota that reddenedrain.extrospection/rain.factory.deploy, a pruning node, a dead host — cannot be recovered insideforge. This recovers it one step earlier: a preflight picks a candidate that is healthy right now and binds it to<NETWORK>_RPC_URLbefore forge starts.What's here
rainix-static rpc-preflight(rainix-static/src/rpc_preflight.rs) — all the logic, per the "tooling is Rust, never bash" rule. 20 unit tests, run inside the nix build..github/actions/rpc-preflight/action.yml— house pattern:nix run "path:$GITHUB_ACTION_PATH/../../..#rainix-static", so the check version always matches the action version regardless of anyRAINIX_SHAthe caller pins.rainix-sol-test.yaml,rainix-tag-release.yaml,rainix-manual-sol-artifacts.yaml— replacing the staticsecrets.X || vars.Xmapping in each.flake.nix:rainix-staticis now wrapped with the pinnedcurlandgitplus a CA bundle. The composite runsnix runoutside any devshell, so ambient PATH is whatever the runner image ships. Verified hermetic — the nix-built binary completes a full selection underenv -i PATH=/nonexistent.Merged pool, not a precedence chain
Per the ruling — "i didn't say fallback if not set i said merge into a list of urls that are all used". The secret and the variable each hold a newline-separated list, and the candidate pool is the concatenation of both plus hardcoded defaults:
RPC_URL_<NET>_FORKRPC_URL_<NET>_FORKA URL in the variable is tried even when the secret is set — a non-empty earlier source never causes a later one to be skipped, it only orders them. Ordering is secret-first because that is the paid/dedicated endpoint; public ones are the safety net. In the demo below
secret[0],secret[1]andvariable[0]are all rejected andvariable[1]is selected, which is only reachable under the merged reading.Separator = newline, because it cannot occur inside a URL (no escaping), GitHub's secret/variable editors take multi-line values natively, and the runner registers each line of a multi-line secret as a separately masked value. A single bare URL parses as a one-element list, which is exactly what these secrets hold today — no configuration change is required for this PR to be an improvement; the hardcoded defaults alone already fix the outage.
#starts a comment so a dead candidate can be parked with a note.Foundry consumes exactly one URL per alias, so "all used" is implemented as all are candidates the preflight will actually try. See the open question at the bottom.The probe is archive-aware, and that is the load-bearing part
eth_blockNumberis useless here: it passes on a pruning node that then fails the suite. So iseth_getCodealone —base.meowrpc.comserves historical code but answers everyeth_callwith "method not supported". Per candidate, fail-cheapest-first:eth_chainIdagainst the expected chain id — a wrong-chain candidate would silently fork the wrong network, worse than a red job.eth_getBalanceat the probe block — historical account state, what forge's fork backend fetches first.eth_callat the probe block — historical execution, and the result must be a full 32-byte word (a0xanswer means the node served empty state).3 consecutive full passes required.
1rpc.io/arbpassed the historical probe 1 time in 5 and1rpc.io/baseflipped from archive tostate is prunedwithin twenty minutes — these hosts round-robin over a mix of archive and pruning backends, so a single sample qualifies an endpoint that dies partway through a suite issuing thousands of historical reads.Probe block = the deepest block any repo in the org pins for that network, because the question is not "is this an archive node" but "can it serve my fork block":
rain.tofu.erc20-decimals(~20 months)rain.deployfindDeployBlockbinary-searches from genesis; the second block stops a node holding only early state from passingrain.metadatametaboard start blockrain.flare, ~19 test files (~23 months)rain.metadatametaboard start blockNote base_sepolia and polygon have no two-arg
createSelectFork(url, block)anywhere — their archive requirement is hidden insideLibRainDeploy.isStartBlock, which forks at latest and thenvm.rollFork()s backwards.archive: 'false'on the deploy path (rainix-manual-sol-artifacts): a broadcast only reads head state, so requiring archive there would reject a healthy pruning endpoint for nothing. The_FORKsuffix on the secret is a misnomer on that path.How the no-leak guarantee works
Not a scrubbing pass — those get bypassed by the next edit. It is structural, in three layers:
struct Url(String)has noDisplay, and itsDebugprints<redacted>. No format string anywhere in the crate can put a URL on stdout or stderr, including a future edit that adds a{:?}to a log line. The inner value is reachable only viaexpose(), which has exactly three call sites: curl's argv, the write to$GITHUB_ENV, and the::add-mask::command in (3) — whose entire purpose is that the runner redacts the value it is handed, including in the command line itself. There is deliberately no stdout mode: invoked without a file to write to, the subcommand fails rather than falling back to printing.Sourceplus aReason, both closed enums over fixed text.Reason'sDisplayis fixed strings plus integers we produced ourselves (a JSON-RPC code, an HTTP status, a curl exit, a chain id, a block). Upstream response bodies are classified and then dropped — an endpoint that echoes the request URL back inside its error message, or serves an HTML error page containing it, cannot leak it. curl's stderr isStdio::null()ed because curl writes the request host into its own error text.::add-mask::on any secret-sourced selection before it reaches$GITHUB_ENV. Redundant with the runner's own per-line masking of multi-line secrets, but free — and it does not help for the case that actually matters, which is a keyed URL misconfigured intovars.(not auto-masked). That case is covered by (1) and (2).Candidate URLs also travel to the composite as env vars, not
with:inputs, because GitHub renders a composite's resolved inputs into the log.There is a test asserting
Url's debug is<redacted>and one asserting noReasonvariant renders ahttp://orhttps://.Blast radius containment
git ls-fileslooks for<NET>_RPC_URL(the foundry.toml alias form) andRPC_URL_<NET>_FORK(the raw-secret formcyclo.solreads viavm.envString). An unmentioned network is never probed, never exported, and can never fail the job. A test assertsBASE_SEPOLIA_RPC_URLdoes not dragbasein.<NET>_RPC_URLandRPC_URL_<NET>_FORK. The second matters: once the secret can hold a list, repos reading the raw secret name would otherwise get a multi-line stringcreateSelectForkcannot use.Riding along
RPC_URL_HYPEREVM_FORKwas declared inrainix-manual-sol-artifacts.yamlonly, so a repo withhyperevmin[rpc_endpoints]could deploy but its fork tests and tag-release verification both failed "env var not found". Added torainix-sol-test.yaml,rainix-sol.yaml(declaration and the explicit forward — that forward is a hard allowlist,secrets: inheritupstream does not get past it) andrainix-tag-release.yaml. README's documented env list was also missingETHEREUM_RPC_URLandHYPEREVM_RPC_URL.Demonstration
Local end-to-end run against a synthetic consumer repo whose
foundry.tomldeclaresarbitrumand whose test readsRPC_URL_BASE_SEPOLIA_FORKdirectly. The secret list is[an exhausted-plan upstream with ?dkey=SUPERSECRETKEY123, a pruning node], the variable list is[publicnode (archive token-gated), thirdweb]. Polygon is configured but not referenced by the repo. The first two upstreams are local mocks replaying the verbatim drpc and geth error bodies; everything else is a real public endpoint.Every rejection is named by source with a typed reason. Polygon is absent because the repo does not reference it. base_sepolia had nothing configured and landed on a hardcoded default — the case of a repo that sets no secret at all today.
Total-failure path — every candidate for a required network unreachable (an outbound proxy pointed at a closed port, so the hardcoded defaults die too):
Note it does not stop at the first dead network — every required network is reported, so one run tells you everything that needs fixing. The secret here was
https://keyed.example/?dkey=SUPERSECRETKEY123; neither the key, nor the host, nor the variable's host appears anywhere in that output.Closing the loop with real forge — same pin
rain.extrospectionuses, isolatedHOMEso there is no warm fork cache:That
failed to get account for 0xA4b0…sequenceris precisely the cryptic mid-suite error this replaces withsecret[0] rejected: quota exhausted.Checks
cargo test— 34 pass (20 new),cargo fmt --checkandcargo clippy -D clippy::allclean.nix develop -c pre-commit run --all-files— yamlfmt, prettier-rainix, shellcheck, statix, taplo, nil, nixfmt, deadnix, denofmt all pass. Therustfmt-conditionalhook fails identically on pristinemain(cargo metadatalooks for aCargo.tomlat the repo root, which rainix does not have); it is not exercised by CI, wherecargo-fmtis off PATH and the hook no-ops. Not touched here — separate issue.Open question
Foundry consumes exactly one URL per alias, so "all used" is implemented as every URL in either list is a candidate the preflight will actually try, which is what the words say. True simultaneous multi-upstream load balancing would need a proxy sitting in front of forge, not a URL list. Flagging rather than silently choosing, in case that was the intent.Two follow-ups deliberately not in this PR, one of which is red on
mainright now:sepolia is not covered, and it is the reason
mainis red.rainix (ubuntu-latest, rainix-sol-artifacts)has failed on everymainrun since 2026-07-24 (green on 07-16), includingaa789b99, which this branch is cut from. The log is the same failure class this PR fixes, in its purest form:--rpc-url ''—secrets.CI_DEPLOY_SEPOLIA_RPC_URL || vars.CI_DEPLOY_SEPOLIA_RPC_URLresolves empty, andflake.nix'suntil do_deploy; do sleep 5loop then re-hits the empty URL ten times over ~50s before failing anyway. A candidate pool with a hardcoded default would make this green with zero configuration.I did not fold sepolia in, because doing so makes two decisions that are yours, not mine: (1) sepolia is the one network outside the
RPC_URL_<NET>_FORKscheme — secretCI_DEPLOY_SEPOLIA_RPC_URL→ envETH_RPC_URLandCI_DEPLOY_SEPOLIA_RPC_URL, withflake.nixhardcoding--rpc-url ${ETH_RPC_URL}, so folding it in either enshrines that naming or migrates it; and (2) that job broadcasts withDEPLOYMENT_KEY, so having a hardcoded default pick the endpoint for a real transaction is a materially bigger call than picking one for a fork read. Say which way you want it and it is a small follow-up. It is also block-pinned by a repo variable (CI_FORK_SEPOLIA_BLOCK_NUMBER), so its archive depth is configuration I cannot see from here.flake.nix'srainix-sol-artifactsretry loop is the other half of that log: ten retries with--resumeagainst the same endpoint is structurally identical to Foundry's--fork-retriesand just as useless against a deterministic failure. Classifying the error once and either failing fast or moving to another candidate is the same fix as this PR, applied to the deploy task.