feat: report foreign chain RPC provider health at Node startup by haiyuechen-nearone · Pull Request #4071 · near/mpc

haiyuechen-nearone · 2026-08-05T15:46:49Z

Closes #4094.

The node now spawns probe_all_providers as a detached task at startup. An e2e test is added to validate the code through asserting metrics emitted.

Notes for review

The status of each provider is logged: This is safe because ProviderStatus renders no upstream text, and the observed fingerprint from provider response is length capped.
The metric gauges are labelled per chain: We are interested in if a chain can be served from by a Node. Logs can be used to figure out exactly which provider is down.
Probing for Ethereum, Solana and TON is not implemented
A completely empty foreign_chains section would produce a warn log.

claude · 2026-08-11T16:36:22Z

Pull request overview

Wires the existing foreign-chain-health-check probe into node startup: run_mpc_node spawns run_startup_probe as a detached task that asks every configured RPC provider which network it serves, logs a line per provider plus an x/y providers healthy summary, and publishes two per-chain gauges. A new ForeignChain::label() supplies the operator-facing chain name used for the metric label, guarded by a config-key/label agreement test. Coverage is a pair of unit tests over the node's reporting plus an e2e test asserting the gauges on a two-node cluster where one node is pointed at the right network and one at the wrong one.

Changes:

crates/node/src/foreign_chain_probe.rs (new): run_startup_probe — empty-config warning, probe_all_providers call, publish_metrics, log_report/summarize (providers with no probe are excluded from both the summary and the gauges).
crates/node/src/run.rs: detached root_runtime.spawn of the probe after the web server starts; crates/node/src/metrics.rs: mpc_foreign_chain_rpc_providers_{configured,healthy} IntGaugeVecs labelled by chain.
ForeignChain::label() in near-mpc-contract-interface, plus an rstest for it and a node-config test asserting every serialized chain key equals its label.
e2e: eth_chainId on the EVM mock, MOCK_EVM_CHAIN_ID, new port seed, metric-name constants, and foreign_chain_probe.rs.
docs/foreign-chain-transactions.md: probe is now wired and detached; documents the two gauges and why unprobeable chains report nothing; drops ton from the "no inspector" lists.

Reviewed changes

Per-file summary

File	Description
Cargo.lock	`mpc-node` gains `foreign-chain-health-check`
crates/node/Cargo.toml	Same dependency added
crates/node/src/lib.rs	Declares `mod foreign_chain_probe`
crates/node/src/foreign_chain_probe.rs	New module: probe orchestration, per-provider logging, per-chain gauge publication, unit tests
crates/node/src/metrics.rs	Two new chain-labelled `IntGaugeVec`s
crates/node/src/run.rs	Spawns the probe detached at startup
crates/near-mpc-contract-interface/src/types/foreign_chain.rs	`ForeignChain::label()` + rstest over three variants
crates/node-config/src/foreign_chains.rs	Test that serialized chain keys equal `label()` for all 12 configurable chains
crates/e2e-tests/src/foreign_chain_mock.rs	`MOCK_EVM_CHAIN_ID` and an `eth_chainId` response
crates/e2e-tests/src/metrics.rs	Metric-name constants for the new gauges
crates/e2e-tests/tests/common.rs	`FOREIGN_CHAIN_PROBE_PORT_SEED = 27`
crates/e2e-tests/tests/e2e.rs	Registers the new test module
crates/e2e-tests/tests/foreign_chain_probe.rs	New e2e test: healthy node vs. wrong-network node
docs/foreign-chain-transactions.md	Probe/metrics behaviour documented; `ton` removed from the no-inspector lists

Findings

Blocking (must fix before merge):

crates/node/src/foreign_chain_probe.rs:191, :210 — the only tests of the reporting logic drive it through probe_all_providers, so each one opens real sockets (http://127.0.0.1:9) and exercises another crate's probe machinery in order to assert this crate's gauge publication. That is what engineering-standards.md §Separate business logic from I/O and §Add tests ("we should not run other functionality than the system under test in unit tests") rule out. The cause is mechanical: publish_metrics/log_report take &ProbeReport, and ProbeReport (crates/foreign-chain-health-check/src/probe.rs:69) has no public constructor, while ProviderHealth is freely constructible — the row() helper at :159 already builds rows for the summarize tests. Add ProbeReport::new(rows: Vec<ProviderHealth>) (or impl From<Vec<ProviderHealth>>) upstream, or change the two functions to take &[ProviderHealth], and then assert the gauges from hand-built rows with no network at all. That also buys the case the current tests cannot express: several chains in one report with a mix of Healthy/unhealthy/ProbeNotImplemented rows, which is precisely the per-chain filtering in publish_metrics:80-99, and a chain whose label differs from its Debug form (hyper_evm) so the label actually used in the series is pinned.
docs/foreign-chain-transactions.md:565 and crates/foreign-chain-inspector/src/lib.rs:59 — both still justify fingerprint truncation with "it is repeated into logs and metric labels", which this PR settles otherwise: line 558 explains the gauges carry a chain label only, and no metric anywhere carries a fingerprint. Same document, two paragraphs apart, now contradict each other; per CLAUDE.md doc drift is fixed in the authoring PR. Drop the metric-label clause in both places (the log rationale still stands on its own).

Non-blocking (nits, follow-ups, suggestions):

crates/node/src/foreign_chain_probe.rs:50, :55, :60 — the per-provider lines emit chain = ?row.chain (HyperEvm), while the metric label and the aggregate warning at :70 use label() (hyper_evm). An operator correlating a gauge series with the log line, or grepping for the key they wrote in foreign_chains.yaml, gets two spellings. chain = %row.chain.label() throughout.
crates/node/src/foreign_chain_probe.rs:49, :54 — healthy and unprobeable providers log at debug!, so at RUST_LOG=info (scripts/launch-localnet.sh:174, docs/localnet/docker-envs/*.env) there is no line per provider, which is what docs:558 and the issue's acceptance criteria promise. Either lift the healthy line to info! (the count is bounded by the config) or say "at debug" in the doc.
crates/e2e-tests/tests/foreign_chain_probe.rs:86-95 — node 1's healthy == Some(0) is read with a bare scrape, relying on both gauges being set back-to-back in publish_metrics; common::wait_metric_on_nodes(&cluster, &[1], …HEALTHY, |v| v == 0, …) closes that window and still fails on a missing series, since the predicate only runs on Some. Also, the // when block is really the first assertion — the only action is cluster startup.
crates/near-mpc-contract-interface/src/types/foreign_chain.rs:1837 — the comment says "The two whose config key is not the variant name lowercased" above three cases, and sui is exactly the variant name lowercased, so it belongs to neither the count nor the category. Drop the sui case or reword. Worth noting the new node-config test already covers all 12 configurable chains by construction, so this rstest is close to redundant.
crates/foreign-chain-health-check/src/lib.rs:52-112 — check_all_providers hardcodes the same strings label() now owns ("base", "hyper_evm", "abstract", …); an opportunistic switch to chain.label() would leave one source of truth. Related: crates/e2e-tests/src/cluster.rs:121 derives chain names from Debug + to_lowercase(), which yields hyperevm and now disagrees with label().
crates/node/src/run.rs:213 — unlike the other background tasks spawned here (run.rs:252 takes cancellation_token.child_token()), the probe gets no cancellation handle, so an in-flight probe can outlive shutdown initiation by up to timeout_sec × max_retries. Also worth recording somewhere: the production build profile sets panic = "abort" (Cargo.toml:339, used by nix/mpc-node.nix:154), so spawn does not isolate the node from a panic in the probe — the "a panic never delays startup" criterion holds only because the probe path is panic-free today (no unwrap/expect outside cfg(test)).
crates/e2e-tests/src/foreign_chain_mock.rs:78 — MOCK_EVM_CHAIN_ID is Base's id but the same mock backs the bnb/arbitrum/abstract/hyper_evm/polygon servers in foreign_chain_tx_validation.rs; harmless today because those sections set expected_network_fingerprint: None (so the probe short-circuits and issues no request, leaving the existing calls() > 0 assertions intact), but a one-line note on the constant would stop the next test from assuming per-chain ids.

⚠️ Issues found

haiyuechen-nearone · 2026-08-11T17:27:26Z

@claude review

claude · 2026-08-11T17:38:03Z

Pull request overview

Wires the existing foreign-chain-health-check probe into node startup. run_mpc_node spawns run_startup_probe as a detached task that asks every configured RPC provider which network it serves, logs a line per provider plus an x/y providers healthy summary, and publishes two per-chain gauges. A new ForeignChain::label() supplies the operator-facing chain name used for the metric label, pinned by a node-config test asserting every serialized chain key equals its label. Coverage is four unit tests over the node's reporting plus an e2e test asserting the gauges on a two-node cluster where one node is pointed at the right network and one at the wrong one.

Since the previous review round both blocking items were addressed: ProbeReport gained From<Vec<ProviderHealth>> so publish_metrics is now asserted from hand-built rows with no sockets opened, and the two stale "repeated into logs and metric labels" passages were corrected. The per-provider log lines now use label() and info!, the e2e test waits on the metric instead of scraping bare, and the sui rstest case is gone.

Changes:

crates/node/src/foreign_chain_probe.rs (new): run_startup_probe — empty-config warning, probe_all_providers call, publish_metrics, log_report/summarize. Providers with no probe are excluded from both the summary and the gauges.
crates/node/src/run.rs: detached root_runtime.spawn of the probe after the web server starts; crates/node/src/metrics.rs: mpc_foreign_chain_rpc_providers_{configured,healthy} IntGaugeVecs labelled by chain.
ForeignChain::label() in near-mpc-contract-interface, plus an rstest for it and a node-config test asserting every serialized chain key equals its label.
impl From<Vec<ProviderHealth>> for ProbeReport in foreign-chain-health-check, so the report is constructible outside the crate.
e2e: eth_chainId on the EVM mock, MOCK_EVM_CHAIN_ID, new port seed, metric-name constants, and foreign_chain_probe.rs.
docs/foreign-chain-transactions.md: probe is now wired and detached; documents the two gauges and why unprobeable chains report nothing.

Reviewed changes

Per-file summary

File	Description
Cargo.lock	`mpc-node` gains `foreign-chain-health-check`
crates/node/Cargo.toml	Same dependency added
crates/node/src/lib.rs	Declares `mod foreign_chain_probe`
crates/node/src/foreign_chain_probe.rs	New module: probe orchestration, per-provider logging, per-chain gauge publication, unit tests
crates/node/src/metrics.rs	Two new chain-labelled `IntGaugeVec`s
crates/node/src/run.rs	Spawns the probe detached at startup
crates/foreign-chain-health-check/src/probe.rs	`From<Vec<ProviderHealth>>` for `ProbeReport`; `probe_all_providers` returns via `.into()`
crates/foreign-chain-inspector/src/lib.rs	Truncation rationale no longer claims metric labels carry the fingerprint
crates/near-mpc-contract-interface/src/types/foreign_chain.rs	`ForeignChain::label()` + rstest over two variants
crates/node-config/src/foreign_chains.rs	Test that serialized chain keys equal `label()` for all 12 configurable chains
crates/e2e-tests/src/foreign_chain_mock.rs	`MOCK_EVM_CHAIN_ID` and an `eth_chainId` response
crates/e2e-tests/src/metrics.rs	Metric-name constants for the new gauges
crates/e2e-tests/tests/common.rs	`FOREIGN_CHAIN_PROBE_PORT_SEED = 27`
crates/e2e-tests/tests/e2e.rs	Registers the new test module
crates/e2e-tests/tests/foreign_chain_probe.rs	New e2e test: healthy node vs. wrong-network node
docs/foreign-chain-transactions.md	Probe/metrics behaviour documented

Findings

Blocking (must fix before merge):

crates/node/src/foreign_chain_probe.rs:45 and :18 — the log output is untested, and it is this PR's headline deliverable. publish_metrics and summarize are now covered, but nothing asserts log_report itself: not the per-row branch selection, not the {healthy}/{probed} providers healthy summary line, not the probed == 0 warning with its chains list, and not the empty-config guard at :18-21. Delete log_report and the is_empty() branch outright and the suite still passes, which is what engineering-standards.md §Add tests rules out ("Any change to our system should come with tests such that if the change was reverted, these tests would fail"). Three of the four notes in the PR description are about log content, and docs/foreign-chain-transactions.md:558 promises "a line per provider" plus the summary. The tooling is already in place: tracing-test is a dev-dependency of this crate (crates/node/Cargo.toml:130) and #[traced_test] + logs_contain is used in crates/node/src/migration_service/onboarding.rs:361-444. All three cases are pure — run_startup_probe returns before any I/O on an empty config, and log_report takes a report you can hand-build with the new From impl:
```
#[tokio::test]
#[traced_test]
async fn run_startup_probe__should_warn_when_no_foreign_chain_is_configured() {
    // Given
    let foreign_chains = ForeignChainsConfig::default();

    // When
    run_startup_probe(foreign_chains).await;

    // Then
    assert!(logs_contain("no foreign chain is configured"));
}
```
plus a log_report__should_summarize_only_the_providers_a_probe_covers over a Healthy + Unreachable + ProbeNotImplemented report asserting 1/2 providers healthy, and one for the all-ProbeNotImplemented warning. If you would rather not assert on logs, extracting the summary line into a fn summary_line(&Summary) -> String lets the third case assert a returned value instead.

Non-blocking (nits, follow-ups, suggestions):

crates/near-mpc-contract-interface/src/types/foreign_chain.rs:1837 — the doc comment still misdescribes its cases after the sui removal: "The two whose config key is not the variant name lowercased" now sits above HyperEvm and Abstract, but Abstract's config key is Abstract lowercased. What is actually special about it is that the config field has to be abstract_chain with #[serde(rename = "abstract")] because abstract is a reserved word (crates/node-config/src/foreign_chains.rs:25-26) — a different failure mode from hyper_evm vs. hyperevm. Reword to cover both reasons, or drop the Abstract case.
crates/node/src/metrics.rs:466 — "RPC providers configured for a foreign chain" over-promises: publish_metrics omits the series entirely for a chain with no probe, so sum(mpc_foreign_chain_rpc_providers_configured) undercounts what the operator actually configured. A dashboard author reading only the help text will take it for a config inventory. Suggest "RPC providers configured for a foreign chain the node can probe", so the omission documented at docs/foreign-chain-transactions.md:560 is visible from the metric itself.
crates/node/src/foreign_chain_probe.rs:15-16 — run_startup_probe's doc comment describes the logging but not the two gauges it publishes, which is the half a consumer (alert or dashboard author) actually needs from the entry point.
crates/node/src/foreign_chain_probe.rs:59 — MissingExpectedFingerprint lands in the unhealthy arm and logs "foreign-chain RPC provider is unhealthy". The provider is fine; the operator omitted expected_network_fingerprint. The status field carries the distinction, but the message sends them to the wrong place first. A third arm naming the config would read better, and the counting behaviour documented around docs/foreign-chain-transactions.md:723 needn't change.

Still open from the previous round, no need to re-litigate: the check_all_providers / cluster.rs:121 duplicate chain-name spellings, the missing cancellation handle on the spawned probe (run.rs:213) together with panic = "abort" in the release profile, and the note on MOCK_EVM_CHAIN_ID being Base's id while the same mock backs five other chains' servers.

⚠️ Issues found

haiyuechen-nearone · 2026-08-11T19:29:29Z

the log output is untested ...

Deliberately not adding tests grepping for log lines. Having test grepping for text is very prone to drift. Log output was verified by running the cluster locally and inspecting it.

All green case:

Case with failures:

haiyuechen-nearone · 2026-08-11T19:31:38Z

MissingExpectedFingerprint lands in the unhealthy arm and logs "foreign-chain RPC provider is unhealthy". The provider is fine; the operator omitted expected_network_fingerprint.

Since we expect the operator to correctly configure all fields in the config file, not adding expected network fingerprint is considered "incorrect" and causes us to not know if a provider is healthy or not/

Spawns the probe detached, so it reports which network each provider serves without delaying startup or gating anything. A status carries no provider text and no auth material, so it is logged whole.

Publishes `mpc_foreign_chain_rpc_providers_configured` and `mpc_foreign_chain_rpc_providers_healthy`, labelled by chain rather than by provider, since a provider name is operator chosen and would put an unbounded label on a time series. `ForeignChain::label()` is the one place the chain label lives, so a metric label and a config key cannot drift apart. An e2e test points one node at a mock serving the expected chain id and another at the same mock while expecting a different network, so the gauges separate a healthy provider from one on the wrong network end to end. The EVM mock answers `eth_chainId` to make that possible. Leaves a provider no probe covers out of the `x/y providers healthy` summary. It counted toward the denominator but could never reach the numerator, so a node configuring a chain without a probe read as permanently degraded.

A chain no probe covers no longer publishes gauges, and an empty foreign chains config no longer warns on every boot. The panic guard is gone: the deployed binary aborts on panic, so it never ran. The summary is a value now, asserted directly instead of through log lines, and a node config test pins every chain's config key to its label.

A node is expected to configure the chains the code supports, so an empty section is a misconfiguration rather than a quiet path. The startup probe is the only place that observes it.

The warning named our own mechanism and left the operator to work out which of their chains went unchecked; it now lists them. The docs claimed a `ton` section exists, and that chains without an inspector report a status name an operator never sees.

`ProbeReport` gains `From<Vec<ProviderHealth>>`, so the gauge tests no longer open sockets and drive another crate's probe just to reach this crate's publication. They now cover several chains in one report, and pin the label of the one chain whose key is not its variant name. Log lines carry the chain's config key rather than its variant name, and a healthy or uncheckable provider logs at info, so the documented line per provider appears at the level operators run. Also drops metric labels from the truncation rationale: no metric carries a fingerprint.

Keep only the claim the fixture does not make, and leave the Given and When markers bare.

haiyuechen-nearone changed the title ~~feat(node): probe the foreign chain RPC providers on startup~~ feat: probe the foreign chain RPC providers on startup Aug 5, 2026

This was referenced Aug 7, 2026

feat: probe Bitcoin provider by genesis block hash #4067

Merged

feat: probe Sui providers by genesis checkpoint digest #4070

Open

haiyuechen-nearone changed the title ~~feat: probe the foreign chain RPC providers on startup~~ feat(node): run the provider identity probe on startup Aug 7, 2026

This was referenced Aug 7, 2026

feat: Starknet RPC healthcheck probing logic #4013

Merged

feat: EVM chains RPC probing logic #4049

Merged

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from 36311f1 to 4ff2996 Compare August 7, 2026 14:35

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from 4ff2996 to c42d937 Compare August 11, 2026 11:25

haiyuechen-nearone changed the title ~~feat(node): run the provider identity probe on startup~~ feat(node): report foreign chain RPC provider health on startup Aug 11, 2026

haiyuechen-nearone changed the title ~~feat(node): report foreign chain RPC provider health on startup~~ feat: report foreign chain RPC provider health on startup Aug 11, 2026

haiyuechen-nearone changed the title ~~feat: report foreign chain RPC provider health on startup~~ feat: report foreign chain RPC provider health at Node startup Aug 11, 2026

haiyuechen-nearone marked this pull request as ready for review August 11, 2026 16:23

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from b060ade to 8b6edca Compare August 11, 2026 16:23

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from 8b6edca to 6023892 Compare August 11, 2026 16:44

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from 35b22c4 to 9ec4b44 Compare August 13, 2026 19:45

haiyuechen-nearone added 5 commits August 14, 2026 09:23

feat(node): probe the foreign chain RPC providers on startup

79eae64

Spawns the probe detached, so it reports which network each provider serves without delaying startup or gating anything. A status carries no provider text and no auth material, so it is logged whole.

docs(node): say precisely why a provider status is safe to log

c978d8f

fix(node): warn when no foreign chain is configured

80d8dc2

A node is expected to configure the chains the code supports, so an empty section is a misconfiguration rather than a quiet path. The startup probe is the only place that observes it.

haiyuechen-nearone added 2 commits August 14, 2026 09:23

haiyuechen-nearone force-pushed the 3764-node-startup-probe branch from 9ec4b44 to b25b5e2 Compare August 14, 2026 07:23

docs(probe): trim the startup probe test comments

ea5e97c

Keep only the claim the fixture does not make, and leave the Given and When markers bare.

Navigation Menu

Uh oh!

feat: report foreign chain RPC provider health at Node startup - #4071

feat: report foreign chain RPC provider health at Node startup#4071
haiyuechen-nearone wants to merge 8 commits into
4003-probe-sui-genesis-digestfrom
3764-node-startup-probe

haiyuechen-nearone commented Aug 5, 2026 •

edited

Loading

Uh oh!

claude Bot commented Aug 11, 2026

Uh oh!

haiyuechen-nearone commented Aug 11, 2026

Uh oh!

claude Bot commented Aug 11, 2026

Uh oh!

haiyuechen-nearone commented Aug 11, 2026 •

edited

Loading

Uh oh!

haiyuechen-nearone commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

haiyuechen-nearone commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes for review

Uh oh!

claude Bot commented Aug 11, 2026

Pull request overview

Reviewed changes

Findings

Uh oh!

haiyuechen-nearone commented Aug 11, 2026

Uh oh!

claude Bot commented Aug 11, 2026

Pull request overview

Reviewed changes

Findings

Uh oh!

haiyuechen-nearone commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haiyuechen-nearone commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

haiyuechen-nearone commented Aug 5, 2026 •

edited

Loading

haiyuechen-nearone commented Aug 11, 2026 •

edited

Loading