Move RPC paging logic from ReScript to Rust

[DZakh]

Summary

Migrates the adaptive block interval (AIMD) paging logic and error classification from ReScript (RpcSource.res) to Rust (EvmRpcClient), enabling the Rust CLI to manage RPC query pagination independently. The ReScript runtime now delegates paging decisions to the Rust client via a structured retry protocol.

Key Changes

• Rust-side paging: Implemented get_next_page in EvmRpcClient to decide the actual toBlock from per-partition AIMD state, fan out concurrent eth_getLogs queries, deduplicate results by (blockNumber, logIndex), and race against a query timeout. On failure, returns a structured retry decision (suggested block range or backoff parameters).

• Error classification: Extracted provider error message parsing and block-range suggestion logic into new classify.rs module. Handles 15+ provider-specific error patterns (Alchemy, QuickNode, Chainstack, etc.) and deterministic "too many logs" responses.

• AIMD state management: New interval.rs module tracks per-partition block intervals and source-wide structural caps. Implements additive increase (grow on success) and multiplicative decrease (shrink on failure) with proper clamping.

• Retry protocol: EvmRpcClient throws a napi error whose message is a JSON payload describing the retry decision:

  {
    "kind": "Retry",
    "attemptedToBlock": int,
    "errorMessage": string | null,
    "requestStats": [{"method": string, "seconds": float}],
    "retry": {
      "tag": "WithSuggestedToBlock",
      "toBlock": int
    } | {
      "tag": "WithBackoff",
      "message": string,
      "backoffMillis": int
    }
  }

• ReScript integration: Updated RpcSource.res to parse the retry protocol via parseGetNextPageRetryError, removing ~400 lines of paging logic. Updated EvmRpcClient.res to accept syncConfig and call getNextPage instead of getLogs.

• Config plumbing: EvmRpcClientConfig now includes sync-tuning knobs (initialBlockInterval, backoffMultiplicative, accelerationAdditive, intervalCeiling, backoffMillis, queryTimeoutMillis) resolved by ReScript's EvmChain.getSyncConfig.

Notable Implementation Details

• Timeout is enforced via tokio::time::timeout, which cancels in-flight requests on expiry.
• Deduplication uses a HashSet keyed by blockNumber-logIndex to handle selections that overlap (e.g., OR'd param groups in a single event's where clause).
• Source-wide structural caps (e.g., "limited to 1000 blocks") only tighten, never loosen, preventing regression when multiple providers report different limits.
• Density-based caps (e.g., "too many logs") trigger immediate shrink-and-retry without backoff, since waiting doesn't help.
• Tests for error classification and AIMD state transitions are included in Rust; ReScript tests for getSuggestedBlockIntervalFromExn and isResponseTooLargeError are removed (logic now Rust-side).

https://claude.ai/code/session_019FA1CWtGQEg7KRMx4NvVn5

Summary by CodeRabbit

• New Features
  • Added page-based EVM log pagination via getNextPage, replacing direct log fetching, with configurable sync/backoff/interval/timeout tuning.
  • Added structured retry decisions and per-request timing stats, while keeping deduplication across log selections.
• Bug Fixes
  • Improved provider error handling by classifying “response too large” and provider block-range limit messages, including suggested block-range retries.
  • Better handling of RPC timeouts and malformed/failed responses.
• Tests
  • Updated pagination tests and added coverage for provider block-range retry classification.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f0a5a1a7-0940-4333-a16f-b21b7fef8869

📥 Commits

Reviewing files that changed from the base of the PR and between 57c0f76 and 902c25c.

📒 Files selected for processing (1)

• packages/cli/src/evm_rpc_source/mod.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• packages/cli/src/evm_rpc_source/mod.rs

---

📝 Walkthrough

Walkthrough

The EVM RPC client now supports adaptive paginated log fetching with provider-specific error classification, interval state, concurrent selection requests, deduplication, structured retry decisions, and updated Rescript integration and tests.

Changes

EVM RPC adaptive paging

Layer / File(s)	Summary
Classification and interval state packages/cli/src/evm_rpc_source/classify.rs, packages/cli/src/evm_rpc_source/interval.rs	Adds provider error classification and partition-aware AIMD interval management with unit tests.
Native paginated RPC client packages/cli/src/evm_rpc_source/mod.rs	Adds paging configuration and N-API types, concurrent eth_getLogs fetching, deduplication, timeout handling, and serialized retry decisions.
Rescript client integration packages/envio/src/sources/EvmRpcClient.res, packages/envio/src/sources/RpcSource.res	Replaces direct log fetching with getNextPage, forwards sync configuration, and parses native retry payloads.
Paging and retry validation scenarios/test_codegen/test/helpers/NativeDecoder.res, scenarios/test_codegen/test/lib_tests/EvmRpcClient_test.res, scenarios/test_codegen/test/RpcSource_test.res	Updates fixtures and tests for paginated requests, unique log indexes, returned block bounds, and provider retry classification.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RpcSource
  participant EvmRpcClient
  participant fetch_page
  participant eth_getLogs
  RpcSource->>EvmRpcClient: getNextPage(params)
  EvmRpcClient->>fetch_page: fetch log selections
  fetch_page->>eth_getLogs: concurrent requests
  eth_getLogs-->>fetch_page: logs or error
  fetch_page-->>EvmRpcClient: items and request stats
  EvmRpcClient-->>RpcSource: page response or retry payload

Loading

Possibly related PRs

• enviodev/hyperindex#1360: Adds the native EVM RPC log-fetching flow that this change replaces with paginated fetching.
• enviodev/hyperindex#1390: Updates request-stat handling in the same RPC source pagination path.
• enviodev/hyperindex#1302: Introduces the EVM RPC client foundation extended here with paging and retry state.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main migration of RPC paging logic from ReScript to Rust.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

packages/cli/src/evm_rpc_source/classify.rs (2)

76-119: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Repeated provider-check blocks could be table-driven.

The nine near-identical if let Some(n) = extract_positive_u64(&X, message) { return Some((n, true)); } blocks (plus the two fixed-value cases) duplicate the same pattern. Adding a new provider currently means copy-pasting another block; a small static table of (&Regex, Option<u64>) (fixed value or None to extract from a capture) iterated in order would reduce duplication and make it obvious the order matters.

♻️ Sketch of a table-driven approach

enum RangeCase {
    Extract(&'static LazyLock<Regex>),
    Fixed(&'static LazyLock<Regex>, u64),
}

// Order matters: more specific patterns first.
static STRUCTURAL_CASES: &[RangeCase] = &[
    RangeCase::Extract(&BLOCK_RANGE_LIMIT),
    RangeCase::Extract(&ALCHEMY_RANGE),
    // ...
    RangeCase::Fixed(&BASE_RANGE, 2000),
    RangeCase::Fixed(&CHAINSTACK, 10000),
];

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/evm_rpc_source/classify.rs` around lines 76 - 119, Refactor
the repeated provider checks in suggested_block_interval_from_message into an
ordered static table of regex cases, supporting both captured values and fixed
intervals. Preserve the existing precedence exactly, including BASE_RANGE and
CHAINSTACK placement, and iterate the table to return the first matching
(interval, true) result.

---

29-68: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Case-sensitivity is inconsistent with TOO_LARGE_PATTERNS.

TOO_LARGE_PATTERNS (lines 13-21) all use the (?i) flag, but none of the range-suggestion regexes here do. If a provider varies casing (e.g. "Limited to a 1000 Blocks range" vs "limited to a 1000 blocks range"), the match silently fails and falls through to the generic backoff path instead of applying the structural cap. Worth double-checking whether these were captured verbatim from real provider responses and are stable, or whether (?i) should be added defensively.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/evm_rpc_source/classify.rs` around lines 29 - 68, The
range-suggestion patterns are case-sensitive unlike TOO_LARGE_PATTERNS, so
provider message casing changes can bypass structural caps. Update every regex
in the range-detection statics, including SUGGESTED_RANGE, BLOCK_RANGE_LIMIT,
ALCHEMY_RANGE, CLOUDFLARE_RANGE, THIRDWEB_RANGE, BLOCKPI_RANGE, BASE_RANGE,
MAX_ALLOWED_BLOCKS, BLAST_PAID, CHAINSTACK, COINBASE, PUBLIC_NODE, and
HYPERLIQUID, to use the case-insensitive `(?i)` flag while preserving their
existing matching behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/evm_rpc_source/mod.rs`:
- Around line 205-221: Validate that params.from_block is less than or equal to
params.to_block_ceiling in the existing block-bounds validation before
converting them to u64; return the same mapped error with both values when the
ordering is invalid. Keep the to_block calculation in the surrounding RPC range
logic, including its defensive lower-bound behavior for valid inputs.
- Around line 161-178: Validate cfg.backoff_multiplicative while constructing
SyncConfig, requiring it to be finite and strictly between 0.0 and 1.0; return a
clear configuration error through the existing map_err path when invalid, and
only assign the validated value to backoff_multiplicative.

---

Nitpick comments:
In `@packages/cli/src/evm_rpc_source/classify.rs`:
- Around line 76-119: Refactor the repeated provider checks in
suggested_block_interval_from_message into an ordered static table of regex
cases, supporting both captured values and fixed intervals. Preserve the
existing precedence exactly, including BASE_RANGE and CHAINSTACK placement, and
iterate the table to return the first matching (interval, true) result.
- Around line 29-68: The range-suggestion patterns are case-sensitive unlike
TOO_LARGE_PATTERNS, so provider message casing changes can bypass structural
caps. Update every regex in the range-detection statics, including
SUGGESTED_RANGE, BLOCK_RANGE_LIMIT, ALCHEMY_RANGE, CLOUDFLARE_RANGE,
THIRDWEB_RANGE, BLOCKPI_RANGE, BASE_RANGE, MAX_ALLOWED_BLOCKS, BLAST_PAID,
CHAINSTACK, COINBASE, PUBLIC_NODE, and HYPERLIQUID, to use the case-insensitive
`(?i)` flag while preserving their existing matching behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6fb9fc57-e975-4c17-b115-b370b35b4141

📥 Commits

Reviewing files that changed from the base of the PR and between 00bb717 and 8e53236.

📒 Files selected for processing (8)

• packages/cli/src/evm_rpc_source/classify.rs
• packages/cli/src/evm_rpc_source/interval.rs
• packages/cli/src/evm_rpc_source/mod.rs
• packages/envio/src/sources/EvmRpcClient.res
• packages/envio/src/sources/RpcSource.res
• scenarios/test_codegen/test/RpcSource_test.res
• scenarios/test_codegen/test/helpers/NativeDecoder.res
• scenarios/test_codegen/test/lib_tests/EvmRpcClient_test.res

💤 Files with no reviewable changes (1)

• scenarios/test_codegen/test/RpcSource_test.res