Cap per-chain query concurrency to 100 - #1440
Merged
DZakh merged 2 commits intoJul 16, 2026
Merged
Conversation
The water-fill acceptance pass now counts in-flight queries plus newly accepted ones against a chain-wide cap of 100, so chains with many partitions can't admit unbounded concurrent queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfCa3PhypeqQTWSLK1tYhr
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
…aude/waterfill-chain-concurrency-limit-595bfg
DZakh
merged commit Jul 16, 2026
0e1cd57
into
claude/multichain-indexer-query-control-kebggn
15 checks passed
DZakh
added a commit
that referenced
this pull request
Jul 17, 2026
* Make multichain fetch scheduling chain-controlled Replace the per-partition/per-query greedy admission scheduler with a per-chain waterfall: CrossChainState.checkAndFetch visits chains furthest-behind first, handing each its remaining share of the shared buffer budget. ChainState turns that budget into a soft target block using a new chain-wide event density (seeded from cumulative progress, smoothed with an EMA per batch), and FetchState.getNextQuery sizes known-density partitions against that target block while splitting whatever budget is left across partitions with unknown density. This concentrates fetch effort on the bottleneck chain per tick instead of scattering a shared item budget across every chain's full candidate query set. * Fix probe-split eligibility and query ordering in FetchState.getNextQuery The unknown-density probe split counted partitions with nothing left to query (already at their endBlock/mergeBlock/knownHeight ceiling), inflating the divisor and under-sizing eligible partitions' queries. Add a hasEligibleRange check mirroring pushQueriesForRange's own gate to exclude them. Splitting partitions into known/unknown passes also broke the original idsInAscOrder query ordering that several tests assert on positionally; restore it by sorting the final query list back into partition order. Update FetchState_test.res fixtures accordingly, including a case that needed distinct expected values across three eligibility scenarios that previously shared one fixture. * Redesign FetchState.getNextQuery as an even per-partition water-fill Query creation now splits the chain's range budget evenly across in-range partitions each round, rather than sizing every known-density partition against the full chain target while unknown-density partitions fought over the leftover. A partition already holding more budget than its even share (e.g. from an earlier tick's in-flight query) sits out a round so its share flows to the others, and the split is recomputed each round against the shrinking set of partitions still needing more. Also: - Rename estResponseSize -> itemsTarget throughout, since the field is now both the server-side maxNumLogs-style cap and the budget reservation/consumption unit, not just an estimate. - Bucket queries by partition index as they're created instead of sorting the whole result at the end of every tick. - Only trust a partition's density once it has two responses (matching the existing chunking-heuristic gate); a single response is too noisy to size the next query from. - Smooth the chain-wide density EMA as (old + new) / 2 instead of (2*old + new) / 3. * Address review feedback on the water-fill scheduler: dedupe reserved-sum walk, tighten round bound, add coverage - getNextQuery walked every partition's mutPendingQueries twice (once for chainReserved, again to seed reservedByPartition per partition). Merge into a single pass. - Replace the unproven roundsRef < 1000 safety cap with a provable bound: every active partition either finishes or advances its chunk count each round, capped at maxPendingChunksPerPartition, and all active partitions progress in lockstep (not one at a time), so no partition can outlive maxPendingChunksPerPartition + 1 rounds. - Add ChainState_test.res covering the chain density seed (from resumed progress) and the EMA blend. - Add a CrossChainState_test.res case pinning the waterfall's actual cross-chain budget flow: a chain whose real range caps its consumption below its share leaves the remainder for the next chain. * Make the water-fill round's per-partition share order-independent Each round computed ipb = rangeBudget/n once, but then capped every partition's actual budget at min(rangeBudget, ipb - reserved) and decremented rangeBudget after each partition — so a partition processed earlier in the same round (e.g. one forced to overshoot its share via the "at least one full chunk" rule) shrank the pool for whoever came after it. Same reservations, different iteration order, different split (and total consumption could even exceed rangeBudget depending on order). Fix: every partition's share for a round is ipb - reserved, fixed for the whole round; rangeBudget is only re-derived once, from the round's actual total consumption, after every partition has had its fixed shot. A partition can still overshoot its own share, but it can no longer steal from another partition in the same round. Also fixes a SourceManager_test.res assertion that was pinned to the old order-dependent rounding artifact (three identical partitions splitting a budget three ways used to get 16667/16667/16666; they now all get 16667, as they should since they're indistinguishable). * Redistribute a filled partition's leftover budget in the water-fill (#1394) The per-partition round budget was `ipb - reserved`, where `ipb` was an even share of only the *remaining fresh* budget (`rangeBudget / n`) while `reserved` accumulated each partition's full footprint (existing in-flight + gap-fill + this call's prior-round emissions). Those two are on different scales, so once a chunked partition's running reservation passed a later round's fresh share, `ipb - reserved` went negative and the partition was dropped — leaving budget unspent even though it still had range to fetch and a sibling had just freed its share by filling early. Compute the round's level as a real water-fill line — (remaining fresh budget + the still-not-filled partitions' current footprint) / count — and top each partition up toward it. A partition already above the line gets nothing (its head start is its whole share); the rest absorb the leftover, so the budget is fully used and per-partition totals stay even. The loop now runs until either the not-filled set drains or the whole fresh budget is reserved, dropping the redundant round cap: a partition survives a round only by advancing chunksUsedThisCall (bounded by maxPendingChunksPerPartition) or consuming budget, so it terminates on its own. Add a regression test: a range-capped partition and a deep partition splitting a 900-item budget — the deep one now absorbs the capped one's freed share (4 chunks / 720 items) instead of stopping at 2. Claude-Session: https://claude.ai/code/session_0134TTgxQ3ci5mWUnt928yr9 Co-authored-by: Claude <noreply@anthropic.com> * Cap unknown-density probe query at maxItemsTarget An unknown-density partition's open-ended probe was sized to its full even share of the chain's budget with no ceiling. When such a chain leads the furthest-behind waterfall, that share is the entire cross-chain buffer pool, so its single probe consumed 100% of the remaining budget and starved any sibling chain needing its own first probe in the same tick (e.g. multiple chains entering the reorg threshold together). Cap the probe at maxItemsTarget (10_000), restoring the old bounded-default ceiling. The leftover budget flows to the next chain via checkAndFetch's remaining subtraction, exactly as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBSAcMXTNy2K8i16bkJELo * Don't seed chain density from a zero-event batch The resume-seed path only sets chainDensity once numEventsProcessed > 0, but the per-batch EMA update seeded Some(0.) after any progress-only batch (blocks advanced, no events), contradicting that documented behavior and making the first real batch blend against 0 instead of seeding from its own density. Guard the EMA seed on the batch having events, matching the resume path. The Some(oldDensity) blend is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBSAcMXTNy2K8i16bkJELo * Adjust itemsTarget setting logic * Enforce reservation == server cap and stop chunking without trusted density - Floor itemsTarget at 1 at creation (densityItemsTarget, water-fill chunk loop, probe) so a query's budget reservation always equals the maxNumLogs-style cap sent to the server; drop SourceManager's 2000-item fallback that let density-0 queries return up to 2000 unaccounted items. - Emit density-priced chunks only for a trusted positive density; density-0 and unknown-density partitions get a single open-ended probe sized at the even split of the tick's fresh budget (maxItemsTarget cap removed). This removes the chunkCost=0 path that flooded 10 free hard-bounded chunks per partition and froze the 1.8x range growth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Extract getTrustedDensity helper for water-fill chunk sizing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Make query itemsTarget an int and trim redundant comments The ceil-to-int conversion now happens once at query creation, so the reservation, the budget accounting, and the server cap all use the same integer value; SourceManager passes it through untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Price gap-fill by trusted density with available-density fallback Gap queries now use getTrustedDensity: chunks only on a trusted positive density (same rule as the water-fill); a trusted-zero density prices the whole gap as one open query, and a partition with no density signal prices it by available density — its equal-divide budget spread over the remaining range this tick — so a small gap reserves proportionally little instead of a noisy one-sample estimate or a NaN from dividing by a zero range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Cap follower chains at the leader's target progress in the waterfall Chains beyond the most-behind one in the budget waterfall are now capped at that leader's target progress, mapped onto their own block range (ChainState.progressAtBlock/blockAtProgress), so no chain runs further ahead than the chain the shared buffer pool is prioritizing. A chain visited after the pool is exhausted simply sits out the round — its reservations release as responses land, so the next tick redistributes. FetchState's dynamic-contract partition merge now inherits the sum of its parents' trusted densities (weighted onto the merged partition's min query range) instead of resetting to 0, so a merge with density history doesn't regress to an unpriced probe. Update E2E/rollback tests to the now-serialized cross-chain query dispatch (most-behind chain queries first; siblings follow once its response releases budget) and to give density-dependent chunking tests a nonzero item count to trust. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Cap a clamped chain's fresh budget at its density-priced range cost When a chain's target block is clamped (head, endBlock, or the cross-chain alignment cap), a known-density chain's fresh budget is now capped at density x clamped range (in-flight reservations stay on top so they don't crowd out new partitions). The unused remainder stays in the waterfall's pool and flows to the next chain in the same tick, instead of being held by an oversized probe until the response lands. This also removes the drain loop the infinite-reorg-loop test needed: the non-reorg chain's post-rollback refetch now reserves only its real range cost, so the reorg chain gets budget immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Give head-bound queries 2x density headroom in the budget cap A query clamped at the head sized exactly at density x range truncates at the server cap whenever the range is slightly denser than the estimate, forcing an immediate catch-up query for the last few blocks. Double the range cost for head-bound targets so one query usually suffices; the extra reservation releases as soon as the response lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Refine chain budget caps: endBlock ceiling, 5k probe cap, 3x head headroom - targetBlock now clamps at endBlock (when below the head) via a shared fetchCeiling helper, so endBlock'd chains stop sizing and aligning against range they'll never fetch. - A chain with no positive density signal caps its fresh budget at 5k, so one unknown chain measuring its first responses no longer holds the whole cross-chain pool. - Head/endBlock-bound queries get 3x (was 2x) density headroom against truncating at the server cap and needing a catch-up query. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Fix clippy::useless_borrows_in_formatting across cli package Remove redundant & references in format!/anyhow! arguments flagged by the CI-pinned clippy (rust 1.97). Pre-existing on the base branch, unrelated to the SourceManager/waterfall changes in this PR — fixed here since it was blocking cargo-test from going green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw * Fix clippy::to_string_in_format_args exposed by the previous fix Removing the redundant & in anyhow!'s self.id.to_string() surfaced a second lint on the same line: ChainId (u64) already implements Display, so .to_string() inside the format arg is itself redundant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01McpcXkR3pPWfEcq4mCj9Sw --------- Co-authored-by: Claude <noreply@anthropic.com> * Fix clippy useless_borrows_in_formatting errors blocking CI main's cargo-test job started failing clippy (-D warnings) on pre-existing code after a stable-toolchain drift (no rust-toolchain pin), unrelated to this PR's scheduling changes but inherited via the origin/main merge. Removed the redundant `&` in format!/anyhow! args across 6 files, and dropped a now-also-flagged explicit .to_string() on a Display type in validation.rs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBSAcMXTNy2K8i16bkJELo * Pour water-fill budget at an exact level and tighten chain budget edges - Replace the per-round mean line in FetchState.getNextQuery with an exact water level (sum of top-ups equals the poured budget), so uneven in-flight reservations can no longer inflate other partitions' allotments past the fresh budget - Size unknown-density probes by their water-fill allotment instead of a fixed pre-round even split, so leftover budget reaches the partitions without reservations instead of being stranded - Gate the 3x head headroom on the chain having caught up once (isReady) - Blend chain density weighted by the batch's block span instead of a flat (old + new) / 2 - Clamp progressAtBlock at 0 for the initial -1 fetch frontier - Skip chains with no known height in the cross-chain waterfall so they wait for a block instead of setting a degenerate alignment line Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvUhc8DhbxjGPEtHoDNThJ * Shrink density blend window to 100 blocks Small batches (a few blocks) should barely nudge the chain density estimate, while anything spanning 100+ blocks is a trustworthy fresh sample. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvUhc8DhbxjGPEtHoDNThJ * Add chunk headroom multiplier for budget-aware query sizing (#1400) * Add chunk itemsTarget headroom and budget-driven chunk emission Chunk reservations now carry a headroom multiplier over the density estimate (1.5x during backfill, 3x in realtime, chosen in CrossChainState.checkAndFetch and threaded down to FetchState.getNextQuery), so a denser-than-expected range doesn't truncate at the server cap. Open-ended probes stay allotment-sized. The emit loop replaces the precomputed chunkCost/affordable estimate with per-chunk actual itemsTarget accounting: the first chunk always emits full-size, subsequent chunks only while they fit the budget, and the min-one-chunk force applies once per call instead of once per water-fill round. Cap-hit truncations (partial response with itemsCount >= itemsTarget) no longer update the chunk range history — they reflect our own reservation, not server capacity. Sub-cap partials still do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KpFcYPn8UbQfaEfW6gjant * Restore min-one-chunk per water-fill round A leftover re-pour forces a full chunk again, so the budget never strands on chunk quantization; the overshoot stays bounded at one chunk per partition per round and self-corrects via the reported reservations. Drops the per-call emittedThisCall bookkeeping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KpFcYPn8UbQfaEfW6gjant --------- Co-authored-by: Claude <noreply@anthropic.com> * Implement cold-chain targeting and density-aware query sizing (#1401) * Contain queries to the chain target block and rework cold-start sizing - No chunk or gap-fill query starts past chainTargetBlock; emitted chunks keep their full span, with endBlock/mergeBlock staying the hard bounds. Skipped gaps regenerate from the pending-walk and fill once the target reaches them. - A chain with no density signal targets frontier + coldTargetRange (init 20k), doubling whenever it goes idle without producing a signal, capped at the fetch ceiling. The cross-chain waterfall clamps a cold chain to min(5k, targetBufferSize), replacing the internal probe clamp, and a cold leader no longer sets the alignment line. - Query sizing uses effectiveDensity = max(processing EMA, ready-buffer density), so a dense buffer overrides a stale-low EMA and ready items alone take a chain out of cold mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CiEziFbVe1P3Y6iuyfT5Z * Replace cold-window doubling with a fixed 20k range Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CiEziFbVe1P3Y6iuyfT5Z * Span ready-buffer density from the processing block number The buffer is consumed at batch creation while committed progress only catches up after the batch commits, so mid-batch the density's numerator shrank without the denominator following. Track the in-flight batch's progress as processingBlockNumber (advanced in advanceAfterBatch, caught up in applyBatchProgress, rewound on rollback) and use it as the span's lower boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CiEziFbVe1P3Y6iuyfT5Z * Warm the chain with seed events in the partition-merge E2E test A chain with no density signal now targets frontier + 20k, which gates the far DC partitions this test fetches in parallel. Seed 100 events in the registering response so the chain has a density signal and enough range budget for DC2's full 10-chunk pipeline; cold gating itself is covered by unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CiEziFbVe1P3Y6iuyfT5Z --------- Co-authored-by: Claude <noreply@anthropic.com> * Reserve budget at honest itemsEst and tune scheduler defaults (#1403) * Reserve budget at honest itemsEst instead of headroomed itemsTarget Queries now carry both itemsTarget (server-side cap, sized with the chunk headroom multiplier) and itemsEst (raw density estimate). Reservations, pendingBudget, and water-fill footprints use itemsEst, so headroom no longer throttles pipeline depth. The extra 3x budget cap for caught-up chains is dropped — truncation safety lives solely in the itemsTarget cap, keeping realtime headroom at 3x instead of compounding to 9x. Aligned chains may now run 5% past the leader's line to stop clamp flapping when progress tracks closely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyDQ2imktXA4jL8PtWYSmd * Raise default target buffer to 100k and chunk pipeline cap to 12 Measured on the erc20 template against real HyperSync data: at 50k the dense chain's buffer drained to zero in a quarter of samples (processing starved on fetching), while at 100k it almost never does and throughput matches the processing ceiling. Beyond 100k there's no further gain — 300k only grows the resident buffer. The chunk cap rarely binds at 12 but gives the pipeline headroom at the larger budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QyDQ2imktXA4jL8PtWYSmd --------- Co-authored-by: Claude <noreply@anthropic.com> * Fix future end block progress alignment (#1406) * Keep below-head chains polling instead of dropping them (NothingToQuery) At realtime (and during backfill), when one chain falls far behind and its query reservation drains the shared fetch-buffer budget, a chain that is below its own head gets no query this tick. Being below head it also won't wait for a new block, so getNextQuery returns NothingToQuery. checkAndFetch never dispatches NothingToQuery, so that chain stops fetching AND stops polling getHeightOrThrow — its head tracking freezes. This reproduced two production stalls: one right before the indexer enters isReady, and one after isReady having queried only a few items. Dispatch such a chain as WaitingForNewBlock so it keeps polling, mirroring the existing knownHeight == 0 guard. A chain is still left idle (undispatched) when it is genuinely so: caught up to its head/endblock, still draining in-flight queries, or holding ready items that batch processing will drain and re-schedule from. Add an E2E regression test that drives two chains to realtime, then a divergent height jump (leader far ahead, follower just past its own head), and asserts the near-head follower keeps polling getHeightOrThrow while the leader backfills. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBSAcMXTNy2K8i16bkJELo * Extract client-side address filtering from FetchState (#1414) * Filter over-fetched events before contract registration Over-fetched events (a merged partition returning an address before its effectiveStartBlock, or a wildcard param referencing an address registered after the log's block) were running their contractRegister handlers and spawning dynamic contracts before being dropped from the buffer. Apply the client-side address filter to the contract-register set before running the handlers. Extract the predicate as FetchState.filterByClientAddress and expose it through ChainState so ChainFetching can gate registration; the buffer is still filtered after registration in handleQueryResult, so events referencing a contract registered in the same batch keep routing to handlers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xrChwhzSShi39DwN2iKV5 * Move client address filter fully before contract registration Follow-up to the previous commit: instead of only gating the contract-register set, apply the client-side address filter to the whole response up front, so an over-fetched event neither spawns dynamic contracts nor enters the buffer. This is only correct if a non-wildcard event for an address registered in the same batch can't appear before its registration — which a real backend guarantees, since a query only returns logs for the addresses it was sent. The simulate source didn't model that (it dumped every item on the first call), so make it faithful: return only items matching the query's block range, selection, and (for non-wildcard events) address set, delivering each once; wildcards are over-fetched for the client filter to gate, mirroring HyperSync. A contract registered mid-run now surfaces its events in the follow-up query the registration triggers, exactly as in production. Parse simulate items at the process's startBlock (not the config default) so they land in the range the source is queried over. The dead-input tracker stays downstream, observing processed batches, so it still reports items excluded by any filter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xrChwhzSShi39DwN2iKV5 * Merge buffer with a single sort-free pass instead of re-sorting Buffer accumulation re-sorted the whole buffer on every response via Array.sort(compareBufferItem) — an O(n log n) pass whose comparator crosses the JS↔native boundary on each comparison — plus a Set of string keys for dedup. Replace it with mergeIntoBuffer: the buffer is already sorted, so insertion-sort just the (small, usually ascending) response and merge the two runs in one linear pass, dropping duplicates as adjacent-equal. Comparison is inlined (compareBufferItem now returns an int with explicit field compares and a registration-index tiebreaker) with no Array.sort callback and no allocated key. updateInternal assumes a sorted buffer (hot paths pass mutItemsSorted=true) and normalizes arbitrary input otherwise; onBlock items are generated as their own sorted run and merged in the same way. ~14-20x faster on realistic buffers (see packages/envio/bench). Adds a mergeIntoBuffer correctness test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xrChwhzSShi39DwN2iKV5 * Address review: drop bench, single onBlock merge, simplify test helper - Delete the standalone benchmark script. - updateInternal now folds onBlock items into the buffer with a single merge at the end instead of merging mid-function; block items stay their own sorted run so the merge remains linear. - makeInitialWithOnBlock returns the fetch state directly (indexing addresses were unused by every caller). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017xrChwhzSShi39DwN2iKV5 --------- Co-authored-by: Claude <noreply@anthropic.com> * Replace water-fill budget algorithm with greedy fromBlock-sorted pass (#1415) * Cap open-ended probe fan-out in the fetch water-fill When the fresh per-tick budget is thin relative to the number of partitions, the water-fill split gives each partition a sub-item allotment that the open-ended emit floors to a 1-item query, so a single tick fires a burst of near-empty probes and overshoots the budget. Concentrate instead: serve only the neediest ceil(rangeItemsTarget / minQueryItems) probe partitions this tick, each taking a full ~minQueryItems-sized probe, and let the rest wait until freed reservations grow the budget. Chunk partitions self-limit via density-sized chunks and are never capped, so normal fan-out and post-rollback resume are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Keqij8JHea79B2rqJX4ApZ * Select fetch queries by fromBlock against a chain budget Replace the per-partition water-fill (and the earlier probe-fan-out cap) with a single budget pass: 1. Generate every candidate query for the tick with no budget check — gap-fill holes, plus each in-range partition's density-sized chunks or, for an unknown-density partition, one open-ended probe sized to its even share of the fresh budget (freshBudget / inRangeCount). 2. Sort all candidates by fromBlock. 3. Accept them in that order while the budget (chainTargetItems minus in-flight reservations) stays positive; the query that tips it negative is still accepted, everything after it waits for a later tick. Selecting by fromBlock spends the budget on the earliest blocks across all partitions first, so the frontier advances evenly and no partition is starved by iteration order — and gap-fill, chunks, and probes all stop together once the budget is spent. Removes waterLevel and the minQueryItems cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Keqij8JHea79B2rqJX4ApZ * Size open-ended probes by chain density over the range to the target An open-ended probe now reserves chainDensity × (chainTargetBlock − fromBlock + 1) / partitionCount — the events its range to the target is expected to hold, split across partitions — instead of an even share of the fresh budget. ChainState passes its effectiveDensity down for this. When the chain has no density signal, or the partition is already at the target (no range), it falls back to the even budget share so cold chains and caught-up partitions still probe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Keqij8JHea79B2rqJX4ApZ * Size probes by budget-implied density over the in-range coverage Replace the passed-in chainDensity with a rangeTargetDensity derived inside getNextQuery: freshBudget / (chainTargetBlock − frontierCursor + 1), where frontierCursor is the furthest-behind in-range cursor. A probe then reserves rangeTargetDensity × (chainTargetBlock − fromBlock + 1) / inRangeCount, so a partition covering less of the range to the target (it sits further ahead) gets proportionally fewer items, while the furthest- behind partition gets the full even share. Measuring the range from the in-range frontier (not the chain buffer frontier) keeps a lone in-range partition on the full budget instead of having it diluted by out-of-range laggards, and drops the chainDensity parameter ChainState was threading down. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Keqij8JHea79B2rqJX4ApZ * Optimize greedy budget pass: fewer sweeps, bounded generation (#1418) - Fold the chainReserved sum and partitionIndexById build into the Phase A partition sweep (3 full passes over partitions -> 1). - Cap per-partition chunk generation at the fresh budget: a partition can be accepted at most the budget plus one overshoot, so further chunks can never be accepted. Shrinks the candidate set and sort cost when the budget is small relative to the pending-chunk cap. - Acceptance pass: sort candidates in place and stop at the first candidate that can't be accepted, instead of copying via toSorted and scanning the whole tail with forEach. - Hoist the loop-invariant chunk-start ceiling out of the chunk loop. - Rename waterFillState -> partitionFillState (no water-fill left). Claude-Session: https://claude.ai/code/session_01Cj7fN5nh9d2rLeWAXnD1d5 Co-authored-by: Claude <noreply@anthropic.com> * Fix budget deadlock when gap-fill precedes returned query (#1419) * Let gap fills bypass the fresh-budget gate A gap-fill candidate was gated on the fresh forward-progress budget, so a partition could deadlock after a partial/out-of-order chunk: chunk [101,200] returns and lingers in mutPendingQueries behind an unfilled [51,100] hole, its reservation already released by ChainState, yet the FetchState budget sweep still counted it — driving freshBudget to 0 and dropping the [51,100] gap-fill every tick, so the returned query could never be consumed. Fix by budgeting acceptance against the full chainTargetItems and reserving in-flight queries per-query in fromBlock order: a gap-fill, whose fromBlock precedes the query it unblocks, claims budget ahead of that reservation. Returned-but-unconsumed queries (fetchedBlock set) no longer count toward the budget, matching the release ChainState already performed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6hp9FfBBT5insJwt2e28F * Charge same-block reservations before fresh candidates On a fromBlock tie, order in-flight reservations ahead of fresh candidates in the acceptance stream. A same-block candidate could otherwise be emitted while the pool budget was already exhausted (chainTargetItems still carrying pendingBudget), pushing total reserved work past the target buffer. Only a strictly-earlier candidate — a gap-fill preceding the query it unblocks — should borrow ahead of a reservation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6hp9FfBBT5insJwt2e28F --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * Separate event density from source range capacity (#1423) * Separate and smooth per-partition event density (#1426) * Separate event density from source range capacity * Enable strict warning checks in ReScript configurations (#1424) * Treat ReScript warning 23 as an error in indexer configs Promote the "useless record with clause" warning to an error in the generated-project template and the test scenarios. The envio runtime package already errors on all warnings via "+a". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGX7HGV5nCwFHHoazo8dCM * Enforce all ReScript warnings as errors in test scenarios Set warnings.error to "+a" for the test_codegen, fuel_test, and svm_test scenarios, matching the envio runtime package. Leave the user-facing generated-project template without a warnings override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGX7HGV5nCwFHHoazo8dCM --------- Co-authored-by: Claude <noreply@anthropic.com> * Smooth per-partition event density * Fix strict ReScript warnings after main merge * Trust event density independently from source capacity --------- Co-authored-by: Claude <noreply@anthropic.com> * Fix reorg-threshold cross-chain query stall (#1430) * Add PIN test reproducing the below-head chain silence stall Reverts the earlier fix and exploratory tests and pins the exact production stall on the unfixed scheduler: when one chain falls far behind and its query reservation drains the shared fetch-buffer budget, a chain below its own head but starved of budget emits no query and (being below head) won't wait for a new block, so getNextQuery returns NothingToQuery. checkAndFetch never dispatches NothingToQuery, so that chain stops querying AND stops polling getHeightOrThrow and goes silent. The test asserts the correct behavior — the starved below-head follower keeps polling getHeightOrThrow. It is RED on this unfixed scheduler (the follower never re-polls) and turns green once below-head chains are dispatched as WaitingForNewBlock instead of being dropped (the "Keep below-head chains polling" change). Verified red without the change and green with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VhitjdvBNbfY6tRnv6RQcw * Fix reorg-threshold cross-chain query stall * Deduplicate fetch progress calculation --------- Co-authored-by: Claude <noreply@anthropic.com> * Add minimum query admission budget (#1429) * Add minimum query admission budget * Keep block waiters outside query admission * Keep block waiting in query selection * Pause all chain actions below admission floor * Anchor cross-chain alignment to most-behind chain's frontier (#1434) * Anchor cross-chain alignment line at the most-behind chain's frontier The waterfall's alignment line was only established on ticks where the most-behind chain itself emitted a fresh query and had a density signal. While that chain's queries were in flight (or it was still cold), every other chain fetched unclamped to its own head, defeating the cross-chain ordering the line exists for. - Derive the line from the most-behind known-height chain's fetch-frontier progress before dispatching, so it holds on every tick, including ones where the anchor is mid-fetch or cold. - Drop the clamp entirely once the indexer is realtime: chains at head shouldn't be starved behind a chain that temporarily falls behind. - Make progressRange's lower bound the static startBlock: a lower bound at firstEventBlock collapsed a chain's progress to 0 the moment its first event was discovered, yanking other chains' clamp below their frontiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9rjeiYEtjQKRKxCtfFdnR * Clarify why progressRange uses a static startBlock lower bound The comment now names the actual failure mode: a firstEventBlock lower bound makes two chains at the same block read different progress fractions depending on whether each has discovered its first event, so the anchor's alignment line can map below a follower's own frontier and stall it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9rjeiYEtjQKRKxCtfFdnR * Anchor cross-chain alignment by frontier progress and lagged head Two consistency fixes to the alignment anchor, addressing PR review: - Select the anchor as the known-height chain with the lowest fetch-frontier progress — the same metric the clamp maps followers against. Selecting by getProgressPercentage instead let a chain with no discovered first event (which that metric reports as 0%) win the anchor slot while its frontier sat far ahead, so it established a non-clamping line and let genuinely-behind chains run to head. - Make fetchCeiling the fetchable (lagged) head, min(endBlock, knownHeight - blockLag), matching isFetchingAtHead. A chain parked at its lagged head now reads 100% progress instead of looking partially behind against blocks it can't fetch, so it never anchors a spurious below-head clamp. Adds regression tests for both: frontierProgress at the lagged head, and a firstEventBlock=None chain that must not become a dead anchor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9rjeiYEtjQKRKxCtfFdnR * Address review nits: drop dead progressAtBlock export, destructure anchor test - progressAtBlock has no external caller after the alignment rework (only frontierProgress uses it internally); remove it from the interface. - Replace the dead-anchor regression test's hardcoded item counts with a structural check (scanning chain idles, anchor fetches freely, follower held far below the anchor) so it doesn't churn on query-sizing tweaks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9rjeiYEtjQKRKxCtfFdnR * Order the fetch waterfall by frontier progress The alignment anchor is chosen by frontier progress, but the waterfall still drew budget in getProgressPercentage order, so a chain ahead by frontier but behind by event-progress (e.g. a late/absent firstEventBlock) could take budget before the anchor and leave it with NothingToQuery for the tick. Sort priorityOrder by frontierProgress instead. The anchor — the first known-height chain in that order — is now the furthest behind by the same metric the clamp uses and is served first. getProgressPercentage stays the batch-ordering measure, untouched. The anchor selection simplifies back to find-first-in-order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9rjeiYEtjQKRKxCtfFdnR --------- Co-authored-by: Claude <noreply@anthropic.com> * Limit per-chain fetch concurrency to 100 queries (#1440) The water-fill acceptance pass now counts in-flight queries plus newly accepted ones against a chain-wide cap of 100, so chains with many partitions can't admit unbounded concurrent queries. Claude-Session: https://claude.ai/code/session_01NfCa3PhypeqQTWSLK1tYhr Co-authored-by: Claude <noreply@anthropic.com> * Optimize query pipeline and partition reordering (#1443) * Make getNextQuery concurrency-aware in budget split and generation The per-chain concurrency cap only acted as a stop condition in the acceptance walk, so with more partitions than free slots the budget was split across all of them while only a capped subset could be admitted, leaving most of the tick's budget undispatched. Divide the fresh budget by the number of queries the cap can actually admit, skip candidate generation for partitions past that count (sound: their candidates sit behind at least availableConcurrency earlier ones in fromBlock order), and generate nothing when the chain is already at the cap. Also count only still-being-fetched queries against the 12-slot per-partition pipeline cap, so fetched chunks parked behind a slow head query no longer starve the partition's pipeline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KkGzAFGmr8JfU88JiXCDZH * Clean up fetching logic for clarity and reliability FetchState: - Split getNextQuery into named stages: a single partition scan (wait-state bookkeeping + in-flight accounting, previously two passes), walkPartitionPending (gap-fill walk), pushForwardCandidates (forward chunks/probes), and acceptCandidates (budgeted admission). - Deduplicate query pricing into pushDensityPricedQuery and name the 1.8 chunk growth factor (chunkRangeGrowthFactor); drop pushGapFillQueries' unused cost return. - Collapse pushGapFillQueries' zero/unknown-density branches into one available-density fallback. Both are unreachable today (a gap implies pipelined chunks, which imply a positive observed density), but the old zero-density branch would have priced a dense gap at itemsTarget 1. - Rename maxPendingChunksPerPartition to maxInFlightChunksPerPartition to match its semantics. - handleQueryResponse fast path: with no dynamic contracts registered a full OptimizedPartitions.make would only re-sort, so update the one partition in place and restore idsInAscOrder with a rightward walk. - consumeFetchedQueries: replace the shift-per-query loop with a single splice. - Document the mutPendingQueries cross-version mutability invariant. CrossChainState: - Extract idleOrWaitAction and use it on the admission-floor path too, so pool pressure can't freeze an idle chain's head tracking; check knownHeight == 0 before the floor so a heightless chain starts height tracking even while other chains hold the whole pool. SourceManager: - Remove dead inFlightCount (its comment described an indexer-wide concurrency budget that was never implemented) and fix the Querieng typo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KkGzAFGmr8JfU88JiXCDZH * Size probes by the full in-range count, not the concurrency cap Sizing by the admittable-query count let every accepted probe over-fetch its per-partition share, weakening budget control: itemsEst reservations no longer matched what the range would actually cost across ticks. Keep the concurrency-aware truncation purely as a generation bound — pushForwardCandidates now takes the pre-truncation inRangeCount for sizing — and restore the plain partitionsCount divisor for gap-probe pricing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KkGzAFGmr8JfU88JiXCDZH --------- Co-authored-by: Claude <noreply@anthropic.com> * Simplify no-query actions and clamp gap-fill to the lagged head Below the admission floor a saturated pool guarantees a wake-up (some chain holds ready items or in-flight reservations, and batch completions and landing responses both re-enter scheduleFetch), so starved chains no longer issue pointless head polls — only chains without a known height keep polling, since height discovery is their sole way in. That branch also short-circuits to WaitingForNewBlock directly instead of a pretend-zero-budget getNextQuery call. Gap-fill admission now compares against knownHeight - blockLag: blocks past the lagged head can't be queried, matching the ceiling used everywhere else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015NTCdaXe1QwxHs5LrbBR6m * Floor bounded queries' server cap at the per-slot buffer share A bounded query's block range is already the hard bound on its response, so a low density estimate shrinking the itemsTarget cap below the useful size only buys self-truncated responses — each a wasted roundtrip that opens a gap. Floor the cap at targetBufferSize / maxChainConcurrency (1000 items at defaults): even every in-flight bounded query returning a full floored response at once overshoots the pool by at most ~one buffer target. Open-ended queries keep the pure density cap, since there it is the only response bound. itemsEst is untouched, so budget accounting and acceptance are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015NTCdaXe1QwxHs5LrbBR6m --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a global concurrency limit for in-flight queries across all partitions on a single chain to prevent source load spikes on chains with many partitions.
Changes
maxChainConcurrencyconstant set to 100, limiting total parallel queries per chain across all partitionsusedConcurrencyduring query acceptance to count both in-flight queries (fromreservations) and newly accepted queriesfromBlockand later ones can waitImplementation Details
The per-partition cap (
maxPendingChunksPerPartition = 12) alone could admit thousands of concurrent queries on chains with many partitions. The new chain-level cap bounds this by rejecting candidates onceusedConcurrency >= maxChainConcurrency, which is safe because the acceptance stream is ordered byfromBlock— later candidates can be deferred to subsequent ticks.https://claude.ai/code/session_01NfCa3PhypeqQTWSLK1tYhr