fix(store+bridge-monitor): fail-fast on missing blob + sub-50ms histogram buckets#1152
Merged
Conversation
added 2 commits
July 13, 2026 19:24
…he miss (bnb-rpc awaiting incident)
Histogram Buckets grew from [50, 100, 200, 500, 1000, 2000, 5000, 10000] to [10, 25, 50, 100, 200, 500, 1000, 2000, 5000, 10000]. Near Intents on solver-cached corridors (HyperCore in particular) returns via a 1Click coordinator fast path in <50ms, and the previous scheme lumped every one of those observations into (0, 50] so histogram_quantile linearly interpolated to ~25ms regardless of true value. The p50 shown on /benchmarks/bridge-quote-latency?chain=HyperCore was therefore a bucket floor, not a measurement. Sub-50 buckets let the real bimodality surface. Also swap Observe(float64(quoteLatency.Milliseconds())) for Observe(float64(quoteLatency.Nanoseconds()) / 1e6) across all 6 per-bridge observers (across, debridge, lifi, mobula, nearintents, relay). int64 Milliseconds() truncates every sub-1ms round-trip to 0, same bug class as the RPC harness fix in PR #1128. Not a hot path here (network RTT dominates), but keeps precision consistent with the new low buckets and cheap to fix while we are already touching each file. Bench YAML methodology + FAQ updated to reflect the new bucket set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes bundled together on this branch.
1. Spec store miss throws instead of caching (a5eb22e)
Previously, when the materialize store had no blob for a live bench (e.g. bnb-rpc during the awaiting-samples incident), the loader silently cached an empty result and every downstream reader saw a permanent draft state until the cache TTL rolled over. Now the loader throws on miss for live benches, so the ISR regenerator sees the failure, retries, and never poisons the cache with a "no data" sentinel.
2. Bridge-monitor sub-50ms histogram buckets + nanosecond precision (213a2ff)
Buckets grew from `[50, 100, 200, 500, 1000, 2000, 5000, 10000]` to `[10, 25, 50, 100, 200, 500, 1000, 2000, 5000, 10000]`.
Near Intents on solver-cached corridors (HyperCore in particular) returns via the 1Click coordinator fast path in <50ms. The previous scheme lumped every one of those observations into (0, 50] so `histogram_quantile` linearly interpolated to ~25ms regardless of the true value, hiding real bimodality. `/benchmarks/bridge-quote-latency?chain=HyperCore` reported p50 = 30ms which was a bucket floor, not a measurement.
Also swap `Observe(float64(quoteLatency.Milliseconds()))` → `Observe(float64(quoteLatency.Nanoseconds()) / 1e6)` across all 6 per-bridge observers (across, debridge, lifi, mobula, nearintents, relay). Same truncation-to-zero bug class as the RPC harness fix in #1128.
Bench YAML methodology + FAQ updated to match the new bucket set. Container rebuilt + restarted on VPS (`ocb-bridge-monitor`), Prom now emitting `le="10"` and `le="25"` series. 24h until the rolling window is fully populated.
Test plan