rpc-keyed harness: record sub-millisecond latency without truncation#1128
Merged
Conversation
time.Duration.Milliseconds() returns int64 and truncates every probe under 1 ms to zero. QuickNode is measured via the Mobula shared paid fleet whose endpoints are peer-adjacent to the harness (short RTT + HTTP keep-alive), so a significant share of successful probes returned in ~0.3 to 0.9 ms and were recorded as 0 ms in Prometheus. The 24h aggregate on the BNB + US-East cell landed at effectively 0 ms and rendered as 'QuickNode leads at 0 ms' on /benchmarks/rpc-keyed-latency?chain=bnb®ion=us-east. Fix: latencyMs = float64(time.Since(start).Nanoseconds()) / 1e6. Same wall-clock delta, real precision preserved down to microsecond granularity. Every other provider stays within noise (their probes were already above 1 ms so the truncation never bit them). Requires a Railway harness rebuild + redeploy to take effect on staging and production (bench 069 runs on Railway, not the shared worker VPS). The materialized store will catch up on the next 24h window after redeploy.
Flotapponnier
added a commit
that referenced
this pull request
Jul 12, 2026
…1128) time.Duration.Milliseconds() returns int64 and truncates every probe under 1 ms to zero. QuickNode is measured via the Mobula shared paid fleet whose endpoints are peer-adjacent to the harness (short RTT + HTTP keep-alive), so a significant share of successful probes returned in ~0.3 to 0.9 ms and were recorded as 0 ms in Prometheus. The 24h aggregate on the BNB + US-East cell landed at effectively 0 ms and rendered as 'QuickNode leads at 0 ms' on /benchmarks/rpc-keyed-latency?chain=bnb®ion=us-east. Fix: latencyMs = float64(time.Since(start).Nanoseconds()) / 1e6. Same wall-clock delta, real precision preserved down to microsecond granularity. Every other provider stays within noise (their probes were already above 1 ms so the truncation never bit them). Requires a Railway harness rebuild + redeploy to take effect on staging and production (bench 069 runs on Railway, not the shared worker VPS). The materialized store will catch up on the next 24h window after redeploy. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Flotapponnier
pushed a commit
that referenced
this pull request
Jul 13, 2026
Histogram Buckets grew from [50, 100, 200, 500, 1000, 2000, 5000, 10000] to [10, 25, 50, 100, 200, 500, 1000, 2000, 5000, 10000]. Near Intents on solver-cached corridors (HyperCore in particular) returns via a 1Click coordinator fast path in <50ms, and the previous scheme lumped every one of those observations into (0, 50] so histogram_quantile linearly interpolated to ~25ms regardless of true value. The p50 shown on /benchmarks/bridge-quote-latency?chain=HyperCore was therefore a bucket floor, not a measurement. Sub-50 buckets let the real bimodality surface. Also swap Observe(float64(quoteLatency.Milliseconds())) for Observe(float64(quoteLatency.Nanoseconds()) / 1e6) across all 6 per-bridge observers (across, debridge, lifi, mobula, nearintents, relay). int64 Milliseconds() truncates every sub-1ms round-trip to 0, same bug class as the RPC harness fix in PR #1128. Not a hot path here (network RTT dominates), but keeps precision consistent with the new low buckets and cheap to fix while we are already touching each file. Bench YAML methodology + FAQ updated to reflect the new bucket set.
6 tasks
Flotapponnier
added a commit
that referenced
this pull request
Jul 13, 2026
…gram buckets (#1152) * spec: missing store blob for a live bench throws instead of caching the miss (bnb-rpc awaiting incident) * bridge-monitor: sub-50ms buckets + nanosecond precision on Observe Histogram Buckets grew from [50, 100, 200, 500, 1000, 2000, 5000, 10000] to [10, 25, 50, 100, 200, 500, 1000, 2000, 5000, 10000]. Near Intents on solver-cached corridors (HyperCore in particular) returns via a 1Click coordinator fast path in <50ms, and the previous scheme lumped every one of those observations into (0, 50] so histogram_quantile linearly interpolated to ~25ms regardless of true value. The p50 shown on /benchmarks/bridge-quote-latency?chain=HyperCore was therefore a bucket floor, not a measurement. Sub-50 buckets let the real bimodality surface. Also swap Observe(float64(quoteLatency.Milliseconds())) for Observe(float64(quoteLatency.Nanoseconds()) / 1e6) across all 6 per-bridge observers (across, debridge, lifi, mobula, nearintents, relay). int64 Milliseconds() truncates every sub-1ms round-trip to 0, same bug class as the RPC harness fix in PR #1128. Not a hot path here (network RTT dominates), but keeps precision consistent with the new low buckets and cheap to fix while we are already touching each file. Bench YAML methodology + FAQ updated to reflect the new bucket set. --------- Co-authored-by: Florent Tapponnier <contact@mobula.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
time.Duration.Milliseconds()returns int64 and truncates every probe under 1 ms to zero. QuickNode is measured via the Mobula shared paid fleet whose endpoints are peer-adjacent to the harness (short RTT + HTTP keep-alive), so a significant share of successful probes returned in ~0.3 to 0.9 ms and were recorded as 0 ms in Prometheus. The 24h aggregate on the BNB + US-East cell landed at effectively 0 ms and rendered as QuickNode leads at 0 ms on the UI.Fix
Same wall-clock delta, real precision preserved down to microsecond granularity. Every other provider (Alchemy, Chainstack, Ankr, Helius, Infura) was already above 1 ms because their endpoints are not peer-adjacent to the harness, so the truncation never bit them and this fix does not change their numbers within noise.
Deploy sequence
?chain=bnb®ion=us-eastwill heal over the following window.Verified
go build ./...inharnesses/rpc-keyed-latency: clean.Milliseconds()call in the harness was the one ontime.Since; the metrics client is unaffectedTest plan
ocb:rpc_latency_milliseconds:p50_24h{provider="quicknode", chain="bnb", region="us-east"}on Prom, verify the value climbs off 0vercel --prod