From 7a1e476ccb37fdc2b255bf6c4e6a6fe8dda15a36 Mon Sep 17 00:00:00 2001 From: Florent Tapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Wed, 5 Aug 2026 18:11:03 +0200 Subject: [PATCH 1/2] feat: split token-quote availability from route coverage Add token_quote_api_ok_total counter (HTTP 200 regardless of outAmount) and RecordProbeDetailed(apiOk, routeFound) in all 6 provider adapters. YAML gets availability and coverage_given_available query slots per provider. --- benchmarks/token-quote-coverage.yml | 14 ++++++++++--- .../cmd/monitor/jupiter.go | 10 +++++----- .../cmd/monitor/kyberswap.go | 12 +++++------ .../cmd/monitor/metrics.go | 20 ++++++++++++++++--- .../cmd/monitor/mobula.go | 14 ++++++------- .../token-quote-coverage/cmd/monitor/odos.go | 16 +++++++-------- .../cmd/monitor/paraswap.go | 12 +++++------ .../token-quote-coverage/cmd/monitor/relay.go | 12 ++++++----- 8 files changed, 67 insertions(+), 43 deletions(-) diff --git a/benchmarks/token-quote-coverage.yml b/benchmarks/token-quote-coverage.yml index 2bcc7d42..f0c2874c 100644 --- a/benchmarks/token-quote-coverage.yml +++ b/benchmarks/token-quote-coverage.yml @@ -77,7 +77,7 @@ methodology: - "Aggregator endpoints: Jupiter lite-api.jup.ag/swap/v1/quote (Solana), Mobula /api/2/swap/quoting (EVM + Solana, API key), Relay api.relay.link/quote (EVM + Solana, no key), KyberSwap aggregator-api.kyberswap.com/{chain}/api/v1/routes (EVM only, no key), ParaSwap apiv5.paraswap.io/prices (EVM only, no key). Each adapter normalises chain identifiers, input/output decimals, and error response shapes before recording outcome." - "Jupiter strictMode=false: queried without the curated-token filter so Jupiter will attempt routing for any token with a discoverable on-chain pool, matching how most wallets call the API. strictMode=true would trivially exclude new tokens and defeat the purpose of the benchmark." - "Chain-scope enforcement: Jupiter probes Solana-only. KyberSwap and ParaSwap probe EVM only (Base, BNB, Robinhood). Mobula and Relay probe all four chains including Solana. Providers are never penalised for chains they do not support: null cells are excluded from cross-chain aggregates." - - "Coverage metric: Prometheus counters token_quote_coverage_success_total and token_quote_coverage_attempts_total carry labels {provider, venue, chain}. The bench queries compute increase(success[24h]) / increase(attempts[24h]) * 100 live at read time. No pre-aggregation; the ratio reflects exactly the tokens probed in the rolling window." + - "Coverage metric: three Prometheus counters per probe. token_quote_coverage_attempts_total (every probe), token_quote_api_ok_total (HTTP 200 received, regardless of outAmount), and token_quote_coverage_success_total (HTTP 200 + outAmount > 0). The headline coverage rate (p50 slot) remains success / attempts for historical continuity. Two derived slots expose the split: availability = api_ok / attempts (is the API reachable and returning 200?); coverage_given_available = success / api_ok (given the API responded, did it find a route?)." - "Probe cadence and deduplication: each batch cycles through the current Dexscreener token-boosts feed. Tokens are not deduplicated across batches; the same token may appear in multiple 30-minute windows if it remains boosted. The score therefore reflects freshness of the aggregator's current routing graph, not historical coverage depth." - "Single origin: all probes originate from the eu-west OCB VPS. Regional latency is not a factor in this bench (the outcome is binary hit/miss, not latency). No region dimension in v1." @@ -150,6 +150,8 @@ providers: success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="mobula"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h])), 1) sample_size: sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h])) series: sum(rate(token_quote_coverage_success_total{provider="mobula"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="mobula"}[1h])) * 100 + availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="mobula"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h])), 1) * 100 + coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="mobula"}[24h])) / sum(increase(token_quote_api_ok_total{provider="mobula"}[24h])) * 100 - slug: jupiter name: Jupiter @@ -163,6 +165,8 @@ providers: success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="jupiter"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h])), 1) sample_size: sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h])) series: sum(rate(token_quote_coverage_success_total{provider="jupiter"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="jupiter"}[1h])) * 100 + availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="jupiter"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h])), 1) * 100 + coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="jupiter"}[24h])) / sum(increase(token_quote_api_ok_total{provider="jupiter"}[24h])) * 100 - slug: relay name: Relay @@ -176,6 +180,8 @@ providers: success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="relay"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h])), 1) sample_size: sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h])) series: sum(rate(token_quote_coverage_success_total{provider="relay"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="relay"}[1h])) * 100 + availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="relay"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h])), 1) * 100 + coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="relay"}[24h])) / sum(increase(token_quote_api_ok_total{provider="relay"}[24h])) * 100 - slug: kyberswap name: KyberSwap @@ -189,6 +195,8 @@ providers: success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h])), 1) sample_size: sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h])) series: sum(rate(token_quote_coverage_success_total{provider="kyberswap"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="kyberswap"}[1h])) * 100 + availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h])), 1) * 100 + coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_api_ok_total{provider="kyberswap"}[24h])) * 100 - slug: paraswap name: ParaSwap @@ -202,5 +210,5 @@ providers: success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="paraswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h])), 1) sample_size: sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h])) series: sum(rate(token_quote_coverage_success_total{provider="paraswap"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="paraswap"}[1h])) * 100 - - + availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="paraswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h])), 1) * 100 + coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="paraswap"}[24h])) / sum(increase(token_quote_api_ok_total{provider="paraswap"}[24h])) * 100 diff --git a/harnesses/token-quote-coverage/cmd/monitor/jupiter.go b/harnesses/token-quote-coverage/cmd/monitor/jupiter.go index 0f0197e2..4ad1460e 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/jupiter.go +++ b/harnesses/token-quote-coverage/cmd/monitor/jupiter.go @@ -40,14 +40,14 @@ func (p *JupiterProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } resp, err := p.client.Do(req) if err != nil { fmt.Printf("[jupiter] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() @@ -55,7 +55,7 @@ func (p *JupiterProvider) Quote(ctx context.Context, token Token) (ok bool) { if resp.StatusCode != 200 { fmt.Printf("[jupiter] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -64,12 +64,12 @@ func (p *JupiterProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[jupiter] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false) return false } n, _ := strconv.ParseInt(r.OutAmount, 10, 64) ok = n > 0 - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok) return ok } diff --git a/harnesses/token-quote-coverage/cmd/monitor/kyberswap.go b/harnesses/token-quote-coverage/cmd/monitor/kyberswap.go index fdae51b9..eb404c6b 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/kyberswap.go +++ b/harnesses/token-quote-coverage/cmd/monitor/kyberswap.go @@ -43,7 +43,7 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { slug = "robinhood" usdcAddr = kyberUSDGRobinhood default: - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -56,7 +56,7 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } req.Header.Set("x-client-id", "openchainbench") @@ -64,7 +64,7 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { resp, err := p.client.Do(req) if err != nil { fmt.Printf("[kyberswap] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() @@ -72,7 +72,7 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { if resp.StatusCode != 200 { fmt.Printf("[kyberswap] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -85,11 +85,11 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[kyberswap] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false) return false } ok = r.Data.RouteSummary.AmountOut != "" && r.Data.RouteSummary.AmountOut != "0" - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok) return ok } diff --git a/harnesses/token-quote-coverage/cmd/monitor/metrics.go b/harnesses/token-quote-coverage/cmd/monitor/metrics.go index af183f0f..c1766571 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/metrics.go +++ b/harnesses/token-quote-coverage/cmd/monitor/metrics.go @@ -10,6 +10,7 @@ import ( var ( coverageSuccess *prometheus.CounterVec coverageAttempts *prometheus.CounterVec + coverageAPIok *prometheus.CounterVec ) func init() { @@ -30,12 +31,25 @@ func init() { []string{"provider", "venue", "chain"}, ) prometheus.MustRegister(coverageAttempts) + + coverageAPIok = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "token_quote_api_ok_total", + Help: "HTTP 200 responses received per provider/venue/chain, regardless of outAmount", + }, + []string{"provider", "venue", "chain"}, + ) + prometheus.MustRegister(coverageAPIok) } -// RecordProbe records one probe attempt and optionally a success. -func RecordProbe(provider, venue, chain string, ok bool) { +// RecordProbeDetailed records one probe: apiOk=true when the provider returned HTTP 200, +// routeFound=true when outAmount > 0 was parsed from the response. +func RecordProbeDetailed(provider, venue, chain string, apiOk, routeFound bool) { coverageAttempts.WithLabelValues(provider, venue, chain).Inc() - if ok { + if apiOk { + coverageAPIok.WithLabelValues(provider, venue, chain).Inc() + } + if routeFound { coverageSuccess.WithLabelValues(provider, venue, chain).Inc() } } diff --git a/harnesses/token-quote-coverage/cmd/monitor/mobula.go b/harnesses/token-quote-coverage/cmd/monitor/mobula.go index 8ad927de..082de4cb 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/mobula.go +++ b/harnesses/token-quote-coverage/cmd/monitor/mobula.go @@ -54,7 +54,7 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) { case "solana": chainId, tokenIn, wallet = "solana", mobulaUSDCSolana, mobulaSolanaWallet default: - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -72,7 +72,7 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } req.Header.Set("Authorization", p.apiKey) @@ -80,7 +80,7 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) { resp, err := p.client.Do(req) if err != nil { fmt.Printf("[mobula] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() @@ -88,7 +88,7 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) { if resp.StatusCode != 200 { fmt.Printf("[mobula] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -100,18 +100,18 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[mobula] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false) return false } // For Solana: Mobula finds the route but fails to build the tx because the // dummy wallet has no SOL for fees. Route found = coverage HIT. if token.Chain == "solana" && r.Error == "Transaction build failed" { - RecordProbe(p.Slug(), token.Venue, token.Chain, true) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, true) return true } ok = r.Data.AmountOutTokens != "" && r.Data.AmountOutTokens != "0" - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok) return ok } diff --git a/harnesses/token-quote-coverage/cmd/monitor/odos.go b/harnesses/token-quote-coverage/cmd/monitor/odos.go index 8dae9538..2883f7bf 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/odos.go +++ b/harnesses/token-quote-coverage/cmd/monitor/odos.go @@ -37,7 +37,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { case p.sem <- struct{}{}: defer func() { <-p.sem }() case <-ctx.Done(): - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -54,7 +54,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { chainId = 4663 usdcAddr = odosUSDGRobinhood default: - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -73,7 +73,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "POST", "https://api.odos.xyz/sor/quote/v2", bytes.NewBuffer(bb)) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } req.Header.Set("Content-Type", "application/json") @@ -81,7 +81,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { resp, err := p.client.Do(req) if err != nil { fmt.Printf("[odos] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() @@ -89,12 +89,12 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { if resp.StatusCode == 429 { fmt.Printf("[odos] %s/%s throttled\n", token.Chain, token.Address) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } if resp.StatusCode != 200 { fmt.Printf("[odos] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -103,11 +103,11 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[odos] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false) return false } ok = len(r.OutAmounts) > 0 && r.OutAmounts[0] != "" && r.OutAmounts[0] != "0" - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok) return ok } diff --git a/harnesses/token-quote-coverage/cmd/monitor/paraswap.go b/harnesses/token-quote-coverage/cmd/monitor/paraswap.go index 0be207c7..5d64f220 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/paraswap.go +++ b/harnesses/token-quote-coverage/cmd/monitor/paraswap.go @@ -42,7 +42,7 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { network = "4663" usdcAddr = paraswapUSDGRobinhood default: - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -58,14 +58,14 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } resp, err := p.client.Do(req) if err != nil { fmt.Printf("[paraswap] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() @@ -73,7 +73,7 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { if resp.StatusCode != 200 { fmt.Printf("[paraswap] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -84,11 +84,11 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[paraswap] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false) return false } ok = r.PriceRoute.DestAmount != "" && r.PriceRoute.DestAmount != "0" - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok) return ok } diff --git a/harnesses/token-quote-coverage/cmd/monitor/relay.go b/harnesses/token-quote-coverage/cmd/monitor/relay.go index 62dae5a0..f5e45423 100644 --- a/harnesses/token-quote-coverage/cmd/monitor/relay.go +++ b/harnesses/token-quote-coverage/cmd/monitor/relay.go @@ -49,7 +49,7 @@ func (p *RelayProvider) Quote(ctx context.Context, token Token) (ok bool) { case "robinhood": chainId, tokenIn, user = 4663, relayUSDGRobinhood, relayEVMUser default: - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } @@ -65,7 +65,7 @@ func (p *RelayProvider) Quote(ctx context.Context, token Token) (ok bool) { req, err := http.NewRequestWithContext(ctx, "POST", "https://api.relay.link/quote", bytes.NewReader(payload)) if err != nil { - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } req.Header.Set("Content-Type", "application/json") @@ -73,12 +73,14 @@ func (p *RelayProvider) Quote(ctx context.Context, token Token) (ok bool) { resp, err := p.client.Do(req) if err != nil { fmt.Printf("[relay] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err)) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false) return false } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) + apiOk := resp.StatusCode == 200 + var r struct { Details struct { CurrencyOut struct { @@ -89,7 +91,7 @@ func (p *RelayProvider) Quote(ctx context.Context, token Token) (ok bool) { } if err := json.Unmarshal(body, &r); err != nil { fmt.Printf("[relay] %s/%s parse error: %v\n", token.Chain, token.Address, err) - RecordProbe(p.Slug(), token.Venue, token.Chain, false) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, apiOk, false) return false } @@ -97,6 +99,6 @@ func (p *RelayProvider) Quote(ctx context.Context, token Token) (ok bool) { if !ok && r.Message != "" { fmt.Printf("[relay] %s/%s: %s\n", token.Chain, token.Address, r.Message) } - RecordProbe(p.Slug(), token.Venue, token.Chain, ok) + RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, apiOk, ok) return ok } From 8c89846526e4a8edf4b1678aec9d64667e5bdb9f Mon Sep 17 00:00:00 2001 From: Florent Tapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Wed, 5 Aug 2026 18:25:18 +0200 Subject: [PATCH 2/2] fix: data-api report corrections + drop duplicate citation footer - publishedAt 2026-08-04 -> 2026-08-05 - Mobula aggregate 677 -> 678 ms (2 occurrences: table + scorecard) - GeckoTerminal aggregate 12,340 -> 12,319 ms (3 occurrences) - StatTable caption: p50 -> mean of per-chain p50s - Removed speculative Mobula Solana mechanism sentence; replaced with caveated explanation noting reorg exposure - 8.2x Mobula-Base gap -> 7.9x (773/98) - RH Chain 4x faster blocks -> 2x faster block cadence (Base 200ms/RH 100ms) - 28 pp wallet-labels gap -> 24 pp (79.9 - 55.6 = 24.3) - Quote section: removed 'entire cohort' claim; fixed Mobula framing to reference availability confound instead of 29% miss rate - Methodology: removed dangling Singapore regional-table reference - coverage_given_success -> coverage_given_available (correct metric name) - Removed auto-rendered duplicate citation block from page.tsx layout --- src/app/reports/[category]/[slug]/page.tsx | 21 +------------ .../2026-08-state-of-crypto-data-apis.mdx | 30 +++++++++---------- 2 files changed, 16 insertions(+), 35 deletions(-) diff --git a/src/app/reports/[category]/[slug]/page.tsx b/src/app/reports/[category]/[slug]/page.tsx index ac41b2a7..9baf0893 100644 --- a/src/app/reports/[category]/[slug]/page.tsx +++ b/src/app/reports/[category]/[slug]/page.tsx @@ -407,26 +407,7 @@ export default async function ReportPage({ params }: Props) { - {/* Citation */} -
-

- Cite this report -

-

- OpenChainBench Research. {report.title}. OpenChainBench,{" "} - {new Date(report.publishedAt).toLocaleDateString("en-US", { - year: "numeric", - month: "long", - day: "numeric", - timeZone: "UTC", - })} - .{" "} - - {report.canonical} - - . Licensed under CC BY 4.0. -

-
+ ); } diff --git a/src/content/reports/data-api/2026-08-state-of-crypto-data-apis.mdx b/src/content/reports/data-api/2026-08-state-of-crypto-data-apis.mdx index 71ef6c85..2285f377 100644 --- a/src/content/reports/data-api/2026-08-state-of-crypto-data-apis.mdx +++ b/src/content/reports/data-api/2026-08-state-of-crypto-data-apis.mdx @@ -2,7 +2,7 @@ title: "Best Crypto Data API 2026: Price Feeds, Coverage, and Chain Breadth Ranked" category: "data-api" slug: "2026-08-state-of-crypto-data-apis" -publishedAt: "2026-08-04" +publishedAt: "2026-08-05" period: "August 2026" summary: "Seven live benchmarks across seven categories show a market where no two category leaders are the same provider. This report maps where each provider wins, where it falls short, and what the data actually says about causation." heroFinding: "GeckoTerminal indexes 253 blockchains for DEX data but publishes prices 12 seconds after they move. The fastest price aggregators publish in under one second on some chains, but the driver is integration depth, not the chain's block time. Seven benchmarks, seven different category leaders, zero overlap." @@ -18,7 +18,7 @@ canonical: "https://openchainbench.com/reports/data-api/2026-08-state-of-crypto- - Token metadata coverage: the top two providers are within 0.6 pp of each other and both under 65%. More than one in three newly-launched tokens are missing at least one metadata field across all providers. - Asset registry breadth: CoinGecko (461 chains) leads CoinPaprika (307 chains) by 1.5x and Mobula (81 chains) by 5.7x. - NFT metadata: the gap between leader Moralis (97.1%) and Alchemy (74.3%) is the largest between a cohort leader and a major general-purpose incumbent in any category in this report. -- Wallet labeling: chain-native specialists (Helius 84.1%, XRPScan 80.0%, StellarExpert 79.9%) outpace every multi-chain provider by more than 28 percentage points. +- Wallet labeling: chain-native specialists (Helius 84.1%, XRPScan 80.0%, StellarExpert 79.9%) outpace every multi-chain provider by more than 24 percentage points. @@ -31,7 +31,7 @@ Every number in this report is derived from OpenChainBench's live Prometheus ins Benchmarks in scope: [aggregator-head-lag](/benchmarks/aggregator-head-lag), [metadata-coverage](/benchmarks/metadata-coverage), [asset-registry-coverage](/benchmarks/asset-registry-coverage), [token-quote-coverage](/benchmarks/token-quote-coverage), [wallet-labels-coverage](/benchmarks/wallet-labels-coverage), [dex-network-coverage](/benchmarks/dex-network-coverage), [nft-collection-metadata](/benchmarks/nft-collection-metadata). -The aggregator head-lag harness samples every 15 seconds from three geographic regions (US East, EU West, Singapore) across four chains (Solana, Base, Robinhood Chain, BNB). The cross-chain headline is the cross-region average of per-chain p50s; Singapore contributes to the global aggregate but is not broken out separately in the regional table. Coverage benches run every 30 minutes to every 6 hours depending on bench. All harnesses are open source at [github.com/ChainBench/OpenChainBench](https://github.com/ChainBench/OpenChainBench/tree/main/harnesses). +The aggregator head-lag harness samples every 15 seconds from three geographic regions (US East, EU West, Singapore) across four chains (Solana, Base, Robinhood Chain, BNB). The cross-chain headline is the cross-region average of per-chain p50s. Coverage benches run every 30 minutes to every 6 hours depending on bench. All harnesses are open source at [github.com/ChainBench/OpenChainBench](https://github.com/ChainBench/OpenChainBench/tree/main/harnesses). ## The Seven-Category Divide @@ -41,7 +41,7 @@ Category leaders as of August 5, 2026: | Category | Benchmark | Leader | Value | |---|---|---|---| -| Price Feeds | aggregator-head-lag | Mobula | 677 ms | +| Price Feeds | aggregator-head-lag | Mobula | 678 ms | | Token Metadata | metadata-coverage | Codex | 64.3% | | Asset Registry | asset-registry-coverage | CoinGecko | 461 chains | | Token Quotes | token-quote-coverage | Jupiter | 97.1% | @@ -53,11 +53,11 @@ This fragmentation is structural, not accidental. Price feed freshness, asset re ## Price Feed Head Lag - + -The cross-chain headline is the cross-region average of per-chain p50s: Mobula 677 ms, Codex 1,059 ms, GeckoTerminal 12,340 ms as of the snapshot date. The ratio between chains matters far more than the single aggregate. The next section breaks down each chain individually. +The cross-chain headline is the cross-region average of per-chain p50s: Mobula 678 ms, Codex 1,059 ms, GeckoTerminal 12,319 ms as of the snapshot date. The ratio between chains matters far more than the single aggregate. The next section breaks down each chain individually. -GeckoTerminal is in a separate category entirely. Its p50 of 12,340 ms is not a latency ranking failure. It reflects a batch-sync pipeline architecture that enables 253-chain DEX coverage, the widest in the industry. GeckoTerminal does not compete as a real-time price feed and its benchmark position reflects that trade-off, not a product deficiency. +GeckoTerminal is in a separate category entirely. Its aggregate head lag of 12,319 ms is not a latency ranking failure. It reflects a batch-sync pipeline architecture that enables 253-chain DEX coverage, the widest in the industry. GeckoTerminal does not compete as a real-time price feed and its benchmark position reflects that trade-off, not a product deficiency. If a 12-second delay between an on-chain price event and your UI is visible to users, GeckoTerminal's price endpoint is not viable for that use case. For DEX pool metadata, pool history, or multi-chain analytics where lag is irrelevant, GeckoTerminal is the strongest option. @@ -80,11 +80,11 @@ The block time column makes the causal claim uncomfortable: block speed an The ranking (Solana fastest, BNB slowest) is the inverse of what a block-time explanation predicts. The data points instead to integration depth: how each provider subscribes to on-chain events on that specific chain. -Mobula's 98 ms on Solana is consistent with a native program subscription to Solana's transaction or vote stream, receiving data pre-block-finalization through Solana's streaming RPC. Its EVM chain lags (773 to 961 ms) are consistent with block-polling, waiting for a full block header then resolving prices. The 8.2x gap between Solana and Mobula's next-fastest chain (Base) is not a Solana property. It is the signal of different integration investments per chain. +One explanation consistent with the data (we have not verified provider internals and have invited comment) is a native subscription to Solana's streaming RPC, emitting before full block finalization. If so, the latency advantage carries reorg exposure that a post-finality pipeline does not. Its EVM chain lags (773 to 961 ms) are consistent with block-polling, waiting for a full block header then resolving prices. The 7.9x gap between Solana and Mobula's next-fastest chain (Base) is not a Solana property. It is the signal of different integration investments per chain. Codex shows the same structural pattern at different absolute values. Base at 831 ms is close to Mobula; Solana at 808 ms is 8.2x Mobula's Solana figure. Codex's Solana integration extracts less of the chain's latency advantage. -Robinhood Chain carries an additional caveat: it launched July 1, 2026, 34 days before this snapshot. Both Mobula (881 ms) and Codex (1,106 ms) lag behind Base despite RH Chain's 4x faster blocks, which is consistent with early-stage EVM-polling integrations not yet optimized for RH Chain's sub-second slot cadence. Treat the RH Chain figures as initial baselines, not settled rankings. +Robinhood Chain carries an additional caveat: it launched July 1, 2026, 34 days before this snapshot. Both Mobula (881 ms) and Codex (1,106 ms) lag behind Base despite RH Chain's 2x faster block cadence, which is consistent with early-stage EVM-polling integrations not yet optimized for RH Chain's sub-second slot cadence. Treat the RH Chain figures as initial baselines, not settled rankings. ## Token Metadata Coverage @@ -112,9 +112,9 @@ If you need to answer "does this contract exist on chain X", CoinGecko's registr The token quote bench tests providers on tokens created within the last hour, sourced from live launchpad feeds. This is the hardest case for routing coverage: a token may have no liquidity on major venues, no metadata, and may exist only on a single chain. -**Methodological note.** The bench currently records a hit when the API responds with HTTP 2xx and a non-zero `outAmount`. This means the score reflects API success rate as much as actual routing coverage, and the two cannot be separated in the current implementation. A provider with a 10% error rate is capped at 90% regardless of its routing graph. A future iteration will publish `availability` and `coverage_given_success` as separate columns. Read the figures below with this in mind. +**Methodological note.** The bench currently records a hit when the API responds with HTTP 2xx and a non-zero `outAmount`. This means the score reflects API success rate as much as actual routing coverage, and the two cannot be separated in the current implementation. A provider with a 10% error rate is capped at 90% regardless of its routing graph. A future iteration will publish `availability` and `coverage_given_available` as separate columns. Read the figures below with this in mind. -Jupiter at 97.1% and Moralis NFT at 97.1% share the highest figure in the entire cohort across all seven benchmarks. Jupiter's score reflects its native integration with Solana's pool infrastructure. It discovers new pools seconds after creation. KyberSwap at 87.8% covers EVM chains competently. Mobula at 71.3% means roughly one in three newly-launched tokens across chains cannot be quoted, though some of that gap may reflect API availability rather than missing routing paths. +Jupiter at 97.1% and Moralis NFT at 97.1% share the highest coverage figure among the percentage-based benchmarks. Jupiter's score reflects its native integration with Solana's pool infrastructure. It discovers new pools seconds after creation. KyberSwap at 87.8% covers EVM chains competently. Mobula at 71.3% covers more chains with the lowest score in the bench, though the availability confound above means this is an upper bound on its true routing gap. For applications handling established tokens (top-1,000 by market cap), all three providers will perform near 100%. The quote-coverage bench is specifically relevant for launchpad analytics, meme-token apps, or any product that needs to quote tokens within minutes of creation. @@ -144,7 +144,7 @@ For builders needing collection metadata plus floor prices from a single call, M -The wallet labeling bench has the most providers of any category in this report: nine across eleven chains. Chain-native specialists occupy the top three positions by more than 28 percentage points over the next tier. +The wallet labeling bench has the most providers of any category in this report: nine across eleven chains. Chain-native specialists occupy the top three positions by more than 24 percentage points over the next tier. | Provider | Coverage | Primary Chain | |---|---:|---| @@ -170,9 +170,9 @@ TonAPI at 35.6% reflects TON's still-developing ecosystem tooling. WalletExplore Across seven benchmarks, the competitive landscape resolves into four archetypes. -**Speed specialists** optimize for real-time data at the cost of breadth. Mobula (677 ms cross-chain average, 98 ms on Solana) leads the price feed category. Codex is competitive on Base (831 ms, within 60 ms of Mobula) while trailing significantly on Solana. +**Speed specialists** optimize for real-time data at the cost of breadth. Mobula (678 ms cross-chain average, 98 ms on Solana) leads the price feed category. Codex is competitive on Base (831 ms, within 60 ms of Mobula) while trailing significantly on Solana. -**Coverage maximalists** maximize breadth at the cost of freshness. GeckoTerminal (253 DEX chains, 12,340 ms head lag) and CoinGecko (461 asset registry chains, no real-time price bench) define this archetype. Their value is depth of catalog, not speed of emission. +**Coverage maximalists** maximize breadth at the cost of freshness. GeckoTerminal (253 DEX chains, 12,319 ms head lag) and CoinGecko (461 asset registry chains, no real-time price bench) define this archetype. Their value is depth of catalog, not speed of emission. **Vertical specialists** dominate a single chain or use case. Jupiter (97.1% Solana quote coverage), Helius (84.1% Solana wallet labels), and Moralis (97.1% NFT metadata) each lead in a category where their infrastructure or editorial investment confers a structural advantage. @@ -185,7 +185,7 @@ Price freshness is the primary constraint. Check the per-chain breakdown on the -Quote coverage on fresh tokens is the constraint. Jupiter is the only viable choice for Solana launchpad tokens (97.1%). For EVM chains, KyberSwap (87.8%) leads; Mobula (71.3%) covers more chains at the cost of a roughly 29% miss rate on brand-new tokens. Build a fallback path: no single provider routes every freshly launched token across all chains. +Quote coverage on fresh tokens is the constraint. Jupiter is the only viable choice for Solana launchpad tokens (97.1%). For EVM chains, KyberSwap (87.8%) leads; Mobula (71.3%) covers more chains with the lowest score in the bench, though the availability confound above means this is an upper bound on its true routing gap. Build a fallback path: no single provider routes every freshly launched token across all chains.