Skip to content

feat: split token-quote availability from route coverage #1787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 2 commits into from
Aug 5, 2026
Merged

feat: split token-quote availability from route coverage #1787

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a1e476
feat: split token-quote availability from route coverage
Flotapponnier Aug 5, 2026
8c89846
fix: data-api report corrections + drop duplicate citation footer
Flotapponnier Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions benchmarks/token-quote-coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ methodology:
- "Aggregator endpoints: Jupiter lite-api.jup.ag/swap/v1/quote (Solana), Mobula /api/2/swap/quoting (EVM + Solana, API key), Relay api.relay.link/quote (EVM + Solana, no key), KyberSwap aggregator-api.kyberswap.com/{chain}/api/v1/routes (EVM only, no key), ParaSwap apiv5.paraswap.io/prices (EVM only, no key). Each adapter normalises chain identifiers, input/output decimals, and error response shapes before recording outcome."
- "Jupiter strictMode=false: queried without the curated-token filter so Jupiter will attempt routing for any token with a discoverable on-chain pool, matching how most wallets call the API. strictMode=true would trivially exclude new tokens and defeat the purpose of the benchmark."
- "Chain-scope enforcement: Jupiter probes Solana-only. KyberSwap and ParaSwap probe EVM only (Base, BNB, Robinhood). Mobula and Relay probe all four chains including Solana. Providers are never penalised for chains they do not support: null cells are excluded from cross-chain aggregates."
- "Coverage metric: Prometheus counters token_quote_coverage_success_total and token_quote_coverage_attempts_total carry labels {provider, venue, chain}. The bench queries compute increase(success[24h]) / increase(attempts[24h]) * 100 live at read time. No pre-aggregation; the ratio reflects exactly the tokens probed in the rolling window."
- "Coverage metric: three Prometheus counters per probe. token_quote_coverage_attempts_total (every probe), token_quote_api_ok_total (HTTP 200 received, regardless of outAmount), and token_quote_coverage_success_total (HTTP 200 + outAmount > 0). The headline coverage rate (p50 slot) remains success / attempts for historical continuity. Two derived slots expose the split: availability = api_ok / attempts (is the API reachable and returning 200?); coverage_given_available = success / api_ok (given the API responded, did it find a route?)."
- "Probe cadence and deduplication: each batch cycles through the current Dexscreener token-boosts feed. Tokens are not deduplicated across batches; the same token may appear in multiple 30-minute windows if it remains boosted. The score therefore reflects freshness of the aggregator's current routing graph, not historical coverage depth."
- "Single origin: all probes originate from the eu-west OCB VPS. Regional latency is not a factor in this bench (the outcome is binary hit/miss, not latency). No region dimension in v1."

Expand Down Expand Up @@ -150,6 +150,8 @@ providers:
success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="mobula"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h])), 1)
sample_size: sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h]))
series: sum(rate(token_quote_coverage_success_total{provider="mobula"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="mobula"}[1h])) * 100
availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="mobula"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="mobula"}[24h])), 1) * 100
coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="mobula"}[24h])) / sum(increase(token_quote_api_ok_total{provider="mobula"}[24h])) * 100

- slug: jupiter
name: Jupiter
Expand All @@ -163,6 +165,8 @@ providers:
success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="jupiter"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h])), 1)
sample_size: sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h]))
series: sum(rate(token_quote_coverage_success_total{provider="jupiter"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="jupiter"}[1h])) * 100
availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="jupiter"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="jupiter"}[24h])), 1) * 100
coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="jupiter"}[24h])) / sum(increase(token_quote_api_ok_total{provider="jupiter"}[24h])) * 100

- slug: relay
name: Relay
Expand All @@ -176,6 +180,8 @@ providers:
success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="relay"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h])), 1)
sample_size: sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h]))
series: sum(rate(token_quote_coverage_success_total{provider="relay"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="relay"}[1h])) * 100
availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="relay"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="relay"}[24h])), 1) * 100
coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="relay"}[24h])) / sum(increase(token_quote_api_ok_total{provider="relay"}[24h])) * 100

- slug: kyberswap
name: KyberSwap
Expand All @@ -189,6 +195,8 @@ providers:
success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h])), 1)
sample_size: sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h]))
series: sum(rate(token_quote_coverage_success_total{provider="kyberswap"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="kyberswap"}[1h])) * 100
availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="kyberswap"}[24h])), 1) * 100
coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="kyberswap"}[24h])) / sum(increase(token_quote_api_ok_total{provider="kyberswap"}[24h])) * 100

- slug: paraswap
name: ParaSwap
Expand All @@ -202,5 +210,5 @@ providers:
success: clamp_max(sum(increase(token_quote_coverage_success_total{provider="paraswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h])), 1)
sample_size: sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h]))
series: sum(rate(token_quote_coverage_success_total{provider="paraswap"}[1h])) / sum(rate(token_quote_coverage_attempts_total{provider="paraswap"}[1h])) * 100


availability: clamp_max(sum(increase(token_quote_api_ok_total{provider="paraswap"}[24h])) / sum(increase(token_quote_coverage_attempts_total{provider="paraswap"}[24h])), 1) * 100
coverage_given_available: sum(increase(token_quote_coverage_success_total{provider="paraswap"}[24h])) / sum(increase(token_quote_api_ok_total{provider="paraswap"}[24h])) * 100
10 changes: 5 additions & 5 deletions harnesses/token-quote-coverage/cmd/monitor/jupiter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,22 +40,22 @@ func (p *JupiterProvider) Quote(ctx context.Context, token Token) (ok bool) {

req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil {
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

resp, err := p.client.Do(req)
if err != nil {
fmt.Printf("[jupiter] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)

if resp.StatusCode != 200 {
fmt.Printf("[jupiter] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -64,12 +64,12 @@ func (p *JupiterProvider) Quote(ctx context.Context, token Token) (ok bool) {
}
if err := json.Unmarshal(body, &r); err != nil {
fmt.Printf("[jupiter] %s/%s parse error: %v\n", token.Chain, token.Address, err)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false)
return false
}

n, _ := strconv.ParseInt(r.OutAmount, 10, 64)
ok = n > 0
RecordProbe(p.Slug(), token.Venue, token.Chain, ok)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok)
return ok
}
12 changes: 6 additions & 6 deletions harnesses/token-quote-coverage/cmd/monitor/kyberswap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {
slug = "robinhood"
usdcAddr = kyberUSDGRobinhood
default:
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -56,23 +56,23 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {

req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil {
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
req.Header.Set("x-client-id", "openchainbench")

resp, err := p.client.Do(req)
if err != nil {
fmt.Printf("[kyberswap] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)

if resp.StatusCode != 200 {
fmt.Printf("[kyberswap] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -85,11 +85,11 @@ func (p *KyberSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {
}
if err := json.Unmarshal(body, &r); err != nil {
fmt.Printf("[kyberswap] %s/%s parse error: %v\n", token.Chain, token.Address, err)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false)
return false
}

ok = r.Data.RouteSummary.AmountOut != "" && r.Data.RouteSummary.AmountOut != "0"
RecordProbe(p.Slug(), token.Venue, token.Chain, ok)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok)
return ok
}
20 changes: 17 additions & 3 deletions harnesses/token-quote-coverage/cmd/monitor/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
var (
coverageSuccess *prometheus.CounterVec
coverageAttempts *prometheus.CounterVec
coverageAPIok *prometheus.CounterVec
)

func init() {
Expand All @@ -30,12 +31,25 @@ func init() {
[]string{"provider", "venue", "chain"},
)
prometheus.MustRegister(coverageAttempts)

coverageAPIok = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "token_quote_api_ok_total",
Help: "HTTP 200 responses received per provider/venue/chain, regardless of outAmount",
},
[]string{"provider", "venue", "chain"},
)
prometheus.MustRegister(coverageAPIok)
}

// RecordProbe records one probe attempt and optionally a success.
func RecordProbe(provider, venue, chain string, ok bool) {
// RecordProbeDetailed records one probe: apiOk=true when the provider returned HTTP 200,
// routeFound=true when outAmount > 0 was parsed from the response.
func RecordProbeDetailed(provider, venue, chain string, apiOk, routeFound bool) {
coverageAttempts.WithLabelValues(provider, venue, chain).Inc()
if ok {
if apiOk {
coverageAPIok.WithLabelValues(provider, venue, chain).Inc()
}
if routeFound {
coverageSuccess.WithLabelValues(provider, venue, chain).Inc()
}
}
Expand Down
14 changes: 7 additions & 7 deletions harnesses/token-quote-coverage/cmd/monitor/mobula.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) {
case "solana":
chainId, tokenIn, wallet = "solana", mobulaUSDCSolana, mobulaSolanaWallet
default:
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -72,23 +72,23 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) {

req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil {
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
req.Header.Set("Authorization", p.apiKey)

resp, err := p.client.Do(req)
if err != nil {
fmt.Printf("[mobula] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)

if resp.StatusCode != 200 {
fmt.Printf("[mobula] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -100,18 +100,18 @@ func (p *MobulaProvider) Quote(ctx context.Context, token Token) (ok bool) {
}
if err := json.Unmarshal(body, &r); err != nil {
fmt.Printf("[mobula] %s/%s parse error: %v\n", token.Chain, token.Address, err)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false)
return false
}

// For Solana: Mobula finds the route but fails to build the tx because the
// dummy wallet has no SOL for fees. Route found = coverage HIT.
if token.Chain == "solana" && r.Error == "Transaction build failed" {
RecordProbe(p.Slug(), token.Venue, token.Chain, true)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, true)
return true
}

ok = r.Data.AmountOutTokens != "" && r.Data.AmountOutTokens != "0"
RecordProbe(p.Slug(), token.Venue, token.Chain, ok)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok)
return ok
}
16 changes: 8 additions & 8 deletions harnesses/token-quote-coverage/cmd/monitor/odos.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) {
case p.sem <- struct{}{}:
defer func() { <-p.sem }()
case <-ctx.Done():
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -54,7 +54,7 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) {
chainId = 4663
usdcAddr = odosUSDGRobinhood
default:
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -73,28 +73,28 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) {

req, err := http.NewRequestWithContext(ctx, "POST", "https://api.odos.xyz/sor/quote/v2", bytes.NewBuffer(bb))
if err != nil {
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
req.Header.Set("Content-Type", "application/json")

resp, err := p.client.Do(req)
if err != nil {
fmt.Printf("[odos] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)

if resp.StatusCode == 429 {
fmt.Printf("[odos] %s/%s throttled\n", token.Chain, token.Address)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
if resp.StatusCode != 200 {
fmt.Printf("[odos] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -103,11 +103,11 @@ func (p *OdosProvider) Quote(ctx context.Context, token Token) (ok bool) {
}
if err := json.Unmarshal(body, &r); err != nil {
fmt.Printf("[odos] %s/%s parse error: %v\n", token.Chain, token.Address, err)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false)
return false
}

ok = len(r.OutAmounts) > 0 && r.OutAmounts[0] != "" && r.OutAmounts[0] != "0"
RecordProbe(p.Slug(), token.Venue, token.Chain, ok)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok)
return ok
}
12 changes: 6 additions & 6 deletions harnesses/token-quote-coverage/cmd/monitor/paraswap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {
network = "4663"
usdcAddr = paraswapUSDGRobinhood
default:
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -58,22 +58,22 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {

req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil {
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

resp, err := p.client.Do(req)
if err != nil {
fmt.Printf("[paraswap] %s/%s net error: %s\n", token.Chain, token.Address, classifyNetErr(err))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)

if resp.StatusCode != 200 {
fmt.Printf("[paraswap] %s/%s status=%d body=%s\n", token.Chain, token.Address, resp.StatusCode, snippet(body))
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, false, false)
return false
}

Expand All @@ -84,11 +84,11 @@ func (p *ParaSwapProvider) Quote(ctx context.Context, token Token) (ok bool) {
}
if err := json.Unmarshal(body, &r); err != nil {
fmt.Printf("[paraswap] %s/%s parse error: %v\n", token.Chain, token.Address, err)
RecordProbe(p.Slug(), token.Venue, token.Chain, false)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, false)
return false
}

ok = r.PriceRoute.DestAmount != "" && r.PriceRoute.DestAmount != "0"
RecordProbe(p.Slug(), token.Venue, token.Chain, ok)
RecordProbeDetailed(p.Slug(), token.Venue, token.Chain, true, ok)
return ok
}
Loading
Loading