fix(chain): count distinct weight setters with GROUP BY, not COUNT(DISTINCT) - #9261
Merged
Conversation
…STINCT) /api/v1/chain/weights/setters served a card of zeros in production while all three surfaces were correctly wired to the shared loader (#9249). The reader was declining, not missing. loadChainEventIdentityRollup computed its totals with an ungrouped count(DISTINCT uid), which R2 SQL rejects outright at this scale: 40015: scan budget exceeded: scanning too much data for count(DISTINCT) without GROUP BY A rejected query makes the reader return null, the handler falls through to its empty builder, and the route publishes zeros -- which read as measured zeros rather than as a decline. Same expression #9252 removed from the sibling netuid rollup; the identity rollup kept it. Even where the ungrouped form did run it answered the wrong question. A uid is unique only WITHIN a subnet, so count(DISTINCT uid) collapses uid 5 on twenty subnets into one and lands near the 256 uid ceiling regardless of the truth: 254 reported against 1,284 real (netuid, uid) participants over the same 7d window. Split the totals in two. count(*) stays ungrouped -- it is a plain row count with no distinct in it, and it is the honest share denominator, since the row page is capped by `limit` and summing the page would make each share depend on the page size. The distinct half becomes a GROUP BY netuid, <identity> subquery, which the engine accepts and which counts the pair. The reader still declines when any of the three halves misses, so a caller's own empty shape stands rather than a partial answer being published as data. Verified live against the lakehouse over 7d: weight_sets 261,163, distinct_setters 1,284. The test harness no longer discriminates queries on "GROUP BY" -- the distinct subquery contains one too, so that would have answered the row fixture to two different questions. Each query is selected on the clause only it can have. Closes #9258
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | 8ffeb22 | Aug 03 2026, 12:06 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-registry-sync-api | 8ffeb22 | Aug 03 2026, 12:06 PM |
JSONbored
added a commit
that referenced
this pull request
Aug 3, 2026
…rd (#9282) /api/v1/accounts/{ss58} and get_account_snapshot published event_count 0 and subnet_count 0 for an address whose own /events feed returned rows. Same address, same stream, opposite answers -- so not an empty account. loadAccountSummaryColdTier computed its aggregates with an ungrouped count(DISTINCT netuid), which R2 SQL rejects outright: 40015: scan budget exceeded: scanning too much data for count(DISTINCT) without GROUP BY A rejected read declines the whole reader, the handler falls through to its empty builder, and the card publishes zeros that read as measured fact. Third instance of this expression: #9252 removed it from the netuid rollup and #9261 from the identity rollup. A BOUNDED CTE DOES NOT BOUND THE BUDGET, which is why the earlier two passes did not catch this one. The distinct ran over a `scan` CTE already capped at ACCOUNT_EVENT_SUMMARY_SCAN_CAP (5,000) rows, so it reads as an aggregate over 5,000 rows and looks obviously safe. The engine costs the distinct against the underlying scan, not the materialized cap. Wrapping a distinct in a bounded CTE is not a workaround; only GROUP BY is. Split the aggregate read in two, as in #9252/#9261: count(*) and the min/max bounds stay together (no distinct, no budget problem), and the distinct-subnet count becomes count(*) FROM (SELECT netuid FROM scan GROUP BY netuid). The reader declines when the new read misses, so a card can never pair a real event_count with subnet_count 0. Verified live for the address above: 5,000 events across blocks 8,759,294 to 8,763,565, 129 distinct netuids -- consistent with ~129 active subnets. The test asserts the SHAPE of the distinct read, not merely the absence of the word DISTINCT, because the bounded-CTE form is the trap. Its fixture also keeps the subnet count out of the aggregate row so a regression back to one combined read is visible here rather than only in production. Closes #9280
JSONbored
added a commit
that referenced
this pull request
Aug 3, 2026
…9304) /api/v1/subnets/{netuid}/event-summary reported total_events 0 for EVERY netuid -- 1, 8, 19 and 64 all answered zero -- while /subnets/{netuid}/events served real rows off the same account_events stream. Two views of one stream, opposite answers, the same shape as #9260. All three surfaces ran tryPostgresTier(METAGRAPH_ACCOUNT_EVENTS_SOURCE) ?? buildSubnetEventSummary([], [], ...) and nothing Cloudflare-native replaced the deleted Postgres tier, so add one shared reader and wire all three at once. THE OBVIOUS PORT IS REJECTED. One grouped rollup carrying count(*) plus count(DISTINCT hotkey) and count(DISTINCT coldkey) fails at this route's own default window: 40015: scan budget exceeded: scanning too much data for count(DISTINCT), count(DISTINCT) with GROUP BY Note "with GROUP BY". Unlike #9252, #9261 and #9280 -- where the fix was ADDING a GROUP BY -- adding one is not the fix here: two distincts in a single grouped scan exceed the budget on their own, and this route also offers 90d, three times the span that already fails. So each distinct is distributed into its own nested aggregation, grouping to the (kind, key) pairs and then counting the pairs per kind. `hotkey IS NOT NULL` is load-bearing rather than tidy. COUNT(DISTINCT col) ignores NULLs while GROUP BY col yields a NULL group, so without it every kind whose rows carry no hotkey -- WeightsSet, and every Balances kind for coldkey -- would report one participant that does not exist. An empty window is a MEASURED ZERO, not a decline: the query layer returns null on failure and [] on a successful empty scan, so a quiet subnet can say so rather than being indistinguishable from the broken tier this fixes. A failed read still declines, so the card can never pair real counts with zeroed participants. Verified live for netuid 64 over 30d: WeightsSet 9,832 events, StakeAdded 8,517 events across 66 distinct hotkeys and 2,109 distinct coldkeys. Writing the limit test found a real defect in the first draft: `??` only catches null/undefined, so a literal 0 sailed through the default resolution and produced LIMIT 0 -- a silently empty recent-events page. parseLimitParam rejects 0 at the REST edge, but MCP and GraphQL call this reader directly. The positivity check is now part of resolving an unusable limit rather than a separate guard. Three guards that could never fire were removed rather than left as untestable defensive code: a cap floor the resolution already guarantees, an arithmetic check on a cutoff derived from a validated window, and a column-name guard over two string literals. Closes #9303
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/api/v1/chain/weights/settersserved a card of zeros in production while all three surfaces werecorrectly wired to the shared loader (#9249). The reader was declining, not missing.
Root cause
loadChainEventIdentityRollupcomputed its totals with an ungroupedcount(DISTINCT uid), whichR2 SQL rejects outright at this scale — confirmed by running the reader's exact query:
A rejected query makes the reader return
null, the handler falls through to its empty builder,and the route publishes zeros — which read as measured zeros rather than as a decline. This is
the same expression #9252 removed from the sibling netuid rollup; the identity rollup kept it.
The quieter half
Even where the ungrouped form did run it answered the wrong question. A
uidis unique onlywithin a subnet, so
count(DISTINCT uid)collapses uid 5 on twenty subnets into one and landsnear the 256 uid ceiling regardless of the truth:
count(DISTINCT uid), ungrouped(netuid, uid)participantsFix
The totals split in two:
count(*)stays ungrouped — a plain row count with no distinct in it, so neither problemapplies, and it is the honest share denominator: the row page is capped by
limit, so summingthe page would make each share depend on the page size.
GROUP BY netuid, <identity>subquery — the form the engineaccepts, and the one that counts the participant rather than the uid number.
The reader still declines when any of the three halves misses, so a caller's own empty shape
stands rather than a partial answer being published as data.
Verification
Both queries run live against the lakehouse over the 7d window:
count(*)weight_sets = 261,163GROUP BY netuid, uidsubquerydistinct_setters = 1,2841,284 matches the figure #9252 verified for the same stream, from an independent query shape.
Four mutations of the new code, each caught:
!distinctRowsguard!distinctguardcount(DISTINCT)Patch coverage 5/5 = 100%, measured by intersecting the diff's line ranges with v8's uncovered
set. 69 tests pass across the four affected suites;
tsc --noEmitclean; prettier and eslint clean.One test-harness note
The fake engine no longer discriminates queries on
"GROUP BY"— the new distinct subquerycontains one too, so that discriminator would have silently answered the row fixture to two
different questions. Each query is now selected on the clause only it can have (
ORDER BYfor theranked page,
FROM (for the subquery).Closes #9258