Skip to content

docs: six of the ten ClickBench query shape labels in the 2026-08-05 section describe a different query #533

Description

@ChronicallyJD

The 2026-08-05 ClickBench section in docs/benchmarks.md labels each query with
a shape. Six of the ten labels do not describe the query they are attached to.

The timings are fine. Only the shape column is wrong, which is the awkward part:
the numbers look checkable and the prose beside them is not.

Losses table

doc says the query actually is
q24: SearchPhrase LIKE, ORDER BY EventTime LIMIT 10 SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10
q23: SELECT * of every column, ORDER BY EventTime LIMIT 10 SELECT SearchPhrase, MIN(URL), MIN(Title), COUNT(*), COUNT(DISTINCT UserID) ... WHERE Title LIKE '%Google%' ... GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10
q21: COUNT(*) WHERE URL LIKE '%google%' correct
q28: GROUP BY a normalised URL, with HAVING SELECT CounterID, AVG(length(URL)), COUNT(*) ... WHERE URL <> '' GROUP BY CounterID HAVING .... The normalised-URL description belongs to q29
q22: SearchPhrase LIKE, ORDER BY EventTime LIMIT 10 ... WHERE URL LIKE '%google%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10

The SELECT * label is on q23 when it belongs to q24, and the paragraph under
the table inherits it: "q23 selects all 105 columns, so there is no projection
to make." That is q24. The same paragraph says "q24 and q22 sort a large
intermediate to return ten rows"; q22 sorts groups after a GROUP BY, not a
large intermediate.

Wins table

doc says the query actually is
q1: COUNT(*) correct
q3: SUM, COUNT, AVG of three integer columns correct
q41, q42: GROUP BY a URL prefix, with a filter q41 groups by URLHash, EventDate; q42 by WindowClientWidth, WindowClientHeight. Neither is a URL prefix. The closest queries to that description are q37 and q39
q7: MIN and MAX of a date correct
q20: COUNT(*) with a LIKE on a short column SELECT UserID FROM hits WHERE UserID = 435090932899640449. No COUNT, no LIKE; it is a point lookup on a bigint

It is not an off-by-one

That was the first hypothesis, because upstream ClickBench numbers from q0 while
the harness prints from q1, and two of the labels fit the shift exactly: the doc's
q23 is the file's q24, and the doc's q28 is the file's q29. But q3, q7 and q21 fit
the 1-based numbering and not the shifted one, and "GROUP BY a URL prefix" fits
neither q41/q42 nor q42/q43.

The magnitudes confirm the numbering is 1-based and only the prose is wrong. The
doc's biggest loss is q24 at 11.6x, and the file's q24 is the SELECT * query,
which is the biggest loss on the 2026-08-09 run too, at 6.19x. The doc's q41 and
q42 are 25x wins, and those are 17.8x and 16.8x wins on 2026-08-09. So the labels
were written from memory rather than read off the definition.

Caveat on the fix

The 2026-08-05 run predates the definition digest being recorded, so there is no
proof it fetched the same queries.sql. Any correction is a match against the
definition as recorded on 2026-08-06,
queries.sql a7d6673357348ee9, and should say so rather than imply the run
itself was pinned.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions