v0.8.24

Built against Mojo 1.0.0 (ed45d567).

A patch release over 0.8.23, and most of it is the other half of a change that release carried.

Changed: a streaming group by on a text key does most of its work on the cores

The map a text key was given in 0.8.23 did less work than the stacking merge it replaced and did all of it on one thread, and on one query that was a bad trade. The stacking merge concatenated the running table with the chunk's and grouped the whole thing again, which is a factorize over a column far larger than one morsel, so the kernel spread it over the machine. The map replaced that with a single loop on the thread that owns the pipeline. SELECT URL, COUNT(*) FROM hits GROUP BY URL over a million rows and two hundred and seventy five thousand distinct URLs went from two and a half cores to one, and took twice as long while doing less.

A chunk is taken in three passes now and only the last of them writes the map. Hashing a row is a pass over that row's bytes, and looking a row up is a probe and a byte comparison that changes nothing, so both of those go on the cores. Only a row carrying a key the map has never seen has to be settled on one thread, because the ordinal a key gets is the position of the row that first carried it. Both probe loops prefetch the slot eight rows ahead, which every other probe loop in hash/table.mojo already did.

Measured on the ClickBench 1M set through the planner, three drivers built from the same source and run back to back, four interleaved rounds, five runs a query, wall clock in milliseconds. The columns are the build before the map, the build with it, and this one: the unfiltered group by on URL 50.6, 102.9, 74.3; the filtered one 85.1, 51.4, 38.7; the group by on SearchPhrase 16.4, 8.2, 7.2; the count of distinct users by search phrase 23.8, 13.8, 16.7; the group by on Title 44.7, 38.7, 30.1; the second filtered URL one 11.9, 8.1, 6.3. A group by on an int key does not move.

The unfiltered group by on URL is still above what it cost before the map and it is the only one of the six that is. It groups every row of the table with no filter under it, so the pass that has to stay on one thread is the longest it gets. That is #991 and it stays open.

All forty three ClickBench statements agree with the build before the map on every row count, sum and hash, as do ten extra statements written for the shapes the suite does not have.

Changed: the test suite runs on other machines, and runs fewer files

Nothing about the library changed here, only what it costs to work on it. The suite is twelve thousand seconds and a test file spends nearly all of that compiling the library again rather than running assertions, which leaves two levers: more machines and fewer files.

pixi run test-remote rsyncs the working tree to the hosts in ~/.ssh/config and runs one shard on each. A host holds one shared copy of the tree, so a run takes a lock on the directory and waits its turn rather than taking files out from under another run's compiler, and a run records its session id in that lock so that closing the connection that started it stops it.

pixi run test-changed leaves out the files that cannot have broken. A test cannot break unless something it imports changed and Mojo imports are static, so the set is readable straight off the source. Over all 199 library files the median one selects sixteen per cent of the suite.

Neither of these is what decides whether a branch is good and both say so in their own output. CI still runs everything.