Skip to content

The engine hands over the buffers it filled - #23

Merged
tamnd merged 1 commit into
mainfrom
engine-columns
Aug 19, 2026
Merged

The engine hands over the buffers it filled#23
tamnd merged 1 commit into
mainfrom
engine-columns

Conversation

@tamnd

@tamnd tamnd commented Aug 19, 2026

Copy link
Copy Markdown
Owner

The pin moves to the engine commit where the sink of a plain projection
keeps its vectors instead of transposing them into rows. Nothing in this
client's surface changes for it. What changes is what happens under
columnar(): the buffers a caller gets are now the ones the pipeline
filled, so a columnar read costs a move and not two walks over a million
rows that were built to be walked twice.

One behaviour did change, and it is the better one. A statement that
matched nothing used to come back with columns typed null, because
nothing had been seen to settle the type. It now comes back with the
types the plan declared, and with the buffers to match: a string column
of no rows carries the one starting offset every string column has, an
integer column of no rows carries a buffer of no elements. So a table
built from an empty answer has the schema the same statement would have
had with rows in it, which is what a loop concatenating a page at a time
wants and what it had to special case before.

Two tests were measuring the machine rather than the code and are
rewritten to measure the code. The one that says a million row read
leaves the event loop free counted a fixed twenty timer ticks, which the
faster read now finishes inside, so it asks for a tick every ten
milliseconds of however long the read took: a blocked loop fires none of
them whatever the speed. The one that says registering a frame does not
follow its rows had a two millisecond budget, which a loaded machine
misses without anything having walked a row, so it compares the five
million row call against the ten row call instead, where walking would
show up as five hundred thousand times and not twenty.

The pin moves to the engine commit where the sink of a plain projection
keeps its vectors instead of transposing them into rows. Nothing in this
client's surface changes for it. What changes is what happens under
columnar(): the buffers a caller gets are now the ones the pipeline
filled, so a columnar read costs a move and not two walks over a million
rows that were built to be walked twice.

One behaviour did change, and it is the better one. A statement that
matched nothing used to come back with columns typed null, because
nothing had been seen to settle the type. It now comes back with the
types the plan declared, and with the buffers to match: a string column
of no rows carries the one starting offset every string column has, an
integer column of no rows carries a buffer of no elements. So a table
built from an empty answer has the schema the same statement would have
had with rows in it, which is what a loop concatenating a page at a time
wants and what it had to special case before.

Two tests were measuring the machine rather than the code and are
rewritten to measure the code. The one that says a million row read
leaves the event loop free counted a fixed twenty timer ticks, which the
faster read now finishes inside, so it asks for a tick every ten
milliseconds of however long the read took: a blocked loop fires none of
them whatever the speed. The one that says registering a frame does not
follow its rows had a two millisecond budget, which a loaded machine
misses without anything having walked a row, so it compares the five
million row call against the ten row call instead, where walking would
show up as five hundred thousand times and not twenty.
@tamnd
tamnd merged commit 04f02ad into main Aug 19, 2026
19 of 21 checks passed
@tamnd
tamnd deleted the engine-columns branch August 19, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant