The engine hands over the buffers it filled - #23
Merged
Merged
Conversation
The pin moves to the engine commit where the sink of a plain projection keeps its vectors instead of transposing them into rows. Nothing in this client's surface changes for it. What changes is what happens under columnar(): the buffers a caller gets are now the ones the pipeline filled, so a columnar read costs a move and not two walks over a million rows that were built to be walked twice. One behaviour did change, and it is the better one. A statement that matched nothing used to come back with columns typed null, because nothing had been seen to settle the type. It now comes back with the types the plan declared, and with the buffers to match: a string column of no rows carries the one starting offset every string column has, an integer column of no rows carries a buffer of no elements. So a table built from an empty answer has the schema the same statement would have had with rows in it, which is what a loop concatenating a page at a time wants and what it had to special case before. Two tests were measuring the machine rather than the code and are rewritten to measure the code. The one that says a million row read leaves the event loop free counted a fixed twenty timer ticks, which the faster read now finishes inside, so it asks for a tick every ten milliseconds of however long the read took: a blocked loop fires none of them whatever the speed. The one that says registering a frame does not follow its rows had a two millisecond budget, which a loaded machine misses without anything having walked a row, so it compares the five million row call against the ten row call instead, where walking would show up as five hundred thousand times and not twenty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pin moves to the engine commit where the sink of a plain projection
keeps its vectors instead of transposing them into rows. Nothing in this
client's surface changes for it. What changes is what happens under
columnar(): the buffers a caller gets are now the ones the pipeline
filled, so a columnar read costs a move and not two walks over a million
rows that were built to be walked twice.
One behaviour did change, and it is the better one. A statement that
matched nothing used to come back with columns typed null, because
nothing had been seen to settle the type. It now comes back with the
types the plan declared, and with the buffers to match: a string column
of no rows carries the one starting offset every string column has, an
integer column of no rows carries a buffer of no elements. So a table
built from an empty answer has the schema the same statement would have
had with rows in it, which is what a loop concatenating a page at a time
wants and what it had to special case before.
Two tests were measuring the machine rather than the code and are
rewritten to measure the code. The one that says a million row read
leaves the event loop free counted a fixed twenty timer ticks, which the
faster read now finishes inside, so it asks for a tick every ten
milliseconds of however long the read took: a blocked loop fires none of
them whatever the speed. The one that says registering a frame does not
follow its rows had a two millisecond budget, which a loaded machine
misses without anything having walked a row, so it compares the five
million row call against the ten row call instead, where walking would
show up as five hundred thousand times and not twenty.