This repository was archived by the owner on Jul 6, 2023. It is now read-only.
Stream results of autocommit queries #141
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When executing queries in autocommit mode, results are no longer materialized before printing. This was causing issues for big queries that would materialize as OOM crashes or just generally slow execution. For --format=PLAIN this means each records is printed directly as it arrived in the shell. For --format=VERBOSE, we cannot simply stream, because we need to know the widths of the columns before we can print even the column header. To achieve this we materialize the 1000 first rows, and compute the column widths from this sample. Any remaining rows will be streamed. If any remaining row contains a value that overflows it's column width, that value will either be wrapped to the next row (--wrap=true), or truncated (--wrap=false)
...and generally clean up lot's of mocking.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When executing queries in autocommit mode, results are no longer materialized before printing. This was causing issues for big queries that would materialize as OOM crashes or just generally slow execution.
For --format=PLAIN this means each records is printed directly as it arrives in the shell.
For --format=VERBOSE, we cannot simply stream, because we need to know the widths of the columns before we can print even the column header. To achieve this we materialize the 1000 first rows, and compute the column widths from this sample. Any remaining rows will be streamed. If any remaining row contains a value that overflows it's column width, that value will either be wrapped to the next row (--wrap=true), or truncated (--wrap=false).
This replaces the streaming aspects of #108. I felt unsure about the maxWidth aspects, so left that out for now.
[changelog]: queries executing in auto-commit mode do no longer materialize results. For --format=VERBOSE this means that column widths are computed from the first 1000 rows (--sample-rows), and any later values that overflow their colums are either wrapped or truncated (--wrap)