Add interface support for incremental/deferred query execution in main thread#2797
Merged
Mytherin merged 22 commits intoduckdb:masterfrom Dec 15, 2021
Merged
Add interface support for incremental/deferred query execution in main thread#2797Mytherin merged 22 commits intoduckdb:masterfrom
Mytherin merged 22 commits intoduckdb:masterfrom
Conversation
…cessary try/catch from value comparison
…tialize and ExecuteTask)
…rather than in a background thread
…nd move progress function to double
Contributor
|
So glad to see this has landed! @Mytherin are there plans in the roadmap to implement query cancellation from client libraries? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for incremental query execution from the main thread to the C++ API. For example:
Such an interface allows incrementally processing a query while periodically checking or updating external state. For example, as part of this PR we rework the progress bar to no longer require an extra thread to run in the background. Instead, we use the pending query API to process a part of the query, update/print the progress bar, and repeat until we are done. Other potential uses for this interface are adding support for query cancellation without requiring extra background threads or updating external interfaces based on progress.
This interface also creates a path to interrupt a task without completing it and giving control back to the user, which is an initial step towards supporting async I/O. For example, it would now be quite easily possible to extend the
GetSourcemethod with a return code such asWAITING_FOR_INPUT, which will cause a task to be interrupted early. The task could then be re-scheduled later when the async I/O requests have been completed.Internally the ExecuteTask will run part of a pipeline task. Specifically, we currently process 50 chunks from the pipeline source before handing control back over to the client. This is accomplished by passing in a
TaskExecutionModeto the::ExecuteTaskmethod, in which we can specify that we only want to complete a task partially (TaskExecutionMode::PROCESS_PARTIAL). When this mode is set, theExecuteTaskmethod can returnTaskExecutionResult::TASK_NOT_FINISHEDwhich will require it to be called again later.CC @ankoh