colexec: add a way to propagate metadata in a streaming fashion #55758
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-sql-queries
SQL Queries Team
Projects
Comments
yuzefovich
added
the
C-enhancement
yuzefovich
added this to Triage
in BACKLOG, NO NEW ISSUES: SQL Execution
via automation
yuzefovich
changed the title
asubiotto
moved this from Triage
to 21.1 Release Issues
in BACKLOG, NO NEW ISSUES: SQL Execution
5 tasks
asubiotto
moved this from 21.1 Release Issues
to 21.1 Stability Issues
in BACKLOG, NO NEW ISSUES: SQL Execution
asubiotto
moved this from 21.1 Stability Issues
to [VECTORIZED BACKLOG] Enhancements/Features/Investigations
in BACKLOG, NO NEW ISSUES: SQL Execution
yuzefovich
removed this from [VECTORIZED BACKLOG] Enhancements/Features/Investigations
in BACKLOG, NO NEW ISSUES: SQL Execution
craig bot
pushed a commit
that referenced
this issue
May 15, 2021
64697: colflow: introduce flow coordinator and simplify materializer r=yuzefovich a=yuzefovich **execinfra: refactor ProcessorBase slightly** Previously, the output of a processor was embedded in `ProcOutputHelper`. This commit moves it out into the `ProcessorBase` because the follow-up commit will take advantage of such placement. Release note: None **execinfra: rename a field in ProcOutputHelper** This commit renames `ProcessorBase.Out` to `ProcessorBase.OutputHelper`. It also removes the large part of the comment on `ProcessorBase` since it has become quite stale, and it is better to take a look at the existing users of the struct as a guide (the actual code should be up-to-date!). Release note: None **execinfra: separate out ProcOutputHelper from ProcessorBase** In some cases, `ProcOutputHelper` that lives in the `ProcessorBase` is not used by the caller, yet it is always allocated. This commit extracts `ProcessorBaseNoHelper` that doesn't contain the helper (as well as some other fields) and uses that in the materializer. This should reduce the size of the materializers slightly. This commit was prompted by the fact that the follow-up commit will introduce another processor (the vectorized flow coordinator) that also doesn't utilize the `ProcOutputHelper`. Release note: None **colflow: introduce flow coordinator and simplify materializer** This commit extracts the logic of shutting down the vectorized flow out of the materializer which simplifies the latter. This allows us to optimize the case when the root of the whole plan is a wrapped row-execution processor. Previously, in such a scenario we would plan a columnarizer followed by a materializer because the latter was needed in order to shut the flow down. This commit removes this redundant pair of operators. Informs: #50857. Informs: #55758. Release note: None Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
|
We have marked this issue as stale because it has been inactive for |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While working on #55713, I realized that the current way of propagating the metadata in the vectorized engine is incompatible with some things we would like to do. As a reminder, we currently buffer up all metadata in
MetadataSources and return it when draining the whole flow, but there are cases when we would like to propagate the metadata once it is created:rowexec.TableReaders which is then used in combination with the estimated number of rows to be scanned (which is obtained during planning) in order provide the progress estimate for a query (shown onSHOW QUERIESinphasecolumn)The current thinking is that we probably want to implement a pull-based model that is able to return the buffered meta since the previous call (something like
DrainBufferedMetawhich - unlikeDrainMeta- doesn't have an assumption of being called when the flow is shutdown) and then have the root materializer periodically poll its metadata sources.Jira issue: CRDB-3634
The text was updated successfully, but these errors were encountered: