Wire up the batch backend into Pipeline::batch() - #145
Merged
Conversation
build lowers the resolved plan into one coln-batch Datalog program, feed stages rows per source (net z-weights; u64 and bool value slice), commit recomputes the full result with the semi-naive fixpoint over worst-case-optimal joins, and output returns a Snapshot: the full current state of a sink, sorted and deduplicated, with a to_debug_zset view for set-level comparison against the incremental backend's consolidated deltas. Anchor tests run the arithmetic-free reachability plan and a plain u64 join through both pipelines and require identical rows. Rows carrying strings fail loudly for now (TODO: remaining scalar types after the end-to-end slice). Structurally, the lowering module leaves its cfg(test) gate and coln-batch becomes a regular dependency of coln-query. The stale, unused coln-integrator dependency of coln-batch is removed; that dead edge is what kept this direction cycle-free.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wiring up the batch engine with @lstwn 's
Pipelinestructure.Fills the four bodies of the batch backend so
Pipeline::batch()runsplans end to end: build lowers, feed stages, commit computes, output
reads. Same pipeline interface as the incremental backend, but the
batch backend recomputes everything on every commit instead of
maintaining deltas.
Heads-up: Because we do not have a
TableReaderAPIyet, we mimic constructing the relation once for now.This means we assume that for feed we receive all deltas and compute a snapshot once which we then use for computing.
(see below)
Why
A batch query is one shot: build the plan, hand over the base tables,
compute, read the result. The pipeline, however, has a single input
interface,
feed. It assumes to receive deltas: rows with z-weights, pushed by thestore transaction by transaction. That is exactly what the incremental
backend needs. A batch backend wants the opposite direction: pull the
full snapshot of every base table at query time. That pull API does not
exist in the pipeline yet. The store side of it does
(
SortedTableSnapshotin coln-store).This PR is arguably a hack to make an E2E test work. As we will unlikely finish the API in time.
The batch runtime accepts the pushed deltas and integrates them into a snapshot
itself. A batch query is then one feed-commit-output cycle over a
snapshot pushed through
feed. Marked as a hackin the code (module docs,
materialize_sources, TODO(Jan)). Swapping itfor the pull API later touches a single function; lowering, fixpoint,
and output stay as they are.
How
buildlowers the resolved plan into one coln-batch Datalog program(the mapping from Add the logical-to-physical lowering with a guided test ladder #135) and keeps it together with the schema.
feedstages the pushed rows per source and keeps a net z-weight perrow. This is where the delta stream turns into a snapshot: +1 and -1
for the same row cancel out. A source the plan does not use reports
false, mirroring the incremental backend.commitfirst materializes the base tables from the net weights(
materialize_sources: net weight > 0 means present, negative is aclear error), then recomputes the full result with a semi-naive
fixpoint over worst-case-optimal joins. The materialization step is
the seam the pull API will replace.
outputreturns the newSnapshot: the full current state of asink, sorted and deduplicated, with
to_debug_zset()for set-levelcomparison. Batch reports states where incremental reports deltas.
Scope of this slice
Rows carrying anything else fail loudly instead of
computing something wrong. TODO(Jan): remaining scalar types
(strings first) after the end-to-end slice.
Testing
Run with:
cargo test -p coln-query batchboth pipelines, batch and incremental, and must produce exactly the
same rows. The first cross-engine differential.
a source the plan does not use.
them, so the net-weight integration is only exercised with
insertions.