Skip to content

Wire up the batch backend into Pipeline::batch() - #145

Merged
szlangini merged 1 commit into
mainfrom
jan/batch-backend-wiring
Sep 4, 2026
Merged

Wire up the batch backend into Pipeline::batch()#145
szlangini merged 1 commit into
mainfrom
jan/batch-backend-wiring

Conversation

@szlangini

@szlangini szlangini commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What

Wiring up the batch engine with @lstwn 's Pipeline structure.
Fills the four bodies of the batch backend so Pipeline::batch() runs
plans end to end: build lowers, feed stages, commit computes, output
reads. Same pipeline interface as the incremental backend, but the
batch backend recomputes everything on every commit instead of
maintaining deltas.

Heads-up: Because we do not have a TableReaderAPI yet, we mimic constructing the relation once for now.
This means we assume that for feed we receive all deltas and compute a snapshot once which we then use for computing.
(see below)

Why

A batch query is one shot: build the plan, hand over the base tables,
compute, read the result. The pipeline, however, has a single input
interface, feed. It assumes to receive deltas: rows with z-weights, pushed by the
store transaction by transaction. That is exactly what the incremental
backend needs. A batch backend wants the opposite direction: pull the
full snapshot of every base table at query time. That pull API does not
exist in the pipeline yet. The store side of it does
(SortedTableSnapshot in coln-store).

This PR is arguably a hack to make an E2E test work. As we will unlikely finish the API in time.
The batch runtime accepts the pushed deltas and integrates them into a snapshot
itself. A batch query is then one feed-commit-output cycle over a
snapshot pushed through feed. Marked as a hack
in the code (module docs, materialize_sources, TODO(Jan)). Swapping it
for the pull API later touches a single function; lowering, fixpoint,
and output stay as they are.

How

  • build lowers the resolved plan into one coln-batch Datalog program
    (the mapping from Add the logical-to-physical lowering with a guided test ladder #135) and keeps it together with the schema.
  • feed stages the pushed rows per source and keeps a net z-weight per
    row. This is where the delta stream turns into a snapshot: +1 and -1
    for the same row cancel out. A source the plan does not use reports
    false, mirroring the incremental backend.
  • commit first materializes the base tables from the net weights
    (materialize_sources: net weight > 0 means present, negative is a
    clear error), then recomputes the full result with a semi-naive
    fixpoint over worst-case-optimal joins. The materialization step is
    the seam the pull API will replace.
  • output returns the new Snapshot: the full current state of a
    sink, sorted and deduplicated, with to_debug_zset() for set-level
    comparison. Batch reports states where incremental reports deltas.

Scope of this slice

  • Values: unsigned integers and booleans, mapped onto the batch engine's u64.
    Rows carrying anything else fail loudly instead of
    computing something wrong. TODO(Jan): remaining scalar types
    (strings first) after the end-to-end slice.

Testing

Run with:

cargo test -p coln-query batch
  • The reachability plan (recursive) and a plain u64 join run through
    both pipelines, batch and incremental, and must produce exactly the
    same rows. The first cross-engine differential.
  • Error paths covered: strings in feed, output before commit, feeding
    a source the plan does not use.
  • Not covered yet: retractions between commits. The demo never issues
    them, so the net-weight integration is only exercised with
    insertions.

build lowers the resolved plan into one coln-batch Datalog program,
feed stages rows per source (net z-weights; u64 and bool value slice),
commit recomputes the full result with the semi-naive fixpoint over
worst-case-optimal joins, and output returns a Snapshot: the full
current state of a sink, sorted and deduplicated, with a
to_debug_zset view for set-level comparison against the incremental
backend's consolidated deltas.

Anchor tests run the arithmetic-free reachability plan and a plain
u64 join through both pipelines and require identical rows. Rows
carrying strings fail loudly for now (TODO: remaining scalar types
after the end-to-end slice).

Structurally, the lowering module leaves its cfg(test) gate and
coln-batch becomes a regular dependency of coln-query. The stale,
unused coln-integrator dependency of coln-batch is removed; that dead
edge is what kept this direction cycle-free.
@szlangini
szlangini added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 985da6d Sep 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant