Update columnar 0.12, timely 0.28, differential-dataflow 0.21 by antiguru · Pull Request #35674 · MaterializeInc/materialize

antiguru · 2026-03-26T05:50:08Z

Summary

Bump columnar 0.11.1→0.12.0, timely 0.27→0.28, differential-dataflow 0.20→0.21
Adapt all downstream code to breaking API changes in these crates

columnar 0.12

EncodeDecode trait and Indexed struct replaced by columnar::bytes::indexed module
HeapSize trait removed
FromBytes now requires const SLICE_COUNT: usize
Strings::Ref changed from &str to &[u8]

timely 0.28

Operate trait redesigned: get_internal_summary()/set_external_summary() replaced by initialize(self: Box<Self>) which returns (Connectivity, SharedProgress, Box<dyn Schedule>)
notify_me() returns &[FrontierInterest] instead of bool
set_notify(false) replaced by per-input set_notify_for(i, FrontierInterest::Never)
MutableAntichain::new_bottom renamed to from_elem

differential-dataflow 0.21

ColMerger renamed to ColInternalMerger
MergerChunk trait renamed to InternalMerge

Test plan

cargo check passes
cargo clippy clean
cargo fmt clean
CI

🤖 Generated with Claude Code

github-actions · 2026-03-26T05:50:18Z

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

def-

The nightly performance regressions are concerning, but you've probably seen those: https://buildkite.com/materialize/nightly/builds/15871

def-

No concerns from testing side!

frankmcsherry · 2026-03-27T13:10:02Z

src/compute/src/extensions/reduce.rs

 use differential_dataflow::lattice::Lattice;
 use differential_dataflow::operators::arrange::{Arranged, TraceAgent};
-use differential_dataflow::trace::implementations::merge_batcher::container::MergerChunk;
+use differential_dataflow::trace::implementations::merge_batcher::container::InternalMerge;


I hope to delete this in the next DD. Should check, but I think it is a near-vestigial use for reduce just to access .clear().

I have a follow-up to vendor everything around columnation/TimelyStack into Mz so we can remove all of it from DD.

frankmcsherry · 2026-03-27T13:11:06Z

src/compute/src/extensions/temporal_bucket.rs

    logger: Option<differential_dataflow::logging::Logger>,
    operator_id: usize,
-    inner: MergeBatcher<Vec<(D, T, R)>, ColumnationChunker<(D, T, R)>, ColMerger<D, T, R>>,
+    inner: MergeBatcher<Vec<(D, T, R)>, ColumnationChunker<(D, T, R)>, ColInternalMerger<D, T, R>>,


I'm not sure what this is through the aliases, but we should double check async. Some of the mergers are just bad, and some of them are better than others.

frankmcsherry · 2026-03-27T13:12:10Z

src/compute/src/logging/timely.rs

+                                // SAFETY: `datum.typ` originates from a Rust `String` field
+                                // (`ChannelsEvent::typ`) serialized through columnar, which
+                                // now returns `&[u8]` instead of `&str` to skip validation.
+                                Datum::String(unsafe { std::str::from_utf8_unchecked(datum.typ) }),


This is a behavioral change where we used to panic, but will now be UB. Worth double-checking which we want (personally prefer the panic potential).

frankmcsherry · 2026-03-27T13:13:08Z

src/expr/src/lib.rs

 //! Core expression language.

 #![warn(missing_debug_implementations)]
+#![recursion_limit = "256"]


If you can leave a note about why, that would be helpful for the future.

frankmcsherry

Looked at everything other than Cargo.lock.

The PR doesn't mention, but there are a substantial number of Columnar derivations added. I think this is good, personally, but it locks in some obligations (e.g. if folks rely on them, they can no longer be removed).
There is new unsafe in the codebase that I think we should avoid. If we find utf8 validation in logging dataflows is a hotspot we can add it in, but imo let's default to safe.

antiguru · 2026-03-27T13:27:56Z

Thanks for the review. I backed out the Columnar derives, and replaced the unsafe with an expect.

Updates * columnar to 0.12.0 * timely to 0.28.0 * differential-dataflow to 0.21.0 Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

…hanges Update our code to match the breaking API changes in the bumped dependencies: **columnar 0.12:** - `EncodeDecode` trait and `Indexed` struct removed; replaced by `columnar::bytes::indexed` module with free functions - `HeapSize` trait removed; delete all implementations - `FromBytes` now requires `const SLICE_COUNT: usize` - `Strings::Ref` is now `&[u8]` instead of `&str`; use `from_utf8_unchecked` where the data is known-valid UTF-8 **timely 0.28:** - `Operate` trait: `get_internal_summary()` + `set_external_summary()` replaced by `initialize(self: Box<Self>)` which consumes the operator and returns `(Connectivity, SharedProgress, Box<dyn Schedule>)` - `notify_me()` returns `&[FrontierInterest]` instead of `bool` - `set_notify(false)` replaced by per-input `set_notify_for(i, FrontierInterest::Never)` - `MutableAntichain::new_bottom` renamed to `from_elem` **differential-dataflow 0.21:** - `ColMerger` renamed to `ColInternalMerger` (in `container` submodule) - `MergerChunk` trait renamed to `InternalMerge` - `ColInternalMerger` now requires `D: Clone` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

antiguru requested review from a team, aljoscha and ggevay as code owners March 26, 2026 05:50

antiguru requested review from SangJunBak and removed request for a team March 26, 2026 05:50

antiguru force-pushed the update_td_dd branch from 1334233 to 36d8178 Compare March 26, 2026 05:52

antiguru requested a review from frankmcsherry March 26, 2026 05:53

def- reviewed Mar 26, 2026

View reviewed changes

antiguru force-pushed the update_td_dd branch 5 times, most recently from e12a447 to 626ab54 Compare March 27, 2026 12:28

antiguru requested a review from def- March 27, 2026 12:38

def- approved these changes Mar 27, 2026

View reviewed changes

antiguru force-pushed the update_td_dd branch from 626ab54 to da6449b Compare March 27, 2026 12:58

frankmcsherry reviewed Mar 27, 2026

View reviewed changes

frankmcsherry approved these changes Mar 27, 2026

View reviewed changes

antiguru and others added 2 commits March 27, 2026 23:06

Update columnar, timely, differential-dataflow

3c8fbd6

Updates * columnar to 0.12.0 * timely to 0.28.0 * differential-dataflow to 0.21.0 Signed-off-by: Moritz Hoffmann <antiguru@gmail.com>

antiguru force-pushed the update_td_dd branch from 620ddc8 to 0a8e520 Compare March 27, 2026 21:11

antiguru merged commit c642b63 into MaterializeInc:main Mar 27, 2026
118 checks passed

antiguru deleted the update_td_dd branch March 27, 2026 21:29

antiguru mentioned this pull request Mar 27, 2026

timely-util: vendor ColumnationStack from differential-dataflow #35760

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update columnar 0.12, timely 0.28, differential-dataflow 0.21#35674

Update columnar 0.12, timely 0.28, differential-dataflow 0.21#35674
antiguru merged 2 commits intoMaterializeInc:mainfrom
antiguru:update_td_dd

antiguru commented Mar 26, 2026

Uh oh!

github-actions bot commented Mar 26, 2026

Uh oh!

def- left a comment

Uh oh!

def- left a comment

Uh oh!

frankmcsherry Mar 27, 2026

Uh oh!

antiguru Mar 27, 2026

Uh oh!

frankmcsherry Mar 27, 2026

Uh oh!

frankmcsherry Mar 27, 2026

Uh oh!

frankmcsherry Mar 27, 2026

Uh oh!

frankmcsherry left a comment

Uh oh!

antiguru commented Mar 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

antiguru commented Mar 26, 2026

Summary

columnar 0.12

timely 0.28

differential-dataflow 0.21

Test plan

Uh oh!

github-actions bot commented Mar 26, 2026

PR title guidelines

Pre-merge checklist

Uh oh!

def- left a comment

Choose a reason for hiding this comment

Uh oh!

def- left a comment

Choose a reason for hiding this comment

Uh oh!

frankmcsherry Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

frankmcsherry Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

frankmcsherry Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

frankmcsherry Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

frankmcsherry left a comment

Choose a reason for hiding this comment

Uh oh!

antiguru commented Mar 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants