feat: serialize ASOF join plans by Xuanwo · Pull Request #23832 · apache/datafusion

Xuanwo · 2026-07-23T06:12:25Z

Which issue does this PR close?

Part of ASOF join support / Specialize Range Joins #318.
Umbrella PR: feat: support ASOF joins #23738.
Depends on feat: add ASOF join logical semantics #23829 (and transitively feat: add ASOF join physical operator #23828).

Rationale for this change

This is the serialization layer of the ASOF JOIN stack. It gives logical and
physical ASOF plans explicit protobuf representations without coupling wire
format review to SQL or DataFrame APIs.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-proto

What changes are included in this PR?

Add protobuf messages and enum values for logical and physical ASOF joins.
Encode and decode equality keys, ordered match expressions, match direction,
join constraint, and right output indices.
Regenerate prost and pbjson sources with the repository generator.
Add logical round trips for all four match directions.
Add physical AsOfJoinExec round-trip coverage.

Are these changes tested?

Yes:

./datafusion/proto-models/regen.sh
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-proto roundtrip_asof_join --all-features
Extended workspace tests from the contributor guide

Are there any user-facing changes?

Logical and physical ASOF join plans can be serialized through
datafusion-proto. This extends the protobuf wire format with new message
variants.

This PR should be reviewed and merged after #23829. After its dependencies
squash-merge, this branch will be refreshed from main without rewriting its
published history.

github-actions · 2026-07-23T06:35:57Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 105.808s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 104.088s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.044s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.588s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 212.143s] datafusion
    Building datafusion-expr v54.1.0 (current)
       Built [  26.985s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.077s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  26.645s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.232s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:245
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:245

--- warning partial_ord_enum_variants_reordered: enum variants reordered in #[derive(PartialOrd)] enum ---

Description:
A public enum that derives PartialOrd had its variants reordered. #[derive(PartialOrd)] uses the enum variant order to set the enum's ordering behavior, so this change may break downstream code that relies on the previous order.
        ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/partial_ord_enum_variants_reordered.ron

Failed in:
  LogicalPlan::Repartition moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:249
  LogicalPlan::Union moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:253
  LogicalPlan::TableScan moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:256
  LogicalPlan::EmptyRelation moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:260
  LogicalPlan::Subquery moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:263
  LogicalPlan::SubqueryAlias moved from position 12 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:265
  LogicalPlan::Limit moved from position 13 to 14, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:267
  LogicalPlan::Statement moved from position 14 to 15, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:269
  LogicalPlan::Values moved from position 15 to 16, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:274
  LogicalPlan::Explain moved from position 16 to 17, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:277
  LogicalPlan::Analyze moved from position 17 to 18, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:281
  LogicalPlan::Extension moved from position 18 to 19, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:284
  LogicalPlan::Distinct moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:287
  LogicalPlan::Dml moved from position 20 to 21, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:289
  LogicalPlan::Ddl moved from position 21 to 22, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:291
  LogicalPlan::Copy moved from position 22 to 23, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:293
  LogicalPlan::DescribeTable moved from position 23 to 24, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:296
  LogicalPlan::Unnest moved from position 24 to 25, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:299
  LogicalPlan::RecursiveQuery moved from position 25 to 26, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  LogicalPlan::Repartition moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:249
  LogicalPlan::Union moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:253
  LogicalPlan::TableScan moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:256
  LogicalPlan::EmptyRelation moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:260
  LogicalPlan::Subquery moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:263
  LogicalPlan::SubqueryAlias moved from position 12 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:265
  LogicalPlan::Limit moved from position 13 to 14, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:267
  LogicalPlan::Statement moved from position 14 to 15, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:269
  LogicalPlan::Values moved from position 15 to 16, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:274
  LogicalPlan::Explain moved from position 16 to 17, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:277
  LogicalPlan::Analyze moved from position 17 to 18, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:281
  LogicalPlan::Extension moved from position 18 to 19, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:284
  LogicalPlan::Distinct moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:287
  LogicalPlan::Dml moved from position 20 to 21, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:289
  LogicalPlan::Ddl moved from position 21 to 22, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:291
  LogicalPlan::Copy moved from position 22 to 23, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:293
  LogicalPlan::DescribeTable moved from position 23 to 24, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:296
  LogicalPlan::Unnest moved from position 24 to 25, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:299
  LogicalPlan::RecursiveQuery moved from position 25 to 26, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [  55.883s] datafusion-expr
    Building datafusion-optimizer v54.1.0 (current)
       Built [  27.098s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  27.022s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.148s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  55.105s] datafusion-optimizer
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  37.265s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.147s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  36.829s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.154s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.663s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  76.181s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  60.268s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  59.738s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.227s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 121.788s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  24.619s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.131s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.852s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.133s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.618s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  52.432s] datafusion-proto-models
    Building datafusion-sql v54.1.0 (current)
       Built [  41.906s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  41.675s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.244s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  84.978s] datafusion-sql
    Building datafusion-substrait v54.1.0 (current)
       Built [ 349.441s] (current)
     Parsing datafusion-substrait v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-substrait v54.1.0 (baseline)
       Built [ 348.096s] (baseline)
     Parsing datafusion-substrait v54.1.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.203s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 699.739s] datafusion-substrait

codecov-commenter · 2026-07-23T07:04:11Z

Codecov Report

❌ Patch coverage is 56.65835% with 869 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.62%. Comparing base (47bd094) to head (d36fd27).

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	341 Missing ⚠️
datafusion/physical-plan/src/joins/asof_join.rs	85.70%	89 Missing and 55 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs	34.13%	99 Missing and 11 partials ⚠️
datafusion/core/src/physical_planner.rs	0.00%	55 Missing ⚠️
datafusion/expr/src/logical_plan/tree_node.rs	0.00%	43 Missing ⚠️
datafusion/proto/src/logical_plan/mod.rs	62.06%	20 Missing and 13 partials ⚠️
datafusion/proto/src/physical_plan/mod.rs	71.17%	19 Missing and 13 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	0.00%	28 Missing ⚠️
datafusion/optimizer/src/push_down_filter.rs	0.00%	24 Missing ⚠️
datafusion/proto-models/src/generated/prost.rs	0.00%	17 Missing ⚠️
... and 6 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23832      +/-   ##
==========================================
- Coverage   80.75%   80.62%   -0.14%     
==========================================
  Files        1089     1090       +1     
  Lines      368948   370950    +2002     
  Branches   368948   370950    +2002     
==========================================
+ Hits       297952   299075    +1123     
- Misses      53237    54009     +772     
- Partials    17759    17866     +107

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Xuanwo added 3 commits July 23, 2026 13:05

feat: add ASOF join physical operator

e546de5

feat: add ASOF join logical semantics

bd58e1d

feat: serialize ASOF join plans

ed2ce92

github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026

Xuanwo mentioned this pull request Jul 23, 2026

feat: support ASOF joins #23738

Open

Xuanwo added 3 commits July 23, 2026 14:30

Merge remote-tracking branch 'origin/main' into xuanwo/asof-physical

68ac8ec

Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical

c8c4813

Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto

ee41c49

github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026

fix: keep generated protobuf code in sync

d36fd27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: serialize ASOF join plans#23832

feat: serialize ASOF join plans#23832
Xuanwo wants to merge 7 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-proto

Xuanwo commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Xuanwo commented Jul 23, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 23, 2026 •

edited

Loading