Skip to content

Implement Protobuf Support for CastExpr and CastColumnExpr #20165

@kosiew

Description

@kosiew

Add protobuf serialization and deserialization support for the unified CastExpr (logical) and CastColumnExpr (physical) to enable distributed execution, Flight SQL, CLI round‑trips, and cross-process plan exchange.

Scope

Implement complete serialization/deserialization round‑trip support in datafusion/proto/ for both logical and physical cast expressions.

Tasks

Logical Cast Serialization:

  • Update datafusion/proto/proto/logical_plan.proto to support enhanced CastExpr
  • Add proto serde for enhanced CastExpr in datafusion/proto/src/logical_plan.rs
  • Implement round-trip tests for logical cast expressions

Physical Cast Serialization:

  • Add proto message for CastColumnExpr in datafusion/proto/proto/physical_plan.proto
  • Implement serialization logic (PhysicalExpr → protobuf)
  • Implement deserialization logic (protobuf → PhysicalExpr)
  • Add round-trip tests for CastColumnExpr variants
  • Test with distributed/Flight SQL scenarios

Integration & Testing:

  • Validate round-trip equality (expr → proto → expr)
  • Test with struct casting scenarios
  • Ensure backward compatibility (if applicable)
  • Add comprehensive proto round-trip test suite

Files/Modules

  • datafusion/proto/proto/logical_plan.proto (logical cast message)
  • datafusion/proto/proto/physical_plan.proto (physical cast message)
  • datafusion/proto/src/logical_plan.rs (logical cast serde)
  • datafusion/proto/src/physical_expr.rs (physical cast serde)
  • Proto-related integration tests

Key Features

  • Lossless round-trip: Serialized and deserialized expressions are equivalent
  • Struct awareness: Serialization preserves struct field casting semantics
  • Nullability propagation: Field-level nullability metadata survives round-trip
  • Backward compatibility: Graceful handling of older proto versions (if applicable)
  • Clear error messages: Helpful errors when deserialization fails

Acceptance Criteria

✅ Protobuf messages defined for both logical and physical cast expressions
✅ Serialization/deserialization code is complete and tested
✅ Round-trip tests pass for all CastExpr and CastColumnExpr variants
✅ Struct casting scenarios work end-to-end with serialization
✅ Code compiles and all proto tests pass
✅ Documented any proto breaking changes or version notes
✅ Integration tests validate distributed/Flight SQL compatibility

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions