-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
Add protobuf serialization and deserialization support for the unified CastExpr (logical) and CastColumnExpr (physical) to enable distributed execution, Flight SQL, CLI round‑trips, and cross-process plan exchange.
Scope
Implement complete serialization/deserialization round‑trip support in datafusion/proto/ for both logical and physical cast expressions.
Tasks
Logical Cast Serialization:
- Update
datafusion/proto/proto/logical_plan.prototo support enhancedCastExpr - Add proto serde for enhanced
CastExprindatafusion/proto/src/logical_plan.rs - Implement round-trip tests for logical cast expressions
Physical Cast Serialization:
- Add proto message for
CastColumnExprindatafusion/proto/proto/physical_plan.proto - Implement serialization logic (
PhysicalExpr→ protobuf) - Implement deserialization logic (protobuf →
PhysicalExpr) - Add round-trip tests for
CastColumnExprvariants - Test with distributed/Flight SQL scenarios
Integration & Testing:
- Validate round-trip equality (expr → proto → expr)
- Test with struct casting scenarios
- Ensure backward compatibility (if applicable)
- Add comprehensive proto round-trip test suite
Files/Modules
datafusion/proto/proto/logical_plan.proto(logical cast message)datafusion/proto/proto/physical_plan.proto(physical cast message)datafusion/proto/src/logical_plan.rs(logical cast serde)datafusion/proto/src/physical_expr.rs(physical cast serde)- Proto-related integration tests
Key Features
- Lossless round-trip: Serialized and deserialized expressions are equivalent
- Struct awareness: Serialization preserves struct field casting semantics
- Nullability propagation: Field-level nullability metadata survives round-trip
- Backward compatibility: Graceful handling of older proto versions (if applicable)
- Clear error messages: Helpful errors when deserialization fails
Acceptance Criteria
✅ Protobuf messages defined for both logical and physical cast expressions
✅ Serialization/deserialization code is complete and tested
✅ Round-trip tests pass for all CastExpr and CastColumnExpr variants
✅ Struct casting scenarios work end-to-end with serialization
✅ Code compiles and all proto tests pass
✅ Documented any proto breaking changes or version notes
✅ Integration tests validate distributed/Flight SQL compatibility
Notes
- Scope: This PR enables distributed execution, Flight SQL, and CLI serialization for the cast expressions introduced in Extend CastExpr and Converge on Single Cast Expression #20164.
- Breaking changes: Proto updates may require versioning or migration notes if deployed to existing systems.