Consider the following places in the physical plan serde code.
serialization:
|
projection: exec.projection().as_ref().map_or_else(Vec::new, |v| { |
|
v.iter().map(|x| *x as u32).collect::<Vec<u32>>() |
deserealization:
|
let projection = if !filter.projection.is_empty() { |
|
Some( |
|
filter |
|
.projection |
|
.iter() |
|
.map(|i| *i as usize) |
|
.collect::<Vec<_>>(), |
|
) |
|
} else { |
|
None |
|
}; |
If we consider FilterExec with an empty projection (vec![]), then it is serialized as an empty vector. Then the deserialization code makes a FilterExec with None as projection, which semantics is a full projection. The problem is that None and Some(vec![]) have the same proto representation.
The simplest solution:
- Make
(0..fields.len()).collect() vec when None projection is serialized.
- Make
None if this vec is met in the deserialization code.
Consider the following places in the physical plan serde code.
serialization:
datafusion/datafusion/proto/src/physical_plan/mod.rs
Lines 2353 to 2354 in 1897c28
deserealization:
datafusion/datafusion/proto/src/physical_plan/mod.rs
Lines 684 to 694 in 1897c28
If we consider
FilterExecwith an empty projection (vec![]), then it is serialized as an empty vector. Then the deserialization code makes aFilterExecwithNoneas projection, which semantics is a full projection. The problem is thatNoneandSome(vec![])have the same proto representation.The simplest solution:
(0..fields.len()).collect()vec whenNoneprojection is serialized.Noneif this vec is met in the deserialization code.