What happened?
VortexSource exposes with_expression_convertor(...), and try_pushdown_filters uses self.expression_convertor to decide whether a
DataFusion physical expression can be pushed down.
However, when VortexSource::create_file_opener constructs VortexOpener, it appears to pass a fresh DefaultExpressionConvertor instead of
the converter stored on VortexSource:
expression_convertor: Arc::new(DefaultExpressionConvertor::default()),
I expected the opener to receive the same converter configured on the source, e.g.:
expression_convertor: Arc::clone(&self.expression_convertor),
Otherwise a custom converter can mark a custom scalar UDF/predicate as pushdown-supported at the VortexSource level, but the actual opener/scan path falls back to the default converter when converting the pushed predicate/projection.
That means custom scalar UDF pushdown may not survive into the file-opening / scan-building phase.
This seems especially relevant after #5853, which made expression conversion pluggable.
Steps to reproduce
-
Implement a custom ExpressionConvertor that supports a custom DataFusion scalar UDF or any custom PhysicalExpr.
-
Use it with:
let source = VortexSource::new(table_schema, session)
.with_expression_convertor(Arc::new(MyExpressionConvertor));
-
Push down a filter/projection that only MyExpressionConvertor supports.
-
Observe that VortexSource::try_pushdown_filters consults the custom converter, but VortexSource::create_file_opener constructs VortexOpener
with DefaultExpressionConvertor.
-
The opener then uses the default converter in its projection/predicate conversion path, so the custom expression is not handled.
Environment
Additional context
Potential fix may be as small as changing VortexSource::create_file_opener to pass through the configured converter:
expression_convertor: Arc::clone(&self.expression_convertor)
What happened?
VortexSourceexposeswith_expression_convertor(...), andtry_pushdown_filtersusesself.expression_convertorto decide whether aDataFusion physical expression can be pushed down.
However, when
VortexSource::create_file_openerconstructsVortexOpener, it appears to pass a freshDefaultExpressionConvertorinstead ofthe converter stored on
VortexSource:I expected the opener to receive the same converter configured on the source, e.g.:
Otherwise a custom converter can mark a custom scalar UDF/predicate as pushdown-supported at the VortexSource level, but the actual opener/scan path falls back to the default converter when converting the pushed predicate/projection.
That means custom scalar UDF pushdown may not survive into the file-opening / scan-building phase.
This seems especially relevant after #5853, which made expression conversion pluggable.
Steps to reproduce
Implement a custom ExpressionConvertor that supports a custom DataFusion scalar UDF or any custom PhysicalExpr.
Use it with:
let source = VortexSource::new(table_schema, session)
.with_expression_convertor(Arc::new(MyExpressionConvertor));
Push down a filter/projection that only MyExpressionConvertor supports.
Observe that VortexSource::try_pushdown_filters consults the custom converter, but VortexSource::create_file_opener constructs VortexOpener
with DefaultExpressionConvertor.
The opener then uses the default converter in its projection/predicate conversion path, so the custom expression is not handled.
Environment
Additional context
Potential fix may be as small as changing VortexSource::create_file_opener to pass through the configured converter:
expression_convertor: Arc::clone(&self.expression_convertor)