Deduplicate ExpressionTransformer and CustomFunctionEnricher function evaluation#18321
Deduplicate ExpressionTransformer and CustomFunctionEnricher function evaluation#18321rsrkpatwari1234 wants to merge 12 commits intoapache:masterfrom
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18321 +/- ##
============================================
- Coverage 63.61% 63.48% -0.14%
- Complexity 1659 1701 +42
============================================
Files 3246 3255 +9
Lines 197514 199131 +1617
Branches 30578 30835 +257
============================================
+ Hits 125656 126409 +753
- Misses 61813 62641 +828
- Partials 10045 10081 +36
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
The idea is to use one function evaluator to process two sides to consolidate & simplify the code. |
Thanks for the clarification. Addressed this by one shared entry point, Policy.enricher() vs Policy.expressionTransformation(...) only switches how each (column, evaluator) is applied (always putValue(evaluate) for the JSON enricher vs. null/overwrite/implicit-MAP/continueOnError/typed collection rules for ExpressionTransformer). ExpressionTransformer and CustomFunctionEnricher both call that same method with the appropriate policy |
Fixes #18216
Motivation
ExpressionTransformerandCustomFunctionEnricherboth apply transform-styleFunctionEvaluatorsto aGenericRow. Duplicating per-column evaluation and write behavior in two places makes drift and subtle bugs more likely.What changed
IngestionFunctionEvaluationas the single place for per-row evaluator application.-- One loop:
applyFunctionEvaluations(record, evaluators, policy)iterates the Map<String, FunctionEvaluator> once; behavior is selected by IngestionFunctionEvaluation.Policy:--
Policy.expressionTransformation(...)— same semantics as before for the record-transformer chain: null / overwrite / implicit MAP-derived columns, collection-or-map backward-compat path, continueOnError with throttled warning and incomplete row marking, and applyTransformedValue null handling.--
Policy.enricher()— JSON enricher path: always putValue(column, evaluate(record)) in map order (exceptions propagate), matching prior CustomFunctionEnricher behavior.ExpressionTransformer: transform delegates toapplyFunctionEvaluationswithPolicy.expressionTransformation; constructor, topological ordering, and implicit-MAP discovery are unchanged.CustomFunctionEnricher: enrich delegates toapplyFunctionEvaluationswithPolicy.enricher(); Javadoc links to the shared helper and ExpressionTransformer for table/schema-driven transforms.Testing
Added
IngestionFunctionEvaluationTestto cover unit tests