-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Performance improvement for IncrementalRelation #14482
Copy link
Copy link
Open
Labels
area:incr-processingIncremental processingIncremental processingarea:performancePerformance optimizationsPerformance optimizationsfrom-jirapriority:criticalProduction degraded; pipelines stalledProduction degraded; pipelines stalledstatus:pr-availablePull request availablePull request availabletype:improvementImprovements to existing functionalityImprovements to existing functionality
Description
Metadata
Metadata
Assignees
Labels
area:incr-processingIncremental processingIncremental processingarea:performancePerformance optimizationsPerformance optimizationsfrom-jirapriority:criticalProduction degraded; pipelines stalledProduction degraded; pipelines stalledstatus:pr-availablePull request availablePull request availabletype:improvementImprovements to existing functionalityImprovements to existing functionality
Type
Fields
Give feedbackNo fields configured for issues without a type.
When HoodieIncrSource is used to fetch the update from another Hudi table, the IncrementalRelation will be used to read the data. But it has a performance issue because the column pruning and predicate pushdown don't happen. As the result, Hudi reads too much useless data.
By enabling the column pruning and predicate pushdown, the data to read is reduced dramatically.
JIRA info