Skip to content

[HUDI-8335] Optimize the mor table filter push down when the file group contains only base file#12080

Merged
danny0405 merged 2 commits intoapache:masterfrom
Zouxxyy:dev/hudi-8335
Oct 14, 2024
Merged

[HUDI-8335] Optimize the mor table filter push down when the file group contains only base file#12080
danny0405 merged 2 commits intoapache:masterfrom
Zouxxyy:dev/hudi-8335

Conversation

@Zouxxyy
Copy link
Contributor

@Zouxxyy Zouxxyy commented Oct 11, 2024

Change Logs

For file groups without delta logs, we can make the following optimizations:
a) If the requested columns are not included in mandatoryColumns, they can be removed from requiredDataSchema.
b) Apply filters to reader for data skipping since no merging.

Impact

Optimize the performance of read mor table with file group contains only base file with filter

Risk level (write none, low medium or high below)

medium

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:S PR with lines of changes in (10, 100] label Oct 11, 2024
@hudi-bot
Copy link
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@Zouxxyy Zouxxyy closed this Oct 12, 2024
@Zouxxyy Zouxxyy reopened this Oct 12, 2024
@danny0405 danny0405 merged commit f9a8d8d into apache:master Oct 14, 2024
@Zouxxyy Zouxxyy deleted the dev/hudi-8335 branch October 14, 2024 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S PR with lines of changes in (10, 100]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants