Skip to content

[HUDI-9146][part3] Integrating FileGroup reader into Flink CDC reader#13357

Merged
danny0405 merged 2 commits intoapache:masterfrom
cshuo:HUDI-9146-part3
May 28, 2025
Merged

[HUDI-9146][part3] Integrating FileGroup reader into Flink CDC reader#13357
danny0405 merged 2 commits intoapache:masterfrom
cshuo:HUDI-9146-part3

Conversation

@cshuo
Copy link
Collaborator

@cshuo cshuo commented May 26, 2025

Change Logs

Integrating FileGroup reader into Flink CDC reader

Impact

Improve performance of flink CDC reader.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:L PR with lines of changes in (300, 1000] label May 26, 2025
@cshuo cshuo force-pushed the HUDI-9146-part3 branch from d91653e to dd74977 Compare May 27, 2025 02:16
@github-actions github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels May 27, 2025
@cshuo cshuo force-pushed the HUDI-9146-part3 branch 3 times, most recently from b9f4507 to 613985e Compare May 27, 2025 10:55
@cshuo cshuo force-pushed the HUDI-9146-part3 branch from 613985e to 98dd515 Compare May 27, 2025 12:33
GenericRecord historyAvroRecord = (GenericRecord) rowDataToAvroConverter.convert(tableSchema, existed);
HoodieRecord<IndexedRecord> merged = mergeRowWithLog(historyAvroRecord, record).get();
if (merged.getData() != historyAvroRecord) {
HoodieRecord<RowData> historyAvroRecord = new HoodieFlinkRecord(existed);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

historyAvroRecord -> historyRecord

@hudi-bot
Copy link
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@danny0405
Copy link
Contributor

@danny0405 danny0405 merged commit 94e4d37 into apache:master May 28, 2025
57 of 58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL PR with lines of changes > 1000

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants