Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support adding a new column with id and name column mapping mode in Delta Lake #17236

Merged
merged 2 commits into from May 29, 2023
Merged

Support adding a new column with id and name column mapping mode in Delta Lake #17236

merged 2 commits into from May 29, 2023

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Apr 26, 2023 •
edited

Description

Support adding a new column with id and name column mapping mode in Delta Lake

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Support adding a new column with `id` and `name` column mapping mode. ({issue}`17236`)

@cla-bot cla-bot bot added the cla-signed label Apr 26, 2023
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from 39adb0b to 3487970 Compare April 26, 2023 01:28
@ebyhr ebyhr self-assigned this Apr 26, 2023
@ebyhr ebyhr mentioned this pull request Apr 26, 2023
Closed
10 tasks
@github-actions github-actions bot added delta-lake Delta Lake connector tests:hive labels Apr 26, 2023
@ebyhr ebyhr marked this pull request as draft April 26, 2023 11:57
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch 4 times, most recently from 3908e5a to 59f9330 Compare April 28, 2023 04:33
@ebyhr ebyhr marked this pull request as ready for review April 28, 2023 08:34
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from 59f9330 to 32209f9 Compare April 29, 2023 05:53
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch 2 times, most recently from d63642c to e03a5ed Compare May 8, 2023 23:12
@ebyhr
Copy link
Member Author

ebyhr commented May 8, 2023

Rebased on master to resolve conflicts.

@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from e03a5ed to 37034c7 Compare May 12, 2023 04:54
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch 3 times, most recently from e2d1b94 to 9a7087b Compare May 16, 2023 06:35
@ebyhr ebyhr mentioned this pull request May 16, 2023
Merged
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from 9a7087b to be7a115 Compare May 16, 2023 12:51
@ebyhr ebyhr requested review from findepi and removed request for krvikash May 16, 2023 12:51
@findinpath findinpath mentioned this pull request May 17, 2023
Merged
@ebyhr ebyhr removed the request for review from findepi May 17, 2023 09:36
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch 2 times, most recently from 141d68a to ba58e83 Compare May 18, 2023 04:36
@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from ba58e83 to 08c1f95 Compare May 18, 2023 05:59
findinpath
findinpath approved these changes May 18, 2023
View reviewed changes
Copy link
Contributor

@findinpath findinpath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 👍

@ebyhr ebyhr requested a review from findepi May 18, 2023 06:56
@ebyhr
Copy link
Member Author

ebyhr commented May 18, 2023

CI hit concurrent modification error. #17550 will fix it.

findepi
findepi reviewed May 23, 2023
View reviewed changes
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakePageSinkProvider.java Outdated
Comment on lines 105 to 106
// TODO support ID and NAME column mapping
ColumnMappingMode columnMappingMode = NONE;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this take actual mapping + verify it's NONE?

Copy link
Member Author

@ebyhr ebyhr May 24, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#17565 will add column mapping to DeltaLakeOutputTableHandle and handle the actual mapping here.
I think the verification isn't necessary verifying in this PR.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakePageSinkProvider.java Outdated
Comment on lines 110 to 111
columnNames.add(column.getColumnName());
columnTypes.put(column.getColumnName(), serializeColumnType(columnMappingMode, new AtomicInteger(), column.getType()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be done on the coordinator?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to DeltaLakeMetadata.

@ebyhr ebyhr force-pushed the ebi/delta-add-column-with-mapping-mode branch from 08c1f95 to 1476ca2 Compare May 24, 2023 01:05
@ebyhr
Copy link
Member Author

ebyhr commented May 28, 2023

Addressed comments.

@ebyhr ebyhr merged commit a5e6495 into master May 29, 2023
60 of 63 checks passed
@ebyhr ebyhr deleted the ebi/delta-add-column-with-mapping-mode branch May 29, 2023 12:29
@ebyhr ebyhr mentioned this pull request May 29, 2023
Closed
@github-actions github-actions bot added this to the 419 milestone May 29, 2023
@colebow colebow mentioned this pull request Jun 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@findepi findepi findepi approved these changes

@findinpath findinpath findinpath approved these changes

@pajaks pajaks Awaiting requested review from pajaks

Assignees

@ebyhr ebyhr

Labels
cla-signed delta-lake Delta Lake connector
Milestone
419
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ebyhr @findepi @findinpath