Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] HIVE-26628: Iceberg table is created when running explain ctas command #3670

Closed
wants to merge 23 commits into from
Closed

[WIP] HIVE-26628: Iceberg table is created when running explain ctas command #3670

wants to merge 23 commits into from

Conversation

kasakrisz
Copy link
Contributor

@kasakrisz kasakrisz commented Oct 13, 2022 •
edited

What changes were proposed in this pull request?

This patch is an attempt to refactor the way ctas is executed:

  • Do not create the table in HiveIcebergSerDe since it is also created at compile time.
  • Add a DDLTask before the TezTask to create the Iceberg table.
  • Collect the properties added to jobconfig from the Serde object and location and fileio from HiveCatalog. This is a limitation since we loose the support if other catalogs. Location can be calculated at compile time using Hive.getTranslateTableDryrun
  • Persist the new table metaobject to a temp file when committing the table creation.
  • Read back the table metaobject anytime it is required from the TezTask and the MoveTask.

Execution before this patch:
HiveIcebergSerDe - table creation
TezTask - run query and write datafiles
MoveTask- commit writes

Execution with this patch:
DDLTask - table creation
TezTask - run query and write datafiles
MoveTask- commit writes

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestIcebergLlapLocalCliDriver -Dqfile=ctas_iceberg_orc.q -pl itests/qtest-iceberg -Piceberg -Pitests

@kasakrisz kasakrisz marked this pull request as draft October 13, 2022 10:37
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from a7e928b to d0fc361 Compare October 19, 2022 11:08
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from d0fc361 to 5960d7d Compare October 20, 2022 13:40
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from 5960d7d to 09481cf Compare October 21, 2022 13:39
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from 09481cf to 6c37c44 Compare October 24, 2022 15:33
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from 6c37c44 to be76101 Compare October 25, 2022 11:39
@kasakrisz
Copy link
Contributor Author

issue is fixed by #3745

kasakrisz and others added 23 commits November 30, 2022 09:27
@kasakrisz
HIVE-26628: Iceberg table is created when running explain ctas command
41d1458
@kasakrisz
store table name in query state for rollback
946fcc2
@kasakrisz
Create external test tables. Truncating managed iceberg tables causes…
c2c6727 
… deadlock.
@kasakrisz
set table identifier after loading table object
6bff396
@kasakrisz
support MV
29090b4
@kasakrisz
support MV
cc61ed4
@kasakrisz
rebase
2a14a75
@kasakrisz
populate predicted location
95ed530
@kasakrisz
remove tmp file
46e0906
@kasakrisz
remove fileio serialization
5cd6711
@kasakrisz
take the location from the db and fallback to default if not set
4f0a8d3
@kasakrisz
return null if table object file not exists
a7042a9
@kasakrisz
handle non partitined tables
af608aa
@kasakrisz
remove property CTAS_LOCATION and use META_TABLE_LOCATION instead if …
7280e17 
…StorageHandler present
@kasakrisz
pass location to commit
890cedd
@kasakrisz
add nullcheck for location
a170007
@kasakrisz
revert MV related code
a86ad08
@kasakrisz
rename directInsert
a0ef372
@kasakrisz
Revert "support MV"
f6ff7e1 
This reverts commit 6853773.
@kasakrisz
extract tmp file handling
650e87c
@kasakrisz
throw exception instead of returning null
b6e1fbb
@kasakrisz
indentation
a3211d1
@kasakrisz
refac: !Boolean.parseBoolean(catalogProperties.getProperty(hive_metas…
529ad73 
…toreConstants.TABLE_IS_CTAS))

reuse String tableIdentifier = catalogProperties.getProperty(Catalogs.NAME);
@kasakrisz kasakrisz force-pushed the HIVE-26628-master-explain-iceberg-ctas branch from 6a6b952 to 529ad73 Compare November 30, 2022 12:44
@kasakrisz
Copy link
Contributor Author

fixed by #3802

@kasakrisz kasakrisz closed this Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@miklosgergely miklosgergely Awaiting requested review from miklosgergely

Assignees
No one assigned
Labels
tests failed
Projects
None yet
Milestone
No milestone
2 participants
@kasakrisz @kgyrtkirk