Skip to content

[#4814] Improvement (catalog-lakehouse-paimon): use purgeTable for Paimon instead of dropTable inGravitino#4806

Merged
FANNG1 merged 9 commits intoapache:mainfrom
caican00:paimon-doc-droptable
Aug 30, 2024
Merged

[#4814] Improvement (catalog-lakehouse-paimon): use purgeTable for Paimon instead of dropTable inGravitino#4806
FANNG1 merged 9 commits intoapache:mainfrom
caican00:paimon-doc-droptable

Conversation

@caican00
Copy link
Contributor

@caican00 caican00 commented Aug 30, 2024 •
edited
Loading

What changes were proposed in this pull request?

the dropTable in Paimon will both delete the metadata and data and skip the trash, as discussed in #1436 , Gravitino Paimon catalog should use purgeTable not dropTable

Why are the changes needed?

#4814

Does this PR introduce any user-facing change?

yes, updated the doc.

How was this patch tested?

existing ITs and UTs.

@caican00
Copy link
Contributor Author

caican00 commented Aug 30, 2024

cc @FANNG1

@FANNG1
Copy link
Contributor

FANNG1 commented Aug 30, 2024

as discussed in #1436 , paimon supports purgeTable not dropTable, as it will delete table metadata and data immediately, could you change corresponding code too ? cc @mchades

@caican00
Copy link
Contributor Author

caican00 commented Aug 30, 2024

as discussed in #1436 , paimon supports purgeTable not dropTable, as it will delete table metadata and data immediately, could you change corresponding code too ? cc @mchades

ok

FANNG1
FANNG1 reviewed Aug 30, 2024
View reviewed changes
@FANNG1
Copy link
Contributor

FANNG1 commented Aug 30, 2024

LGTM except one comment

@caican00 caican00 changed the title [MINOR][doc] Polish the paimon document [#4814] Improvement (catalog-lakehouse-paimon): use purgeTable for Paimon instead of dropTable inGravitino Aug 30, 2024
FANNG1
FANNG1 previously approved these changes Aug 30, 2024
View reviewed changes
@FANNG1
Copy link
Contributor

FANNG1 commented Aug 30, 2024

@caican00 could you fix the ci failure?

CatalogPaimonKerberosFilesystemIT > testPaimonWithKerberos() FAILED
    java.lang.UnsupportedOperationException: Failed to operate table(s) [test_table_2c03604e] operation [DROP] under schema [test_schema_161bf627], reason [Paimon dropTable will both remove the metadata and data, please use purgeTable instead in Gravitino.]
    java.lang.UnsupportedOperationException: Paimon dropTable will both remove the metadata and data, please use purgeTable instead in Gravitino.
    	at org.apache.gravitino.catalog.lakehouse.paimon.PaimonCatalogOperations.dropTable(PaimonCatalogOperations.java:441)
    	at org.apache.gravitino.catalog.TableOperationDispatcher.lambda$dropTable$15(TableOperationDispatcher.java:301)
    	at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithTableOps$1(CatalogManager.java:132)
    	at org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)
    	at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithTableOps(CatalogManager.java:127)
    	at org.apache.gravitino.catalog.TableOperationDispatcher.lambda$dropTable$16(TableOperationDispatcher.java:301)

@caican00
Copy link
Contributor Author

caican00 commented Aug 30, 2024

@caican00 could you fix the ci failure?

CatalogPaimonKerberosFilesystemIT > testPaimonWithKerberos() FAILED
    java.lang.UnsupportedOperationException: Failed to operate table(s) [test_table_2c03604e] operation [DROP] under schema [test_schema_161bf627], reason [Paimon dropTable will both remove the metadata and data, please use purgeTable instead in Gravitino.]
    java.lang.UnsupportedOperationException: Paimon dropTable will both remove the metadata and data, please use purgeTable instead in Gravitino.
    	at org.apache.gravitino.catalog.lakehouse.paimon.PaimonCatalogOperations.dropTable(PaimonCatalogOperations.java:441)
    	at org.apache.gravitino.catalog.TableOperationDispatcher.lambda$dropTable$15(TableOperationDispatcher.java:301)
    	at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithTableOps$1(CatalogManager.java:132)
    	at org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)
    	at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithTableOps(CatalogManager.java:127)
    	at org.apache.gravitino.catalog.TableOperationDispatcher.lambda$dropTable$16(TableOperationDispatcher.java:301)

done

@FANNG1 FANNG1 merged commit ec93308 into apache:main Aug 30, 2024
github-actions bot pushed a commit that referenced this pull request Aug 30, 2024
@caican00 @web-flow
[#4814] Improvement (catalog-lakehouse-paimon): use purgeTable for Pa…
96014e0 
…imon instead of dropTable inGravitino (#4806)

### What changes were proposed in this pull request?

the dropTable in Paimon will both delete the metadata and data and skip
the trash, as discussed in
#1436 , Gravitino Paimon
catalog should use purgeTable not dropTable

### Why are the changes needed?
#4814

### Does this PR introduce any user-facing change?
yes, updated the doc.

### How was this patch tested?
existing ITs and UTs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FANNG1 FANNG1 FANNG1 approved these changes

Assignees

@caican00 caican00

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@caican00 @FANNG1