Skip to content

[hive] Dropping table deletes table directory to avoid schema in filesystem exists#772

Merged
JingsongLi merged 2 commits intoapache:masterfrom
SteNicholas:PAIMON-694
Apr 7, 2023
Merged

[hive] Dropping table deletes table directory to avoid schema in filesystem exists#772
JingsongLi merged 2 commits intoapache:masterfrom
SteNicholas:PAIMON-694

Conversation

@SteNicholas
Copy link
Copy Markdown
Member

@SteNicholas SteNicholas commented Mar 30, 2023

Purpose

HiveCatalog#dropTable interface should delete table directory to avoid schema in filesystem exists, especially for external table which doesn't delete directory in filesystem so that recreating the same table cause exception that schema in filesystem exists.

Fix #694.

Tests

HiveCatalog#dropTable forces to delete table directory to avoid schema in filesystem exists.

API and Format

(Does this change affect API or storage format)

Documentation

(Does this change introduce a new feature)

@SteNicholas SteNicholas reopened this Mar 30, 2023
@SteNicholas SteNicholas closed this Apr 5, 2023
@SteNicholas SteNicholas reopened this Apr 5, 2023
@SteNicholas SteNicholas requested a review from FangYongs April 5, 2023 05:26
JingsongLi
JingsongLi approved these changes Apr 5, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas closed this Apr 6, 2023
@SteNicholas SteNicholas reopened this Apr 6, 2023
@FangYongs
Copy link
Copy Markdown
Contributor

FangYongs commented Apr 7, 2023

+1

@JingsongLi JingsongLi merged commit 2165835 into apache:master Apr 7, 2023
Alibaba-HZY pushed a commit to Alibaba-HZY/incubator-paimon that referenced this pull request Apr 11, 2023
[core] Add streaming read mode from option (apache#778)
2a62022 
[core]  Renaming streaming-read-from to streaming-read-mode (apache#778)
parent 4a56565
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194324 +0800

parent 4a56565
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194307 +0800

[core]  Renaming streaming-read-from to streaming-read-mode(apache#778)

[core]  Renaming streaming-read-from to streaming-read-mode
 (apache#778)

[core] generate doc  (apache#778)

[core] add test (apache#778)

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

[core] add test (apache#778)

[core]  commit
 (apache#778)

[doc] Move Watermark and Bounded Stream to Append only table page

[deploy] Publish paimon-shade to snapshot

[core] Introduce SecurityContext (apache#834)

[deps] Use paimon-shade dependencies (apache#838)

[Document] Adds the Trino feature that create/drop table, create/drop database in the overview document (apache#835)

[docs] Add document for MySqlSyncTableAction (apache#828)

[doc] Document file format (apache#840)

[ci] Add licensing check (apache#839)

[common] Use caffeine to replace with guava cache (apache#665)

[release] Added incubating name in release file (apache#845)

[doc] add flink 1.17 version in Engines Overview (apache#847)

[hive] Dropping table deletes table directory to avoid schema in filesystem exists (apache#772)

[hive] Support hive-conf-dir and hadoop-conf-dir option in hive catalog (apache#831)

[hotfix] Do not swallow InterruptedException in SnapshotManager

[core][flink] Refactor BucketComputer and StreamPartitioner to gather bucketing / partitioning core logic in the same class (apache#844)

[doc] Document Java API (apache#849)

[README] Document mark generated-sources as Sources Root

[flink][bug] MergeIntoAction and DeleteAction work abnormally with partial update merge engine (apache#829)

[hotfix] Set directExecutor to caffeine Cache

Resolve the problem which Sonar reported that InterruptedException need to re-throw or re-interrupt (apache#862)

[hotfix] Enable StreamingReadWriteTableWithKafkaLogITCase.testReadInsertOnlyChangelogFromEnormousTimestamp (apache#854)

[test] Migrate MinioTestContainer to paimon-s3 (apache#869)

[flink] Assign splits evenly in StaticFileStoreSplitEnumerator (apache#856)

[core] Compacted StartingScanner is incorrect (apache#857)

[core]  revert flink.md(apache#778)

[core] change dose to does (apache#778)
Alibaba-HZY pushed a commit to Alibaba-HZY/incubator-paimon that referenced this pull request Apr 11, 2023
[core] Add streaming read mode from option (apache#778)
7543332 
[core] Add streaming read mode from option  (apache#778)

[core]  Renaming streaming-read-from to streaming-read-mode (apache#778)
parent 4a56565
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194324 +0800

parent 4a56565
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194307 +0800

[core]  Renaming streaming-read-from to streaming-read-mode(apache#778)

[core]  Renaming streaming-read-from to streaming-read-mode
 (apache#778)

[core] generate doc  (apache#778)

[core] add test (apache#778)

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

[core] add test (apache#778)

[core]  commit
 (apache#778)

[doc] Move Watermark and Bounded Stream to Append only table page

[deploy] Publish paimon-shade to snapshot

[core] Introduce SecurityContext (apache#834)

[deps] Use paimon-shade dependencies (apache#838)

[Document] Adds the Trino feature that create/drop table, create/drop database in the overview document (apache#835)

[docs] Add document for MySqlSyncTableAction (apache#828)

[doc] Document file format (apache#840)

[ci] Add licensing check (apache#839)

[common] Use caffeine to replace with guava cache (apache#665)

[release] Added incubating name in release file (apache#845)

[doc] add flink 1.17 version in Engines Overview (apache#847)

[hive] Dropping table deletes table directory to avoid schema in filesystem exists (apache#772)

[hive] Support hive-conf-dir and hadoop-conf-dir option in hive catalog (apache#831)

[hotfix] Do not swallow InterruptedException in SnapshotManager

[core][flink] Refactor BucketComputer and StreamPartitioner to gather bucketing / partitioning core logic in the same class (apache#844)

[doc] Document Java API (apache#849)

[README] Document mark generated-sources as Sources Root

[flink][bug] MergeIntoAction and DeleteAction work abnormally with partial update merge engine (apache#829)

[hotfix] Set directExecutor to caffeine Cache

Resolve the problem which Sonar reported that InterruptedException need to re-throw or re-interrupt (apache#862)

[hotfix] Enable StreamingReadWriteTableWithKafkaLogITCase.testReadInsertOnlyChangelogFromEnormousTimestamp (apache#854)

[test] Migrate MinioTestContainer to paimon-s3 (apache#869)

[flink] Assign splits evenly in StaticFileStoreSplitEnumerator (apache#856)

[core] Compacted StartingScanner is incorrect (apache#857)

[core]  revert flink.md(apache#778)

[core] change dose to does (apache#778)

parent 2a62022
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226783 +0800

parent 2a62022
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226730 +0800

parent 2a62022
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800

[core] Add streaming read from option (apache#778)

[core] add test (apache#778)

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java

Co-authored-by: Nicholas Jiang <programgeek@163.com>

[core] add test (apache#778)

merger

sdda 这是一个 3 个提交的组合。

[core]  commit1
 (apache#778)

[core]  commit2
 (apache#778)

[core]  commit3
 (apache#778)

[core]  do commit 1(apache#778)

[core]  do commit 2(apache#778)

sdda 这是一个 3 个提交的组合。

[core]  commit1
 (apache#778)

[core]  commit2
 (apache#778)

[core]  commit3
 (apache#778)

[core] Add streaming read mode option (apache#778)
parent 3acede2
author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226824 +0800
committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226824 +0800

[core] Add streaming read from option (apache#778)

[core] revert flink.md (apache#778)

Delete AbstractChannelComputer.java

delete
@Zouxxyy
Copy link
Copy Markdown
Contributor

Zouxxyy commented May 5, 2023 •
edited
Loading

@SteNicholas Why do we delete data when drop external table?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JingsongLi JingsongLi JingsongLi approved these changes

@FangYongs FangYongs Awaiting requested review from FangYongs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Recreate the same paimon hive catalog table after drop it failed because of schema file already exists

4 participants

@SteNicholas @FangYongs @Zouxxyy @JingsongLi