-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AMORO-2276]: UnifiiedCatalog for Spark Engine #2269
Merged
Merged
Changes from 23 commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
9136dc7
Squash Merged from branch:external-unified-catalog
baiyangtx 9669d64
Merge branch 'master' into spark-unified-catalog
baiyangtx 138e907
spark catalog
baiyangtx 59a6395
Merge branch 'master' into spark-unified-catalog
baiyangtx 1565323
spark test base refactor to adapt UnifiedCatalog test and mixed catal…
baiyangtx 65ed1d6
run tests/ some passed
baiyangtx acd8bea
fix unit tests in truncate sql.
baiyangtx 7e174dc
unified catalog refactors
baiyangtx 99fdddf
support mixed-iceberg
baiyangtx a90a403
paimon catalog loaded.
baiyangtx 5984e0d
support unified catalog in spark
baiyangtx b49a85c
refactor mixed spark catalog. extract common base.
baiyangtx 6f49f01
UnifiedCatalog test.
baiyangtx 84f3c03
remove useless codes
baiyangtx 36a5651
Merge branch 'master' into spark-unified-catalog
baiyangtx 5114a3b
merge from master
baiyangtx 7ae8088
fix unit test cases.
baiyangtx 9d0974e
fix complie error
baiyangtx 0b120db
fix unit tests
baiyangtx 5af7f23
unified catalog and test from spark engine.
baiyangtx 0002e38
Merge branch 'master' into spark-unified-catalog
baiyangtx 190af26
unified session catalog and test case
baiyangtx 80e103b
support sub table for unified session catalog.
baiyangtx 1d0bfad
spotless:apply
baiyangtx 17c731b
remove useless annotation
baiyangtx 04fb603
support paimon in session catalog
baiyangtx bf40a59
Merge branch 'master' into spark-unified-catalog
baiyangtx c07bdef
merged from master && mixed module.
baiyangtx d83e778
Merge branch 'master' into spark-unified-catalog
baiyangtx 919a7a1
fix code reviewer comments.
baiyangtx 2fffedb
Merge branch 'master' into spark-unified-catalog
baiyangtx 05f0465
add catalog() to TableIdWithFormat
baiyangtx 4d24cad
Add docs to class
baiyangtx dd6fc3b
spotless
baiyangtx 153cbeb
fix review comments
baiyangtx 2f18c81
fix review comments
baiyangtx File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -45,16 +45,11 @@ public class HiveCatalogTestHelper implements CatalogTestHelper { | |
private final Configuration hiveConf; | ||
|
||
public static CatalogTestHelper build(Configuration hiveConf, TableFormat... formats) { | ||
Preconditions.checkArgument(formats.length == 1, "Cannot support multiple table formats"); | ||
// Preconditions.checkArgument(formats.length == 1, "Cannot support multiple table formats"); | ||
return new HiveCatalogTestHelper(formats[0], hiveConf); | ||
} | ||
|
||
public HiveCatalogTestHelper(TableFormat tableFormat, Configuration hiveConf) { | ||
Preconditions.checkArgument( | ||
tableFormat.equals(TableFormat.ICEBERG) | ||
|| tableFormat.equals(TableFormat.MIXED_HIVE) | ||
|| tableFormat.equals(TableFormat.MIXED_ICEBERG), | ||
"Cannot support table format:" + tableFormat); | ||
this.tableFormat = tableFormat; | ||
this.hiveConf = hiveConf; | ||
} | ||
|
@@ -72,9 +67,8 @@ public TableFormat tableFormat() { | |
@Override | ||
public CatalogMeta buildCatalogMeta(String baseDir) { | ||
Map<String, String> properties = Maps.newHashMap(); | ||
if (TableFormat.MIXED_ICEBERG == tableFormat) { | ||
properties.put(CatalogProperties.URI, hiveConf.get(HiveConf.ConfVars.METASTOREURIS.varname)); | ||
} | ||
properties.put(CatalogProperties.URI, hiveConf.get(HiveConf.ConfVars.METASTOREURIS.varname)); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should split these codes first? I think the Flink engine also need it. |
||
properties.put(CatalogProperties.WAREHOUSE_LOCATION, baseDir); | ||
return CatalogTestHelpers.buildHiveCatalogMeta( | ||
TEST_CATALOG_NAME, properties, hiveConf, tableFormat); | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I try to complete this #2344 Flink Unified Catalog(Iceberg format table), I find out that this API is aslo needed.