Hive: Configure catalog type on table level. #2129
Conversation
There was a problem hiding this comment.
Thanks @lcspinter, I was planning a similar change but you beat me to it :-D
My only feedback is to use the same idiom as the rest of iceberg for constructing catalogs, I will test this today with the NessieCatalog and post here if successful.
.../test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithMultipleCatalogs.java
Outdated
Show resolved
Hide resolved
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalogs.java
Outdated
Show resolved
Hide resolved
lcspinter
force-pushed
the
CDPD-20021
branch
5 times, most recently
from
5869fa5
to
27f91dc
Compare
lcspinter
force-pushed
the
CDPD-20021
branch
2 times, most recently
from
f274a3c
to
251598f
Compare
|
I am comfortable with the current solution, but I see that you still have some open comments there. @rymurr, @marton-bod: Could we please finalize the review of this patch? @rdblue: Any more comments before merge? Thanks, |
|
Thanks, @rymurr @pvary @marton-bod @rdblue for the reviews! |
|
Merged the PR to master. @lcspinter: Could we please update the docs about the hive catalog configurations? Thanks, |
This reverts commit db8248c.
This reverts commit db8248c.
The current catalog configuration is stored in the main hive configuration, by setting the
iceberg.mr.catalog. This works perfectly when the user is working on a dataset when all the tables are coming from the same catalog.In case of operations involving multiple tables from different catalogs, this implementation fails to serve the need.
This PR provides a solution for this, by implementing Spark-like catalog configuration. The catalog configuration is stored in the hive main configuration, the same way as it handled in Spark, and on table level, the name of the catalog and the table identifier is stored. If catalog name is not defined on the table, a default catalog is used.
Here is an example of how to configure a Hadoop-catalog
In the main hive configuration we store the following properties:
iceberg.catalog.<catalog_name>.type = hadoopiceberg.catalog.<catalog_name>.warehouse = somelocationOn the table level we have the following properties:
iceberg.mr.table.catalog = <catalog_name>iceberg.mr.table.identifier = <database.table_name>If property
iceberg.mr.table.catalogis missing from the table, it starts looking for a catalogue definition with the name "default". If that is also missing, the original implementation is used, where the propertyiceberg.mr.catalogstores the catalog information.