HIVE-28998: Support Iceberg REST Catalog without authentication #5870
Conversation
...tore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthJWT.java
Outdated
Show resolved
Hide resolved
...tore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthJWT.java
Outdated
Show resolved
Hide resolved
...ore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java
Outdated
Show resolved
Hide resolved
...ore/metastore-rest-catalog/src/test/java/org/apache/iceberg/rest/TestHMSCatalogAuthNone.java
Outdated
Show resolved
Hide resolved
...tastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java
Outdated
Show resolved
Hide resolved
...tastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ServletSecurity.java
Outdated
Show resolved
Hide resolved
| @@ -1867,7 +1867,7 @@ public enum ConfVars { | |||
| " positive value will be used as-is." | |||
| ), | |||
| ICEBERG_CATALOG_SERVLET_AUTH("metastore.iceberg.catalog.servlet.auth", | |||
| "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("simple", "jwt"), | |||
| "hive.metastore.iceberg.catalog.servlet.auth", "jwt", new StringSetValidator("none", "simple", "jwt"), | |||
There was a problem hiding this comment.
can we update the HMS config in docker to NONE as well
There was a problem hiding this comment.
OK. I am considering if we should support SIMPLE and JWT. They are likely inherited from hive.metastore.properties.servlet.auth.
There was a problem hiding this comment.
That comes from PROPERTIES_SERVLET_AUTH - another servlet that has 0 usages. I don't understand why we can't have 1 generic config
There was a problem hiding this comment.
I think we should separate one at least for Iceberg. OAuth2 is not used in Hive's endpoints, but the Iceberg community prefers it. Another table format might prefer SPNEGO or something else if we support it. Anyway, a set of auth methods relies on the product, the community, and the client implementation.
There was a problem hiding this comment.
hive.metastore.iceberg.catalog.servlet.auth config rename has caused confusion. It's not meant to be specific to Iceberg - it's intended for use by the HMS RestCatalog (aka UnityCatalog) serving federated metadata (including Hive external and ACID tables)."
There was a problem hiding this comment.
It was a very good discussion. This discussion involved the further development direction and positioning of HMS in the future.
First, I think we should write a detailed proposal and discuss it clearly in the community email. Because I feel that many people are not clear about the positioning of the HMS rest catalog, which may cause various misunderstandings.
Returning to the current issue, I basically agree with Denys' statement that HMS(Rest) in the future will be similar to the Unity Catalog, capable of serving as an Iceberg rest catalog server (its primary function), and also able to register other catalogs (such as Hive tables from other HMS).
There was a problem hiding this comment.
@deniskuzZ @zhangbutao Thanks. I also think we should discuss the excellent vision on the mailing list. I want to support it. However, one point remains unclear to me.
Unity Catalog provides two types of endpoints: One is Unity REST API(/api/2.1/unity-catalog) and the other is Iceberg REST API(/api/2.1/unity-catalog/iceberg-rest). Unity REST API is likely to authenticate a user with Personal Access Token. Iceberg REST API is likely to authenticate a user with OAuth or PAT. I understand we want the equivalent of the Unity REST; Let's call it HMS v2 REST.
I don't understand why we can use the consistent auth across the endpoints. HMS v2 REST clients will be able to use SIMPLE(i.e., a client sends a user name via x-actor-username header) if we implement the client so. However, as a reality, most Iceberg REST clients probably do not support it. I think Trino has no method to use SIMPLE and JWT.
If I were saying something strange, I would follow the recommendation now.
There was a problem hiding this comment.
I'll try to create a draft with the proposal, so we can collaborate more effectively.
You might be right and we should consider the possibility of supporting multiple endpoints, each with its own authentication. I haven't thought much about it.
FYI, we are using JWT downstream to enable Snowflake integration with HMS Iceberg RestCatalog.
There was a problem hiding this comment.
This is a good discussion, but I started thinking it does not need to be a blocker to make the current Iceberg REST API minimally viable. I separated the debate in the following ticket.
https://issues.apache.org/jira/browse/HIVE-29038
There was a problem hiding this comment.
Yes, 100%. That's beyond the scope of this PR
okumin
force-pushed
the
HIVE-28998-rest-none
branch
from
6986d13 to
9258483
Compare
okumin
changed the title
asf-ci-hive
added
tests failed
tests pending
tests unstable
tests passed
and removed
tests pending
tests failed
tests unstable
labels
...metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Show resolved
Hide resolved
|
hi @okumin, could you please rebase, or if you are busy, i can help. we need this in 4.1 |
|
I probably think the consensus is that this is not a blocker, but I try to rebase this pr promptly. |
okumin
force-pushed
the
HIVE-28998-rest-none
branch
from
c5f4c79 to
7e09727
Compare
Somehow, with a constructor, `mvn test` didn't run it
|
@deniskuzZ I rebased it |
|
What changes were proposed in this pull request?
This PR would add a new authentication mode, NONE, to the Iceberg REST Catalog API. The currently supported SIMPLE and JWT are not central authentication. It is inherited from another Hive servlet. It appears that NONE and OAuth2 are widely used.
Why are the changes needed?
To acquire test users.
Does this PR introduce any user-facing change?
No. The feature has not been shipped.
How was this patch tested?
Unit and integration tests