From 5cd0cc9d226ef739e40aaff594977043c2aaa2e4 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Thu, 13 Nov 2025 09:26:08 +0530 Subject: [PATCH 1/6] Added GCS related Properties for vending credentials. Added line width to 120 char --- .../configuring-polaris-for-production.md | 146 +++++++++--------- 1 file changed, 76 insertions(+), 70 deletions(-) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production.md b/site/content/in-dev/unreleased/configuring-polaris-for-production.md index 2a1efe36bb..6b46701afb 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production.md @@ -23,35 +23,39 @@ type: docs weight: 600 --- -The default server configuration is intended for development and testing. When you deploy Polaris in production, -review and apply the following checklist: +The default server configuration is intended for development and testing. When you deploy Polaris in production, review +and apply the following checklist: + - [ ] Configure OAuth2 keys - [ ] Enforce realm header validation (`require-header=true`) - [ ] Use a durable metastore (JDBC + PostgreSQL) - [ ] Bootstrap valid realms in the metastore - [ ] Disable local FILE storage +- [ ] Polaris Server Header +- [ ] Upgrade considerations +- [ ] Cloud Storage Specific Configuration ### Configure OAuth2 -Polaris authentication requires specifying a token broker factory type. Two implementations are -supported out of the box: +Polaris authentication requires specifying a token broker factory type. Two implementations are supported out of the +box: - [rsa-key-pair] uses a pair of public and private keys; - [symmetric-key] uses a shared secret. -[rsa-key-pair]: https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTRSAKeyPairFactory.java -[symmetric-key]: https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTSymmetricKeyFactory.java +[rsa-key-pair]: + https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTRSAKeyPairFactory.java +[symmetric-key]: + https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTSymmetricKeyFactory.java By default, Polaris uses `rsa-key-pair`, with randomly generated keys. -{{< alert important >}} -The default `rsa-key-pair` configuration is not suitable when deploying many replicas of Polaris, -as each replica will have its own set of keys. This will cause token validation to fail when a -request is routed to a different replica than the one that issued the token. -{{< /alert >}} +{{< alert important >}} The default `rsa-key-pair` configuration is not suitable when deploying many replicas of +Polaris, as each replica will have its own set of keys. This will cause token validation to fail when a request is +routed to a different replica than the one that issued the token. {{< /alert >}} -It is highly recommended to configure Polaris with previously-generated RSA keys. This can be done -by setting the following properties: +It is highly recommended to configure Polaris with previously-generated RSA keys. This can be done by setting the +following properties: ```properties polaris.authentication.token-broker.type=rsa-key-pair @@ -73,28 +77,25 @@ polaris.authentication.token-broker.type=symmetric-key polaris.authentication.token-broker.symmetric-key.file=/tmp/symmetric.key ``` -Note: it is also possible to set the symmetric key secret directly in the configuration file. If -possible, pass the secret as an environment variable to avoid storing sensitive information in the -configuration file: +Note: it is also possible to set the symmetric key secret directly in the configuration file. If possible, pass the +secret as an environment variable to avoid storing sensitive information in the configuration file: ```properties polaris.authentication.token-broker.symmetric-key.secret=${POLARIS_SYMMETRIC_KEY_SECRET} ``` -Finally, you can also configure the token broker to use a maximum lifespan by setting the following -property: +Finally, you can also configure the token broker to use a maximum lifespan by setting the following property: ```properties polaris.authentication.token-broker.max-token-generation=PT1H ``` -Typically, in Kubernetes, you would define the keys as a `Secret` and mount them as files in the -container. +Typically, in Kubernetes, you would define the keys as a `Secret` and mount them as files in the container. ### Realm Context Resolver -By default, Polaris resolves realms based on incoming request headers. You can configure the realm -context resolver by setting the following properties in `application.properties`: +By default, Polaris resolves realms based on incoming request headers. You can configure the realm context resolver by +setting the following properties in `application.properties`: ```properties polaris.realm-context.realms=POLARIS,MY-REALM @@ -103,41 +104,38 @@ polaris.realm-context.header-name=Polaris-Realm Where: -- `realms` is a comma-separated list of allowed realms. This setting _must_ be correctly configured. - At least one realm must be specified. -- `header-name` is the name of the header used to resolve the realm; by default, it is - `Polaris-Realm`. +- `realms` is a comma-separated list of allowed realms. This setting _must_ be correctly configured. At least one realm + must be specified. +- `header-name` is the name of the header used to resolve the realm; by default, it is `Polaris-Realm`. -If a request contains the specified header, Polaris will use the realm specified in the header. If -the realm is not in the list of allowed realms, Polaris will return a `404 Not Found` response. +If a request contains the specified header, Polaris will use the realm specified in the header. If the realm is not in +the list of allowed realms, Polaris will return a `404 Not Found` response. -If a request _does not_ contain the specified header, however, by default Polaris will use the first -realm in the list as the default realm. In the above example, `POLARIS` is the default realm and -would be used if the `Polaris-Realm` header is not present in the request. +If a request _does not_ contain the specified header, however, by default Polaris will use the first realm in the list +as the default realm. In the above example, `POLARIS` is the default realm and would be used if the `Polaris-Realm` +header is not present in the request. -This is not recommended for production use, as it may lead to security vulnerabilities. To avoid -this, set the following property to `true`: +This is not recommended for production use, as it may lead to security vulnerabilities. To avoid this, set the following +property to `true`: ```properties polaris.realm-context.require-header=true ``` -This will cause Polaris to also return a `404 Not Found` response if the realm header is not present -in the request. +This will cause Polaris to also return a `404 Not Found` response if the realm header is not present in the request. ### Metastore Configuration -A metastore should be configured with an implementation that durably persists Polaris entities. By -default, Polaris uses an in-memory metastore. +A metastore should be configured with an implementation that durably persists Polaris entities. By default, Polaris uses +an in-memory metastore. -{{< alert important >}} -The default in-memory metastore is not suitable for production use, as it will lose all data -when the server is restarted; it is also unusable when multiple Polaris replicas are used. -{{< /alert >}} +{{< alert important >}} The default in-memory metastore is not suitable for production use, as it will lose all data +when the server is restarted; it is also unusable when multiple Polaris replicas are used. {{< /alert >}} -To enable a durable metastore, configure your system to use the Relational JDBC-backed metastore. -This implementation leverages Quarkus for datasource management and supports configuration through -environment variables or JVM -D flags at startup. For more information, refer to the [Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file). +To enable a durable metastore, configure your system to use the Relational JDBC-backed metastore. This implementation +leverages Quarkus for datasource management and supports configuration through environment variables or JVM -D flags at +startup. For more information, refer to the +[Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file). Configure the metastore by setting the following ENV variables: @@ -149,28 +147,24 @@ QUARKUS_DATASOURCE_PASSWORD= QUARKUS_DATASOURCE_JDBC_URL= ``` +The relational JDBC metastore is a Quarkus-managed datasource and only supports Postgres and H2 as of now. Please refer +to the documentation here: [Configure data sources in Quarkus](https://quarkus.io/guides/datasource) -The relational JDBC metastore is a Quarkus-managed datasource and only supports Postgres and H2 as of now. -Please refer to the documentation here: -[Configure data sources in Quarkus](https://quarkus.io/guides/datasource) - -{{< alert important >}} -Be sure to secure your metastore backend since it will be storing sensitive data and catalog -metadata. -{{< /alert >}} +{{< alert important >}} Be sure to secure your metastore backend since it will be storing sensitive data and catalog +metadata. {{< /alert >}} Note: Polaris will always create schema 'polaris_schema' during bootstrap under the configured database. ### Bootstrapping -Before using Polaris, you must **bootstrap** the metastore. This is a manual operation that must be -performed **only once** for each realm in order to prepare the metastore to integrate with Polaris. +Before using Polaris, you must **bootstrap** the metastore. This is a manual operation that must be performed **only +once** for each realm in order to prepare the metastore to integrate with Polaris. -By default, when bootstrapping a new realm, Polaris will create randomised `CLIENT_ID` and -`CLIENT_SECRET` for the `root` principal and store their hashes in the metastore backend. +By default, when bootstrapping a new realm, Polaris will create randomised `CLIENT_ID` and `CLIENT_SECRET` for the +`root` principal and store their hashes in the metastore backend. -Depending on your database, this may not be convenient as the generated credentials are not stored -in clear text in the database. +Depending on your database, this may not be convenient as the generated credentials are not stored in clear text in the +database. In order to provide your own credentials for `root` principal (so you can request tokens via `api/catalog/v1/oauth/tokens`), use the [Polaris Admin Tool]({{% ref "admin-tool" %}}) @@ -196,9 +190,9 @@ Which should return an access token: } ``` -If you used a non-default realm name, add the appropriate request header to the `curl` command, -otherwise Polaris will resolve the realm to the first one in the configuration -`polaris.realm-context.realms`. Here is an example to set realm header: +If you used a non-default realm name, add the appropriate request header to the `curl` command, otherwise Polaris will +resolve the realm to the first one in the configuration `polaris.realm-context.realms`. Here is an example to set realm +header: ```bash curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \ @@ -210,27 +204,39 @@ curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \ ``` ### Disable FILE Storage Type -By default, Polaris allows using the local file system (`FILE`) for catalog storage. This is fine for testing, -but **not recommended for production**. To disable it, set the supported storage types like this: + +By default, Polaris allows using the local file system (`FILE`) for catalog storage. This is fine for testing, but **not +recommended for production**. To disable it, set the supported storage types like this: + ```hocon polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES" = [ "S3", "Azure" ] ``` + Leave out `FILE` to prevent its use. Only include the storage types your setup needs. ### Polaris Server Header -Polaris can emit an informational `Server` HTTP response header using Quarkus' built-in header -configuration. Add the following property to one of the supported configuration sources (for example, -`application.properties`) to enable it with the Polaris version string: +Polaris can emit an informational `Server` HTTP response header using Quarkus' built-in header configuration. Add the +following property to one of the supported configuration sources (for example, `application.properties`) to enable it +with the Polaris version string: ```properties quarkus.http.header."Server".value=Polaris/${quarkus.application.version} ``` -If you prefer to scope the header to specific environments, only set the property for the desired -profile (for example, `%prod`). +If you prefer to scope the header to specific environments, only set the property for the desired profile (for example, +`%prod`). + +### Cloud Storage Specific Configuration + +GCS + Polaris: When using token vending for fine-grained access in Google Cloud Storage (GCS) with Apache Iceberg on +Polaris, ensure that both IAM roles and HNS ACLs (if HNS is enabled) are properly configured. Even with the correct IAM +role (e.g., `roles/storage.objectAdmin`), access to paths such as `gs:///idsp_ns/sample_table4/` may fail with +403 errors if HNS ACLs are missing for scoped tokens. The original access token may work, but scoped (vended) tokens +require HNS ACLs on the base path or relevant subpath. Polaris does not require HNS to be enabled for basic operation, +but if HNS is enabled and token vending is used, HNS ACLs are mandatory for scoped token access. Always verify HNS ACLs +in addition to IAM roles when troubleshooting GCS access issues with token vending and HNS enabled. ### Upgrade Considerations -The [Polaris Evolution](../evolution) page discusses backward compatibility and -upgrade concerns. +The [Polaris Evolution](../evolution) page discusses backward compatibility and upgrade concerns. From 12e150d7d55521bb628a072836cacc71cd4ee403 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Thu, 20 Nov 2025 08:16:33 +0530 Subject: [PATCH 2/6] Revert "Added GCS related Properties for vending credentials. Added line width to 120 char" This reverts commit 5cd0cc9d226ef739e40aaff594977043c2aaa2e4. --- .../configuring-polaris-for-production.md | 146 +++++++++--------- 1 file changed, 70 insertions(+), 76 deletions(-) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production.md b/site/content/in-dev/unreleased/configuring-polaris-for-production.md index 6b46701afb..2a1efe36bb 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production.md @@ -23,39 +23,35 @@ type: docs weight: 600 --- -The default server configuration is intended for development and testing. When you deploy Polaris in production, review -and apply the following checklist: - +The default server configuration is intended for development and testing. When you deploy Polaris in production, +review and apply the following checklist: - [ ] Configure OAuth2 keys - [ ] Enforce realm header validation (`require-header=true`) - [ ] Use a durable metastore (JDBC + PostgreSQL) - [ ] Bootstrap valid realms in the metastore - [ ] Disable local FILE storage -- [ ] Polaris Server Header -- [ ] Upgrade considerations -- [ ] Cloud Storage Specific Configuration ### Configure OAuth2 -Polaris authentication requires specifying a token broker factory type. Two implementations are supported out of the -box: +Polaris authentication requires specifying a token broker factory type. Two implementations are +supported out of the box: - [rsa-key-pair] uses a pair of public and private keys; - [symmetric-key] uses a shared secret. -[rsa-key-pair]: - https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTRSAKeyPairFactory.java -[symmetric-key]: - https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTSymmetricKeyFactory.java +[rsa-key-pair]: https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTRSAKeyPairFactory.java +[symmetric-key]: https://github.com/apache/polaris/blob/390f1fa57bb1af24a21aa95fdbff49a46e31add7/service/common/src/main/java/org/apache/polaris/service/auth/JWTSymmetricKeyFactory.java By default, Polaris uses `rsa-key-pair`, with randomly generated keys. -{{< alert important >}} The default `rsa-key-pair` configuration is not suitable when deploying many replicas of -Polaris, as each replica will have its own set of keys. This will cause token validation to fail when a request is -routed to a different replica than the one that issued the token. {{< /alert >}} +{{< alert important >}} +The default `rsa-key-pair` configuration is not suitable when deploying many replicas of Polaris, +as each replica will have its own set of keys. This will cause token validation to fail when a +request is routed to a different replica than the one that issued the token. +{{< /alert >}} -It is highly recommended to configure Polaris with previously-generated RSA keys. This can be done by setting the -following properties: +It is highly recommended to configure Polaris with previously-generated RSA keys. This can be done +by setting the following properties: ```properties polaris.authentication.token-broker.type=rsa-key-pair @@ -77,25 +73,28 @@ polaris.authentication.token-broker.type=symmetric-key polaris.authentication.token-broker.symmetric-key.file=/tmp/symmetric.key ``` -Note: it is also possible to set the symmetric key secret directly in the configuration file. If possible, pass the -secret as an environment variable to avoid storing sensitive information in the configuration file: +Note: it is also possible to set the symmetric key secret directly in the configuration file. If +possible, pass the secret as an environment variable to avoid storing sensitive information in the +configuration file: ```properties polaris.authentication.token-broker.symmetric-key.secret=${POLARIS_SYMMETRIC_KEY_SECRET} ``` -Finally, you can also configure the token broker to use a maximum lifespan by setting the following property: +Finally, you can also configure the token broker to use a maximum lifespan by setting the following +property: ```properties polaris.authentication.token-broker.max-token-generation=PT1H ``` -Typically, in Kubernetes, you would define the keys as a `Secret` and mount them as files in the container. +Typically, in Kubernetes, you would define the keys as a `Secret` and mount them as files in the +container. ### Realm Context Resolver -By default, Polaris resolves realms based on incoming request headers. You can configure the realm context resolver by -setting the following properties in `application.properties`: +By default, Polaris resolves realms based on incoming request headers. You can configure the realm +context resolver by setting the following properties in `application.properties`: ```properties polaris.realm-context.realms=POLARIS,MY-REALM @@ -104,38 +103,41 @@ polaris.realm-context.header-name=Polaris-Realm Where: -- `realms` is a comma-separated list of allowed realms. This setting _must_ be correctly configured. At least one realm - must be specified. -- `header-name` is the name of the header used to resolve the realm; by default, it is `Polaris-Realm`. +- `realms` is a comma-separated list of allowed realms. This setting _must_ be correctly configured. + At least one realm must be specified. +- `header-name` is the name of the header used to resolve the realm; by default, it is + `Polaris-Realm`. -If a request contains the specified header, Polaris will use the realm specified in the header. If the realm is not in -the list of allowed realms, Polaris will return a `404 Not Found` response. +If a request contains the specified header, Polaris will use the realm specified in the header. If +the realm is not in the list of allowed realms, Polaris will return a `404 Not Found` response. -If a request _does not_ contain the specified header, however, by default Polaris will use the first realm in the list -as the default realm. In the above example, `POLARIS` is the default realm and would be used if the `Polaris-Realm` -header is not present in the request. +If a request _does not_ contain the specified header, however, by default Polaris will use the first +realm in the list as the default realm. In the above example, `POLARIS` is the default realm and +would be used if the `Polaris-Realm` header is not present in the request. -This is not recommended for production use, as it may lead to security vulnerabilities. To avoid this, set the following -property to `true`: +This is not recommended for production use, as it may lead to security vulnerabilities. To avoid +this, set the following property to `true`: ```properties polaris.realm-context.require-header=true ``` -This will cause Polaris to also return a `404 Not Found` response if the realm header is not present in the request. +This will cause Polaris to also return a `404 Not Found` response if the realm header is not present +in the request. ### Metastore Configuration -A metastore should be configured with an implementation that durably persists Polaris entities. By default, Polaris uses -an in-memory metastore. +A metastore should be configured with an implementation that durably persists Polaris entities. By +default, Polaris uses an in-memory metastore. -{{< alert important >}} The default in-memory metastore is not suitable for production use, as it will lose all data -when the server is restarted; it is also unusable when multiple Polaris replicas are used. {{< /alert >}} +{{< alert important >}} +The default in-memory metastore is not suitable for production use, as it will lose all data +when the server is restarted; it is also unusable when multiple Polaris replicas are used. +{{< /alert >}} -To enable a durable metastore, configure your system to use the Relational JDBC-backed metastore. This implementation -leverages Quarkus for datasource management and supports configuration through environment variables or JVM -D flags at -startup. For more information, refer to the -[Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file). +To enable a durable metastore, configure your system to use the Relational JDBC-backed metastore. +This implementation leverages Quarkus for datasource management and supports configuration through +environment variables or JVM -D flags at startup. For more information, refer to the [Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file). Configure the metastore by setting the following ENV variables: @@ -147,24 +149,28 @@ QUARKUS_DATASOURCE_PASSWORD= QUARKUS_DATASOURCE_JDBC_URL= ``` -The relational JDBC metastore is a Quarkus-managed datasource and only supports Postgres and H2 as of now. Please refer -to the documentation here: [Configure data sources in Quarkus](https://quarkus.io/guides/datasource) -{{< alert important >}} Be sure to secure your metastore backend since it will be storing sensitive data and catalog -metadata. {{< /alert >}} +The relational JDBC metastore is a Quarkus-managed datasource and only supports Postgres and H2 as of now. +Please refer to the documentation here: +[Configure data sources in Quarkus](https://quarkus.io/guides/datasource) + +{{< alert important >}} +Be sure to secure your metastore backend since it will be storing sensitive data and catalog +metadata. +{{< /alert >}} Note: Polaris will always create schema 'polaris_schema' during bootstrap under the configured database. ### Bootstrapping -Before using Polaris, you must **bootstrap** the metastore. This is a manual operation that must be performed **only -once** for each realm in order to prepare the metastore to integrate with Polaris. +Before using Polaris, you must **bootstrap** the metastore. This is a manual operation that must be +performed **only once** for each realm in order to prepare the metastore to integrate with Polaris. -By default, when bootstrapping a new realm, Polaris will create randomised `CLIENT_ID` and `CLIENT_SECRET` for the -`root` principal and store their hashes in the metastore backend. +By default, when bootstrapping a new realm, Polaris will create randomised `CLIENT_ID` and +`CLIENT_SECRET` for the `root` principal and store their hashes in the metastore backend. -Depending on your database, this may not be convenient as the generated credentials are not stored in clear text in the -database. +Depending on your database, this may not be convenient as the generated credentials are not stored +in clear text in the database. In order to provide your own credentials for `root` principal (so you can request tokens via `api/catalog/v1/oauth/tokens`), use the [Polaris Admin Tool]({{% ref "admin-tool" %}}) @@ -190,9 +196,9 @@ Which should return an access token: } ``` -If you used a non-default realm name, add the appropriate request header to the `curl` command, otherwise Polaris will -resolve the realm to the first one in the configuration `polaris.realm-context.realms`. Here is an example to set realm -header: +If you used a non-default realm name, add the appropriate request header to the `curl` command, +otherwise Polaris will resolve the realm to the first one in the configuration +`polaris.realm-context.realms`. Here is an example to set realm header: ```bash curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \ @@ -204,39 +210,27 @@ curl -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \ ``` ### Disable FILE Storage Type - -By default, Polaris allows using the local file system (`FILE`) for catalog storage. This is fine for testing, but **not -recommended for production**. To disable it, set the supported storage types like this: - +By default, Polaris allows using the local file system (`FILE`) for catalog storage. This is fine for testing, +but **not recommended for production**. To disable it, set the supported storage types like this: ```hocon polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES" = [ "S3", "Azure" ] ``` - Leave out `FILE` to prevent its use. Only include the storage types your setup needs. ### Polaris Server Header -Polaris can emit an informational `Server` HTTP response header using Quarkus' built-in header configuration. Add the -following property to one of the supported configuration sources (for example, `application.properties`) to enable it -with the Polaris version string: +Polaris can emit an informational `Server` HTTP response header using Quarkus' built-in header +configuration. Add the following property to one of the supported configuration sources (for example, +`application.properties`) to enable it with the Polaris version string: ```properties quarkus.http.header."Server".value=Polaris/${quarkus.application.version} ``` -If you prefer to scope the header to specific environments, only set the property for the desired profile (for example, -`%prod`). - -### Cloud Storage Specific Configuration - -GCS + Polaris: When using token vending for fine-grained access in Google Cloud Storage (GCS) with Apache Iceberg on -Polaris, ensure that both IAM roles and HNS ACLs (if HNS is enabled) are properly configured. Even with the correct IAM -role (e.g., `roles/storage.objectAdmin`), access to paths such as `gs:///idsp_ns/sample_table4/` may fail with -403 errors if HNS ACLs are missing for scoped tokens. The original access token may work, but scoped (vended) tokens -require HNS ACLs on the base path or relevant subpath. Polaris does not require HNS to be enabled for basic operation, -but if HNS is enabled and token vending is used, HNS ACLs are mandatory for scoped token access. Always verify HNS ACLs -in addition to IAM roles when troubleshooting GCS access issues with token vending and HNS enabled. +If you prefer to scope the header to specific environments, only set the property for the desired +profile (for example, `%prod`). ### Upgrade Considerations -The [Polaris Evolution](../evolution) page discusses backward compatibility and upgrade concerns. +The [Polaris Evolution](../evolution) page discusses backward compatibility and +upgrade concerns. From 2ee7fc4006ba13dde02bf9466d6d4ed54bf90b56 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Thu, 20 Nov 2025 08:53:17 +0530 Subject: [PATCH 3/6] Added GCS related Properties for vending credentials. Added it as sub-page for "Con figuring polaris for production" --- .../_index.md} | 4 ++ .../configuring-cloud-storage-specific.md | 37 +++++++++++++++++++ 2 files changed, 41 insertions(+) rename site/content/in-dev/unreleased/{configuring-polaris-for-production.md => configuring-polaris-for-production/_index.md} (96%) create mode 100644 site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md similarity index 96% rename from site/content/in-dev/unreleased/configuring-polaris-for-production.md rename to site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md index 2a1efe36bb..1ab32c1581 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md @@ -234,3 +234,7 @@ profile (for example, `%prod`). The [Polaris Evolution](../evolution) page discusses backward compatibility and upgrade concerns. + +### Configuring Polaris for Cloud Storages + +For guidance on configuring cloud storage providers (such as Google Cloud Storage, Amazon S3, and Azure Blob Storage) for use with Polaris—including credential vending, IAM roles, ACL requirements, and best practices—see [Configuring Cloud Storage](./configuring-cloud-storage-specific/). diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md new file mode 100644 index 0000000000..f9cc89ce20 --- /dev/null +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md @@ -0,0 +1,37 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Configuring Cloud Storage +linkTitle: Configuring Cloud Storage +type: docs +weight: 600 +--- + +This page provides guidance for configuring cloud storage providers (such as Google Cloud Storage, Amazon S3, and Azure Blob Storage) for use with Polaris. +It covers credential vending, IAM roles, ACL requirements, and best practices to ensure secure and reliable integration. + +#### GCS + +When using credential vending for Google Cloud Storage (GCS) with Apache Iceberg on +Polaris, ensure that both IAM roles and HNS ACLs (if HNS is enabled) are properly configured. Even with the correct IAM +role (e.g., `roles/storage.objectAdmin`), access to paths such as `gs:///idsp_ns/sample_table4/` may fail with +403 errors if HNS ACLs are missing for scoped tokens. The original access token may work, but scoped (vended) tokens +require HNS ACLs on the base path or relevant subpath. Polaris does not require HNS to be enabled for basic operation, +but if HNS is enabled and credential vending is used, HNS ACLs are mandatory for scoped token access. Always verify HNS ACLs +in addition to IAM roles when troubleshooting GCS access issues with credential vending and HNS enabled. \ No newline at end of file From 619188ed2b5880fa2340eadf31f1b88e7bc96fb5 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Thu, 20 Nov 2025 21:57:53 +0530 Subject: [PATCH 4/6] Added GCS related Properties for vending credentials. Added it as sub-page for "Con figuring polaris for production" --- .../unreleased/configuring-polaris-for-production/_index.md | 2 +- ...pecific.md => configuring-gcs-cloud-storage-specific.md} | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) rename site/content/in-dev/unreleased/configuring-polaris-for-production/{configuring-cloud-storage-specific.md => configuring-gcs-cloud-storage-specific.md} (88%) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md index 1ab32c1581..aa4a7db9ac 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md @@ -237,4 +237,4 @@ upgrade concerns. ### Configuring Polaris for Cloud Storages -For guidance on configuring cloud storage providers (such as Google Cloud Storage, Amazon S3, and Azure Blob Storage) for use with Polaris—including credential vending, IAM roles, ACL requirements, and best practices—see [Configuring Cloud Storage](./configuring-cloud-storage-specific/). +For guidance on configuring cloud storage providers ... see child pages (links in the left-hand pane) for use with Polaris—including credential vending, IAM roles, ACL requirements, and best practices—see [Configuring GCS Cloud Storage](./configuring-gcs-cloud-storage-specific/). diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md similarity index 88% rename from site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md rename to site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md index f9cc89ce20..23202ce6d6 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-cloud-storage-specific.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md @@ -17,13 +17,13 @@ # specific language governing permissions and limitations # under the License. # -title: Configuring Cloud Storage -linkTitle: Configuring Cloud Storage +title: Configuring GCS Cloud Storage +linkTitle: Configuring GCS Cloud Storage type: docs weight: 600 --- -This page provides guidance for configuring cloud storage providers (such as Google Cloud Storage, Amazon S3, and Azure Blob Storage) for use with Polaris. +This page provides guidance for Configuring GCS Cloud Storage provider for use with Polaris. It covers credential vending, IAM roles, ACL requirements, and best practices to ensure secure and reliable integration. #### GCS From 368c103339150f5bd24d54d0f72480f543c7f1f1 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Thu, 20 Nov 2025 22:10:11 +0530 Subject: [PATCH 5/6] Added GCS related Properties for vending credentials. Added it as sub-page for "Con figuring polaris for production" --- .../unreleased/configuring-polaris-for-production/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md index aa4a7db9ac..a14e226534 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md @@ -237,4 +237,4 @@ upgrade concerns. ### Configuring Polaris for Cloud Storages -For guidance on configuring cloud storage providers ... see child pages (links in the left-hand pane) for use with Polaris—including credential vending, IAM roles, ACL requirements, and best practices—see [Configuring GCS Cloud Storage](./configuring-gcs-cloud-storage-specific/). +For guidance on configuring specific cloud storage providers see child pages (links in the left-hand pane). From ac9cce91e994bd7dc94900d0f16ff1b5959ad5c0 Mon Sep 17 00:00:00 2001 From: "saksham.ratra" Date: Fri, 21 Nov 2025 08:44:57 +0530 Subject: [PATCH 6/6] 1. Updated the GCS Configuration To make more sense related to HNS 2. Corrected one broken link in "Polaris Configurtion Page" --- .../configuring-polaris-for-production/_index.md | 2 +- .../configuring-gcs-cloud-storage-specific.md | 15 +++++---------- 2 files changed, 6 insertions(+), 11 deletions(-) diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md index a14e226534..0666aea1e7 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/_index.md @@ -173,7 +173,7 @@ Depending on your database, this may not be convenient as the generated credenti in clear text in the database. In order to provide your own credentials for `root` principal (so you can request tokens via -`api/catalog/v1/oauth/tokens`), use the [Polaris Admin Tool]({{% ref "admin-tool" %}}) +`api/catalog/v1/oauth/tokens`), use the [Polaris Admin Tool](../../../../docs/admin-tool.md) You can verify the setup by attempting a token issue for the `root` principal: diff --git a/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md index 23202ce6d6..9417edfcde 100644 --- a/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md +++ b/site/content/in-dev/unreleased/configuring-polaris-for-production/configuring-gcs-cloud-storage-specific.md @@ -23,15 +23,10 @@ type: docs weight: 600 --- -This page provides guidance for Configuring GCS Cloud Storage provider for use with Polaris. -It covers credential vending, IAM roles, ACL requirements, and best practices to ensure secure and reliable integration. +This page provides guidance for configuring GCS Cloud Storage provider for use with Polaris. It covers credential vending, IAM roles, ACL requirements, and best practices to ensure secure and reliable integration. -#### GCS +All catalog operations in Polaris for Google Cloud Storage (GCS)—including listing, reading, and writing objects—are performed using credential vending, which issues scoped (vended) tokens for secure access. -When using credential vending for Google Cloud Storage (GCS) with Apache Iceberg on -Polaris, ensure that both IAM roles and HNS ACLs (if HNS is enabled) are properly configured. Even with the correct IAM -role (e.g., `roles/storage.objectAdmin`), access to paths such as `gs:///idsp_ns/sample_table4/` may fail with -403 errors if HNS ACLs are missing for scoped tokens. The original access token may work, but scoped (vended) tokens -require HNS ACLs on the base path or relevant subpath. Polaris does not require HNS to be enabled for basic operation, -but if HNS is enabled and credential vending is used, HNS ACLs are mandatory for scoped token access. Always verify HNS ACLs -in addition to IAM roles when troubleshooting GCS access issues with credential vending and HNS enabled. \ No newline at end of file +Polaris requires both IAM roles and [Hierarchical Namespace (HNS)](https://docs.cloud.google.com/storage/docs/hns-overview) ACLs (if HNS is enabled) to be properly configured. Even with the correct IAM role (e.g., `roles/storage.objectAdmin`), access to paths such as `gs:///idsp_ns/sample_table4/` may fail with 403 errors if HNS ACLs are missing for scoped tokens. The original access token may work, but scoped (vended) tokens require HNS ACLs on the base path or relevant subpath. + +**Note:** HNS is not mandatory when using GCS for a catalog in Polaris. If HNS is not enabled on the bucket, only IAM roles are required for access. Always verify HNS ACLs in addition to IAM roles when troubleshooting GCS access issues with credential vending and HNS enabled.