docs: add BigQuery Metastore federation guide#4461
Conversation
|
cc: @flyrain, @dimas-b, @jbonofre, @nandorKollar. |
flyrain
left a comment
flyrain
left a comment
There was a problem hiding this comment.
Thank a lot for adding this, @joyhaldar ! LGTM overall. Left some comments.
| (`bigquery.googleapis.com`) over HTTPS. | ||
| - **Authentication:** BigQuery Metastore federation only supports `IMPLICIT` authentication, meaning | ||
| Polaris uses Google Application Default Credentials resolved at process startup. Ensure the | ||
| Polaris process has valid ADC available before starting the server. |
There was a problem hiding this comment.
Might be worth adding a bit more context around ADC here. Not everyone is familiar with Google Application Default Credentials or how they are typically resolved. This would make the operational setup much clearer for first time users.
There was a problem hiding this comment.
I have expanded the ADC bullet with how it's typically resolved and a link to Google's ADC docs.
| "name": "analytics_bigquery", | ||
| "storageConfigInfo": { | ||
| "storageType": "GCS", | ||
| "allowedLocations": ["gs://analytics-bucket/warehouse/"] |
There was a problem hiding this comment.
I think default-base-location is a required field for storageConfigInfo. I'd suggest to replace allowedLocations with default-base-location if there is only one allowed location.
There was a problem hiding this comment.
I have dropped allowedLocations from the example and verified locally.
| polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["BIGQUERY"] | ||
|
|
||
| # Allows IMPLICIT authentication, needed for BigQuery Metastore federation | ||
| polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["IMPLICIT"] |
There was a problem hiding this comment.
Not related to this PR, but I thought IMPLICIT is within the default support list. In that case, we don't have to add the extra config here. This may be worth a dev list discussion.
There was a problem hiding this comment.
I verified both ways, FeatureConfiguration.java has the default for SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES as [OAUTH, BEARER, GCP, SIGV4], and testing locally without the flag, catalog creation fails with Unsupported authentication type: IMPLICIT. Since BigQuery federation only supports IMPLICIT, the flag does need to be set explicitly.
Please let me know if I am incorrect.
There was a problem hiding this comment.
You are right. I am just saying maybe we should think about adding IMPLICIT to the default list. That'd be a separated discussion. We are good here.
| - `gcp.bigquery.impersonate.scopes`: OAuth scopes for the impersonated credential. | ||
| - `gcp.bigquery.impersonate.delegates`: chain of delegate service accounts. | ||
|
|
||
| ## Limitations and operational notes |
There was a problem hiding this comment.
+1 thanks for adding these, great to have them here.
flyrain
left a comment
flyrain
left a comment
There was a problem hiding this comment.
LGTM. Thanks @joyhaldar !
github-project-automation
Bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
|
Would you mind rebasing it, @joyhaldar? There are conflicts. |
Co-authored-by: Joy Haldar <joy.haldar@target.com>
Co-authored-by: Joy Haldar <joy.haldar@target.com>
joyhaldar
force-pushed
the
bqms-federation-docs
branch
from
8a382a0 to
dba1e03
Compare
@flyrain rebased. |
github-project-automation
Bot
moved this from Ready to merge
to Done
in Basic Kanban Board
|
Thanks for adding this, @joyhaldar . Thanks for the review, @dimas-b ! |
3 participants
Follow-up to #4050. Adds a documentation page for BigQuery Metastore federation, modeled on the existing Hive Metastore federation guide. Covers build-time enablement, feature configuration, runtime requirements, catalog creation, and limitations.
Rendered page
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)