-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Congrats on the open source, but where is code and issues? #1
Comments
|
collado-mike
added a commit
that referenced
this issue
Jul 30, 2024
* Initial commit Co-authored-by: Aihua Xu <aihua.xu@snowflake.com> Co-authored-by: Alvin Chen <alvin.chen@snowflake.com> Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evan Gilbert <evan.gilbert@snowflake.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Jonas-Taha El Sesiy <github@elsesiy.com> Co-authored-by: Maninder Parmar <maninder.parmar@snowflake.com> Co-authored-by: Michael Collado <collado.mike@gmail.com> Co-authored-by: Sean Lee <sean.lee@snowflake.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> Co-authored-by: Tyler Jones <tyler.jones@snowflake.com> Co-authored-by: Vivo Xu <git@vivoxu.com> * Add brief description to README * Update package naming to separate snowflake code and oss core, extensions, and service impl * Set up new gradle project structure * Add gradle wrapper * Moved docker files to project root and renamed files and classes to Polaris * Update CI scripts to use new layout * Add missing gradlew file * Updates to READMEs and move manual* scripts to snowflake repository root * Fix SparkIntegrationTest after merge * Fix regtest in polaris application Fix json error messages to return clearer validation causes (#272) Extended gradle format in root project to apply to oss (#273) Improve error message for invalid json and distinguish from invalid values (#274) Update repository references to managed-polaris Removed references and made aws resources configurable Fix references to snowflake reg test resources Update README with instructions on running cloud-specific regtests Copy recommended gradle .gitignore contents Update github actions Add @polaris-catalog/polaris team to codeowners * Merge branch 'managed-polaris' into mcollado-polaris-import Co-authored-by: Dennis Huo <huoisme@gmail.com> * Merged changes into polaris-catalog/polaris-dev Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> * Squashed commit of the following: Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Eric Maynard <xeyerericx@gmail.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Michael Collado <collado.mike@gmail.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> commit bd256f544c069ff15a7a96ab7f2abc650a2e9812 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Tue Jul 23 23:43:38 2024 +0000 Remove s3gov s3china enums and validate roleArn. Removing the enums because the iceberg spec does not have s3gov or s3china prefix for the url, those are snowflake style supported prefix. commit 855dbb702bdc4fc80ca852b8bf563979e08d63d2 Author: Michael Collado <michael.collado@snowflake.com> Date: Tue Jul 23 10:02:35 2024 -0700 Fix credential vending for view creation (#19) Correctly sets vended credentials for view creation commit 0429e6356cd71b3908600b6c5c17f82493f1d37d Author: Eric Maynard <eric.maynard@snowflake.com> Date: Tue Jul 23 09:49:20 2024 -0700 This PR implements a basic CLI for Polaris, supporting simple commands like: ``` polaris catalogs list polaris catalogs create --type --storage-type s3 --default-base-location s3://my-bucket --role-arn ${ARN} polaris principals update emaynard --property foo=bar --property e=mc2 polaris privileges --catalog my_cat --catalog-role my_role namespace grant --namespace a.b.c TABLE_READ_DATA polaris privileges --catalog my_cat --catalog-role my_role table revoke --namespace a.b.c --table t1 TABLE_READ_DATA ``` commit 01d4c294e6f8b3e77bf205af00ea2e1dbef0d362 Author: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Date: Mon Jul 22 11:12:29 2024 -0700 Service Bootstrap (Part 2): we are removing bootstrap code in init methods and updates to In-Memory store (#8) Changing bootstrap logic, moving bootstrap code to a separate method and only use it during service bootstrapping and first time initialization. So moving forward we will not call bootstrap during SessionManager init code as it used to be, as this will be destructive if service gets restarted. For InMemory Store we have special handling and doing bootstrap on a very first initialization of SessionManager for a given realm. And it makes sense as we can't use our custom dropwizard Bootstrap command for bootstrapping in-memory store (as in-memory store is only valid and available during server process lifetime) commit 2c7f3c43c557e521d7177a4d7dd44157147f0a0c Author: Dennis Huo <dennis.huo@snowflake.com> Date: Fri Jul 19 23:33:05 2024 +0000 Defense-in-depth - make FileIO impl come from StorageConfigurationInfo (#15) Description Rather than specifying ResolvingFileIO, we can be more explicit about the FileIO impl we're allowing. Also only allow custom specification of FileIO in test environments using a feature config flag. Even if there are valid FileIO customizations customers could specify, we have only really vetted the enumerated list of impls, so for example we don't want a customer to be able to force Polaris to try to use Hadoop's S3AFileSystem for S3 files even if it "might" work. This in conjunction with omitting `FILE` from SUPPORTED_CATALOG_STORAGE_TYPES for managed environments (https://github.com/snowflakedb/polaris-k8-config/pull/116/files) ensures we won't have a FileIO impl that's capable of reading unexpected files. commit 498861114994b0508efdbdd2167918be5517f4cb Merge: cf07ac0 c100175 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 13:41:02 2024 -0700 Merge branch 'main' into mcollado-update-aws-region commit cf07ac099644b96f93026b209c9938243c1cce18 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 13:38:22 2024 -0700 Stop setting AWS_REGION env and use client.region spark config in tests commit c10017521145e138ae5cdd903d7d51b4bee9e82c Merge: b1de84a d2df00f Author: Eric Maynard <eric.maynard@snowflake.com> Date: Fri Jul 19 12:43:15 2024 -0700 Merge pull request #12 from snowflakedb/confirm-warehouse-non-null commit b1de84ad47f6bdf5be4318d4664767dfc33bb5a0 Merge: 504dcc0 1f79e19 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 09:25:07 2024 -0700 Merge branch 'main' into mcollado-view-tests commit d4c58a6a19756078309229c1de4dbf5f737dbdd0 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Thu Jul 18 02:58:52 2024 -0700 cross region support commit 504dcc05bb33e686f5765e5b2d91aa4dcfe2e5d1 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 00:00:57 2024 -0700 fix regtest failures commit b7ed5d27e2d71708977cc6fe7eac3ab10e8d9836 Author: Michael Collado <michael.collado@snowflake.com> Date: Thu Jul 18 21:52:46 2024 -0700 Add reg tests to verify view support * Squashed commit of the following: commit 4fb3b6c19a8a8a4961b777ad32dbe1b87d5efe94 Author: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Date: Thu Jul 25 14:02:30 2024 -0700 Adding annotation and enforcing size limits for Principal, Role, Catalog and Catalog Role names. Also blocking "SYSTEM$" prefix from being used in names. Adding case-insensitive regex rule to block "SYSTEM$" commit 2fcc2898ea038c074fed075cdc7ff62e4884e76a Author: Alvin Chen <alvin.chen@snowflake.com> Date: Thu Jul 25 11:28:00 2024 -0700 Replace Dropwizard Metrics with Micrometer (#18) <!-- Please describe your change here and remove this comment --> Since the current Dropwizard Metric library 4.x doesn't support adding custom labels to metrics, we cannot define per-account metrics in order As a result, we're migrating to Micrometer metrics to support custom tagging and align with the metric implementations Major changes by component - `PolarisMetricRegistry` - defines caching for timers and error counters as well as abstracts away the creation of two separate metrics, one with and one without the `account` tag - `TimedApplicationEventListener` - an implementation of the Jersey ApplicationEventListener to listen on requests invoking methods with `@TimedApi` annotation, and handles logic of timing resource/counting errors on success/failure cases respectively - `IcebergMappedException` - removed the original logic for counting errors since the code is now centralized in the above two classes ## Test Manual tested by calling the /metrics endpoint. Following is the result of one successful and one failure invoke of the /oauth endpoint. Note that the timer produces a `summary` and a `gauge`, and doesn't get incremented on failure cases. ``` % curl http://localhost:8182/metrics # HELP polaris_OAuth2Api_getToken_error_total # TYPE polaris_OAuth2Api_getToken_error_total counter polaris_OAuth2Api_getToken_error_total{HTTP_RESPONSE_CODE="401"} 1.0 # HELP polaris_OAuth2Api_getToken_error_realm_total # TYPE polaris_OAuth2Api_getToken_error_realm_total counter polaris_OAuth2Api_getToken_error_realm_total{HTTP_RESPONSE_CODE="401",REALM_ID="testpolaris"} 1.0 # HELP polaris_OAuth2Api_getToken_realm_seconds # TYPE polaris_OAuth2Api_getToken_realm_seconds summary polaris_OAuth2Api_getToken_realm_seconds_count{REALM_ID="testpolaris"} 1 polaris_OAuth2Api_getToken_realm_seconds_sum{REALM_ID="testpolaris"} 0.384 # HELP polaris_OAuth2Api_getToken_realm_seconds_max # TYPE polaris_OAuth2Api_getToken_realm_seconds_max gauge polaris_OAuth2Api_getToken_realm_seconds_max{REALM_ID="testpolaris"} 0.384 # HELP polaris_OAuth2Api_getToken_seconds # TYPE polaris_OAuth2Api_getToken_seconds summary polaris_OAuth2Api_getToken_seconds_count 1 polaris_OAuth2Api_getToken_seconds_sum 0.384 # HELP polaris_OAuth2Api_getToken_seconds_max # TYPE polaris_OAuth2Api_getToken_seconds_max gauge polaris_OAuth2Api_getToken_seconds_max 0.384 # HELP polaris_persistence_loadEntity_realm_seconds # TYPE polaris_persistence_loadEntity_realm_seconds summary polaris_persistence_loadEntity_realm_seconds_count{REALM_ID="testpolaris"} 1 polaris_persistence_loadEntity_realm_seconds_sum{REALM_ID="testpolaris"} 0.041 # HELP polaris_persistence_loadEntity_realm_seconds_max # TYPE polaris_persistence_loadEntity_realm_seconds_max gauge polaris_persistence_loadEntity_realm_seconds_max{REALM_ID="testpolaris"} 0.041 # HELP polaris_persistence_loadEntity_seconds # TYPE polaris_persistence_loadEntity_seconds summary polaris_persistence_loadEntity_seconds_count 1 polaris_persistence_loadEntity_seconds_sum 0.041 # HELP polaris_persistence_loadEntity_seconds_max # TYPE polaris_persistence_loadEntity_seconds_max gauge polaris_persistence_loadEntity_seconds_max 0.041 ``` commit 5abee21b07be00f5f3b18faabe61fb88ecec37e0 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Thu Jul 25 17:14:09 2024 +0000 select view hangs in remote polaris because iceberg SDK could not initialize the s3client since it is missing credentials. It works locally because the SDK S3client initialization work if your local environment have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set, and our dev environment does have these two variables set, so it was not using vending scopedcreds. This PR does below things: 1. add scoped creds to the fileIO when select view 2. stops retry for more cases, the `select view` hangs because it keeps retying commit 85d41bcbab30c9fc3fad56dea83f80e8146ee79c Author: Eric Maynard <eric.maynard@snowflake.com> Date: Wed Jul 24 16:33:57 2024 -0700 In this PR, I've regenerated the Python clients from the spec by following the steps [here](https://github.com/snowflakedb/managed-polaris/tree/main/polaris/regtests#python-tests). I ran: ``` docker run --rm \ -v ${PWD}:/local openapitools/openapi-generator-cli generate \ -i /local/spec/polaris-management-service.yml \ -g python \ -o /local/regtests/client/python --additional-properties=packageName=polaris.management --additional-properties=apiNamePrefix=polaris docker run --rm \ -v ${PWD}:/local openapitools/openapi-generator-cli generate \ -i /local/spec/rest-catalog-open-api.yaml \ -g python \ -o /local/regtests/client/python --additional-properties=packageName=polaris.catalog --additional-properties=apiNameSuffix="" --additional-properties=apiNamePrefix=Iceberg ``` commit 485d99c89abd7b7c3690f45d96a5043a47032ba3 Author: Eric Maynard <eric.maynard@snowflake.com> Date: Wed Jul 24 11:27:21 2024 -0700 This PR introduces quickstart documentation and adds a basic structure for OSS docs. commit 4310980aecf81cc23bbf583cfb6c360ca738a788 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Wed Jul 24 17:38:14 2024 +0000 Stop retry 403 Access Denied error (#22) commit 95acd5b3e7983b89d47a915c62ac5bb247730313 Author: Benoit Dageville <59930187+sfc-gh-bdagevil@users.noreply.github.com> Date: Tue Jul 23 22:15:34 2024 -0700 * Fix readme statement and snowflake reference in PolarisDefaultDiagServiceImpl --------- Co-authored-by: Daniel Myers <jdanielmyers@gmail.com> Co-authored-by: Anna Filippova <7892219+annafil@users.noreply.github.com> Co-authored-by: Michael Collado <michael.collado@snowflake.com> Co-authored-by: Aihua Xu <aihua.xu@snowflake.com> Co-authored-by: Alvin Chen <alvin.chen@snowflake.com> Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evan Gilbert <evan.gilbert@snowflake.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Jonas-Taha El Sesiy <github@elsesiy.com> Co-authored-by: Maninder Parmar <maninder.parmar@snowflake.com> Co-authored-by: Sean Lee <sean.lee@snowflake.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> Co-authored-by: Tyler Jones <tyler.jones@snowflake.com> Co-authored-by: Vivo Xu <git@vivoxu.com>
Merged
eric-maynard
referenced
this issue
in eric-maynard/polaris
Jul 30, 2024
Adds a Dockerfile to build and run the service
eric-maynard
referenced
this issue
in eric-maynard/polaris
Jul 30, 2024
…le/actions-3.5.0 SNOW-00000: Bump gradle/actions from 3.4.2 to 3.5.0
eric-maynard
referenced
this issue
in eric-maynard/polaris
Jul 30, 2024
85e69a3 Add ascii art banner after initialization. (apache#45) 398a264 Add issue + PR templates (apache#27) 2607c97 Retain original copyright in `gradlew` (apache#43) bb8dcf5 Simplify spotless-java expression (apache#24) 6f5bda0 Gradle wrapper - download and verify (apache#32) 7640ec8 No copyright in PR template (apache#33) b939de0 Upgrade to Gradle 8.9 (apache#29) 32b9520 Upgrade to Jackson 2.17.2 (apache#30) 875b8fc Set project name in IDE (apache#23) fa430e8 Prominently warn for Java version < 21 (apache#22) a9c1807 Add Apache license header (apache#19) bfc111d Add some more exclusions to `.gitignore` (apache#20) d9abad8 Suppress unchecked cast warning in DefaultConfigurationStore (apache#18) 7cae6f4 Add base skeleton with updated LICENSE and NOTICE file (apache#8) 4988075 Fix typo in the quickstart docs. (apache#17) 1c4455f Bump gradle/actions from 3.4.2 to 3.5.0 (apache#14) 76fd0f4 Merge Polaris catalog code (#1) REVERT: e8e08d6 Merge remote-tracking branch 'managed-polaris-local/polaris-oss-export' into mcollado-polaris-merge REVERT: 842b08d Made regtest resources configurable REVERT: cad1390 Merge branch 'managed-polaris' into mcollado-polaris-import REVERT: 136f67b SNOW-1543690 Make credentialRotationRequired optional when creating a principal (apache#5) REVERT: 1552b35 Reorganized repository and split Snowflake and OSS Polaris projects with distinct builds (apache#268) REVERT: 3063229 Add brief description to README REVERT: bcc8f65 Initial commit REVERT: 6fa40a9 Merge branch 'main' of github.com:snowflakedb/pinnacle into dhuo-merge-pinnacle-main REVERT: 747cf3e SNOW-1528940 Fix usability issues related to the pentest of invalid role ARN (apache#279) REVERT: 0325090 Merge remote-tracking branch 'pinnacle/main' into mcollado-merge-pinnacle REVERT: 01ffd22 Remove the "required" annotation for remoteUrl in ExternalCatalog. (apache#280) REVERT: 6a3de99 Merge remote-tracking branch 'pinnacle/main' REVERT: 1bbd9b1 Minor fixups to preserve semantics of some of the helper scripts after the restructure. (apache#278) REVERT: 2186b72 Adding bootstrap action for self hosted Polaris (apache#277) REVERT: 8f9639d Update model validation to use jakarta.validation annotations (apache#276) REVERT: a80ba68 Update repository references to managed-polaris REVERT: 6736d60 Improve error message for invalid json and distinguish from invalid values (apache#274) REVERT: 92bb3a4 Extended gradle format in root project to apply to oss (apache#273) REVERT: a3f049c Fix json error messages to return clearer validation causes (apache#272) REVERT: a4aecf4 Reorganized repository and split Snowflake and OSS Polaris projects with distinct builds (apache#268) git-subtree-dir: polaris git-subtree-split: 85e69a3007fb89c0aeff7b9aec348538abad3dd1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hope to see the code ASAP.
The text was updated successfully, but these errors were encountered: