Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adding billing_project_id in BigQueryOfflineStoreConfig #3253

Merged
merged 2 commits into from Oct 1, 2022

Conversation

vmallya-123
Copy link
Contributor

Signed-off-by: “Varun varun.mallya@tech.jago.com

What this PR does / why we need it:
One is able to add big query billing project, this can enable to run Bigquery job in a separate project which is not the project which has the BQ feature datasets. Useful in cases when certain projects have reservations (slots) for Bigquery jobs to run.

Signed-off-by: “Varun <varun.mallya@tech.jago.com>
@vmallya-123 vmallya-123 changed the title adding_billing_project_in_config feat: adding billing project parameter in bigquery offline store config Sep 27, 2022
@vmallya-123 vmallya-123 changed the title feat: adding billing project parameter in bigquery offline store config feat: Adding billing project parameter in bigquery offline store config Sep 27, 2022
@adchia
Copy link
Collaborator

adchia commented Sep 29, 2022

/ok-to-test

@adchia
Copy link
Collaborator

adchia commented Sep 29, 2022

were you able to verify this works? also what does the regular project_id do when billing_project_id is set?

@codecov-commenter
Copy link

codecov-commenter commented Sep 29, 2022

Codecov Report

Base: 67.46% // Head: 76.69% // Increases project coverage by +9.23% 🎉

Coverage data is based on head (9bdbab9) compared to base (0ad7fe3).
Patch coverage: 77.27% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3253      +/-   ##
==========================================
+ Coverage   67.46%   76.69%   +9.23%     
==========================================
  Files         179      216      +37     
  Lines       16400    18260    +1860     
==========================================
+ Hits        11064    14005    +2941     
+ Misses       5336     4255    -1081     
Flag Coverage Δ
integrationtests 67.32% <76.19%> (-0.14%) ⬇️
unittests 58.15% <22.72%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdk/python/feast/infra/offline_stores/bigquery.py 85.57% <77.27%> (-0.78%) ⬇️
sdk/python/feast/infra/registry/base_registry.py 92.85% <0.00%> (-0.70%) ⬇️
...on/feast/infra/materialization/snowflake_engine.py 93.18% <0.00%> (-0.41%) ⬇️
...tion/contrib/spark/spark_materialization_engine.py 71.84% <0.00%> (-0.28%) ⬇️
...ests/integration/e2e/test_python_feature_server.py 100.00% <0.00%> (ø)
.../online_stores/contrib/hbase_online_store/hbase.py 37.25% <0.00%> (ø)
...tion/feature_repos/universal/online_store/mysql.py 53.84% <0.00%> (ø)
...tion/feature_repos/universal/online_store/hbase.py 50.00% <0.00%> (ø)
...ib/trino_offline_store/test_config/manual_tests.py 33.33% <0.00%> (ø)
..._stores/contrib/postgres_offline_store/postgres.py 34.14% <0.00%> (ø)
... and 106 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@vmallya-123
Copy link
Contributor Author

@adchia yes I tried it and it works as expected. The project_id is the GCP project which has the feature view datasets, this will be the project_id which will be used by the query template etc (as it is currently being used). billing_project id is the GCP project where the big query job would actually run.

@adchia
Copy link
Collaborator

adchia commented Sep 30, 2022

/lgtm

@adchia
Copy link
Collaborator

adchia commented Sep 30, 2022

mind fixing the lint errors and the title to conform to semantic release convention?

Signed-off-by: Danny Chiao <danny@tecton.ai>
@feast-ci-bot feast-ci-bot removed the lgtm label Sep 30, 2022
@vmallya-123 vmallya-123 changed the title feat: Adding billing project parameter in bigquery offline store config feat: Adding billing_project_id in BigQueryOfflineStoreConfig Oct 1, 2022
@vmallya-123
Copy link
Contributor Author

Thanks for fixing the lint error
Is the title better now? Following the instructions given here

Ensure the title of the PR matches semantic release conventions (e.g. start with feat: or fix: or ci: or chore: or docs:). Keep in mind that any PR with feat: or fix: will directly make it into the change log of a release, so make sure they are understandable!
Please let me know if there is anything else I can do

Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, vmallya-123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit f80f05f into feast-dev:master Oct 1, 2022
kevjumba pushed a commit that referenced this pull request Oct 6, 2022
# [0.26.0](v0.25.0...v0.26.0) (2022-10-06)

### Bug Fixes

* Add `X-Trino-Extra-Credential` header and remove user override ([#3246](#3246)) ([164e666](164e666))
* Add postgres to the feature server Dockerfile to fix helm chart flow ([#3261](#3261)) ([6f6cbb7](6f6cbb7))
* Add stream feature view in the Web UI ([#3257](#3257)) ([1f70b3a](1f70b3a))
* Build dockerfile correctly ([#3239](#3239)) ([a2dc0d0](a2dc0d0))
* Configuration to stop coercion of tz for entity_df ([#3255](#3255)) ([97b7ab9](97b7ab9))
* Enable users to upgrade a batch source into a push source ([#3213](#3213)) ([1b312fb](1b312fb))
* Fix docker image for feature-server ([#3272](#3272)) ([eff01d1](eff01d1))
* Fix Feast UI release process to update the feast-ui package  ([#3267](#3267)) ([a9d48d0](a9d48d0))
* Return 422 on bad push source name ([#3214](#3214)) ([b851e01](b851e01))
* Stream feature view meta undefined created_timestamp issue ([#3266](#3266)) ([12e1a8f](12e1a8f))
* Stream feature view not shown in the UI ([#3251](#3251)) ([e713dda](e713dda))
* Udf in stream feature view UI shows pickled data ([#3268](#3268)) ([0728117](0728117))
* Update snowflake materialization messages ([#3230](#3230)) ([a63d440](a63d440))
* Updated quickstart notebook to patch an incorrect reference to an outdated featureview name ([#3271](#3271)) ([b9b9c54](b9b9c54))
* Use configured user in env var instead of "user" for Trino ([#3254](#3254)) ([532d8a1](532d8a1))

### Features

* Add mysql as online store ([#3190](#3190)) ([cb8db84](cb8db84))
* Add possibility to define feature_store.yaml path with env variable ([#3231](#3231)) ([95fdb19](95fdb19))
* Add request_timeout setting for cassandra online store adapter ([#3256](#3256)) ([da20757](da20757))
* Add tag description in Field in the Feast UI ([#3258](#3258)) ([086f279](086f279))
* Adding billing_project_id in BigQueryOfflineStoreConfig ([#3253](#3253)) ([f80f05f](f80f05f))
* BigTable online store ([#3140](#3140)) ([6bc91c2](6bc91c2)), closes [/github.com/spulec/moto/blob/master/CHANGELOG.md#400](https://github.com//github.com/spulec/moto/blob/master/CHANGELOG.md/issues/400)
* Filter subset features in postgres [#3174](#3174) ([#3203](#3203)) ([b48d36b](b48d36b))
franciscojavierarceo pushed a commit to franciscojavierarceo/feast that referenced this pull request Oct 6, 2022
…dev#3253)

* adding_billing_project_in_config

Signed-off-by: “Varun <varun.mallya@tech.jago.com>

* Fix lint

Signed-off-by: Danny Chiao <danny@tecton.ai>

Signed-off-by: “Varun <varun.mallya@tech.jago.com>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Co-authored-by: Danny Chiao <danny@tecton.ai>
franciscojavierarceo pushed a commit to franciscojavierarceo/feast that referenced this pull request Oct 18, 2022
# [0.26.0](feast-dev/feast@v0.25.0...v0.26.0) (2022-10-06)

### Bug Fixes

* Add `X-Trino-Extra-Credential` header and remove user override ([feast-dev#3246](feast-dev#3246)) ([164e666](feast-dev@164e666))
* Add postgres to the feature server Dockerfile to fix helm chart flow ([feast-dev#3261](feast-dev#3261)) ([6f6cbb7](feast-dev@6f6cbb7))
* Add stream feature view in the Web UI ([feast-dev#3257](feast-dev#3257)) ([1f70b3a](feast-dev@1f70b3a))
* Build dockerfile correctly ([feast-dev#3239](feast-dev#3239)) ([a2dc0d0](feast-dev@a2dc0d0))
* Configuration to stop coercion of tz for entity_df ([feast-dev#3255](feast-dev#3255)) ([97b7ab9](feast-dev@97b7ab9))
* Enable users to upgrade a batch source into a push source ([feast-dev#3213](feast-dev#3213)) ([1b312fb](feast-dev@1b312fb))
* Fix docker image for feature-server ([feast-dev#3272](feast-dev#3272)) ([eff01d1](feast-dev@eff01d1))
* Fix Feast UI release process to update the feast-ui package  ([feast-dev#3267](feast-dev#3267)) ([a9d48d0](feast-dev@a9d48d0))
* Return 422 on bad push source name ([feast-dev#3214](feast-dev#3214)) ([b851e01](feast-dev@b851e01))
* Stream feature view meta undefined created_timestamp issue ([feast-dev#3266](feast-dev#3266)) ([12e1a8f](feast-dev@12e1a8f))
* Stream feature view not shown in the UI ([feast-dev#3251](feast-dev#3251)) ([e713dda](feast-dev@e713dda))
* Udf in stream feature view UI shows pickled data ([feast-dev#3268](feast-dev#3268)) ([0728117](feast-dev@0728117))
* Update snowflake materialization messages ([feast-dev#3230](feast-dev#3230)) ([a63d440](feast-dev@a63d440))
* Updated quickstart notebook to patch an incorrect reference to an outdated featureview name ([feast-dev#3271](feast-dev#3271)) ([b9b9c54](feast-dev@b9b9c54))
* Use configured user in env var instead of "user" for Trino ([feast-dev#3254](feast-dev#3254)) ([532d8a1](feast-dev@532d8a1))

### Features

* Add mysql as online store ([feast-dev#3190](feast-dev#3190)) ([cb8db84](feast-dev@cb8db84))
* Add possibility to define feature_store.yaml path with env variable ([feast-dev#3231](feast-dev#3231)) ([95fdb19](feast-dev@95fdb19))
* Add request_timeout setting for cassandra online store adapter ([feast-dev#3256](feast-dev#3256)) ([da20757](feast-dev@da20757))
* Add tag description in Field in the Feast UI ([feast-dev#3258](feast-dev#3258)) ([086f279](feast-dev@086f279))
* Adding billing_project_id in BigQueryOfflineStoreConfig ([feast-dev#3253](feast-dev#3253)) ([f80f05f](feast-dev@f80f05f))
* BigTable online store ([feast-dev#3140](feast-dev#3140)) ([6bc91c2](feast-dev@6bc91c2)), closes [/github.com/spulec/moto/blob/master/CHANGELOG.md#400](https://github.com//github.com/spulec/moto/blob/master/CHANGELOG.md/issues/400)
* Filter subset features in postgres [feast-dev#3174](feast-dev#3174) ([feast-dev#3203](feast-dev#3203)) ([b48d36b](feast-dev@b48d36b))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants