feat: Add Snowflake online store (again) #2922

sfc-gh-madkins · 2022-07-08T01:48:34Z

What this PR does / why we need it: Adds snowflake to online store options

Which issue(s) this PR fixes:

Fixes #

codecov-commenter · 2022-07-08T01:55:42Z

Codecov Report

Merging #2922 (d7d0c04) into master (5edf4b0) will decrease coverage by 5.56%.
The diff coverage is 85.36%.

@@            Coverage Diff             @@
##           master    #2922      +/-   ##
==========================================
- Coverage   77.69%   72.13%   -5.57%     
==========================================
  Files         186      211      +25     
  Lines       16387    18541    +2154     
==========================================
+ Hits        12732    13374     +642     
- Misses       3655     5167    +1512

Flag	Coverage Δ
integrationtests	`68.03% <84.51%> (-0.75%)`	⬇️
unittests	`58.21% <31.70%> (-0.26%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/utils/snowflake_utils.py	`77.77% <60.00%> (-7.05%)`	⬇️
sdk/python/feast/infra/online_stores/snowflake.py	`98.13% <98.13%> (ø)`
...ts/integration/feature_repos/repo_configuration.py	`86.51% <100.00%> (+0.15%)`	⬆️
sdk/python/tests/utils/online_read_write_test.py	`19.35% <0.00%> (-80.65%)`	⬇️
...ion/registration/test_stream_feature_view_apply.py	`28.84% <0.00%> (-69.24%)`	⬇️
...gration/registration/test_feature_service_apply.py	`31.25% <0.00%> (-68.75%)`	⬇️
sdk/python/tests/utils/logged_features.py	`32.65% <0.00%> (-67.35%)`	⬇️
...sts/integration/offline_store/test_push_offline.py	`38.46% <0.00%> (-61.54%)`	⬇️
...tests/integration/online_store/test_push_online.py	`40.00% <0.00%> (-60.00%)`	⬇️
...ests/integration/scaffolding/test_partial_apply.py	`50.00% <0.00%> (-50.00%)`	⬇️
... and 45 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5edf4b0...d7d0c04. Read the comment docs.

kevjumba

@sfc-gh-madkins Could you add documentation into our docs for the new online store including usage etc?

sdk/python/feast/infra/utils/snowflake_utils.py

kevjumba · 2022-07-22T16:57:10Z

@sfc-gh-madkins if you rebase, the errors should go way

kevjumba

some nits: but please add some documentation and also follow these docs to regenerate the requirements/add to the index.rst file to document code changes. @sfc-gh-madkins
#2946

sdk/python/feast/infra/online_stores/snowflake.py

sdk/python/feast/infra/utils/snowflake_utils.py

kevjumba · 2022-07-26T16:44:21Z

@sfc-gh-madkins please add some documentation on the store refer to https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/adding-a-new-offline-store.md for guidelines

sdk/python/feast/infra/online_stores/snowflake.py

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

sfc-gh-madkins · 2022-07-28T23:22:38Z

@adchia looks like the snowflake warehouse credit limit needs to be increased

achals

/lgtm

feast-ci-bot · 2022-07-29T21:52:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, sfc-gh-madkins

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.23.0](v0.22.0...v0.23.0) (2022-08-02) ### Bug Fixes * Add dummy alias to pull_all_from_table_or_query ([#2956](#2956)) ([5e45228](5e45228)) * Bump version of Guava to mitigate cve ([#2896](#2896)) ([51df8be](51df8be)) * Change numpy version on setup.py and upgrade it to resolve dependabot warning ([#2887](#2887)) ([80ea7a9](80ea7a9)) * Change the feature store plan method to public modifier ([#2904](#2904)) ([0ec7d1a](0ec7d1a)) * Deprecate 3.7 wheels and fix verification workflow ([#2934](#2934)) ([040c910](040c910)) * Do not allow same column to be reused in data sources ([#2965](#2965)) ([661c053](661c053)) * Fix build wheels workflow to install apache-arrow correctly ([#2932](#2932)) ([bdeb4ae](bdeb4ae)) * Fix file offline store logic for feature views without ttl ([#2971](#2971)) ([26f6b69](26f6b69)) * Fix grpc and update protobuf ([#2894](#2894)) ([86e9efd](86e9efd)) * Fix night ci syntax error and update readme ([#2935](#2935)) ([b917540](b917540)) * Fix nightly ci again ([#2939](#2939)) ([1603c9e](1603c9e)) * Fix the go build and use CgoArrowAllocator to prevent incorrect garbage collection ([#2919](#2919)) ([130746e](130746e)) * Fix typo in CONTRIBUTING.md ([#2955](#2955)) ([8534f69](8534f69)) * Fixing broken links to feast documentation on java readme and contribution ([#2892](#2892)) ([d044588](d044588)) * Fixing Spark min / max entity df event timestamps range return order ([#2735](#2735)) ([ac55ce2](ac55ce2)) * Move gcp back to 1.47.0 since grpcio-tools 1.48.0 got yanked from pypi ([#2990](#2990)) ([fc447eb](fc447eb)) * Refactor testing and sort out unit and integration tests ([#2975](#2975)) ([2680f7b](2680f7b)) * Remove hard-coded integration test setup for AWS & GCP ([#2970](#2970)) ([e4507ac](e4507ac)) * Resolve small typo in README file ([#2930](#2930)) ([16ae902](16ae902)) * Revert "feat: Add snowflake online store ([#2902](#2902))" ([#2909](#2909)) ([38fd001](38fd001)) * Snowflake_online_read fix ([#2988](#2988)) ([651ce34](651ce34)) * Spark source support table with pattern "db.table" ([#2606](#2606)) ([3ce5139](3ce5139)), closes [#2605](#2605) * Switch mysql log string to use regex ([#2976](#2976)) ([5edf4b0](5edf4b0)) * Update gopy to point to fork to resolve github annotation errors. ([#2940](#2940)) ([ba2dcf1](ba2dcf1)) * Version entity serialization mechanism and fix issue with int64 vals ([#2944](#2944)) ([d0d27a3](d0d27a3)) ### Features * Add an experimental lambda-based materialization engine ([#2923](#2923)) ([6f79069](6f79069)) * Add column reordering to `write_to_offline_store` ([#2876](#2876)) ([8abc2ef](8abc2ef)) * Add custom JSON table tab w/ formatting ([#2851](#2851)) ([0159f38](0159f38)) * Add CustomSourceOptions to SavedDatasetStorage ([#2958](#2958)) ([23c09c8](23c09c8)) * Add Go option to `feast serve` command ([#2966](#2966)) ([a36a695](a36a695)) * Add interfaces for batch materialization engine ([#2901](#2901)) ([38b28ca](38b28ca)) * Add pages for individual Features to the Feast UI ([#2850](#2850)) ([9b97fca](9b97fca)) * Add snowflake online store ([#2902](#2902)) ([f758f9e](f758f9e)), closes [#2903](#2903) * Add Snowflake online store (again) ([#2922](#2922)) ([2ef71fc](2ef71fc)), closes [#2903](#2903) * Add to_remote_storage method to RetrievalJob ([#2916](#2916)) ([109ee9c](109ee9c)) * Support retrieval from multiple feature views with different join keys ([#2835](#2835)) ([056cfa1](056cfa1))

feast-ci-bot added the size/L label Jul 8, 2022

sfc-gh-madkins added the ok-to-test label Jul 8, 2022

kevjumba reviewed Jul 11, 2022

View reviewed changes

sdk/python/feast/infra/utils/snowflake_utils.py Show resolved Hide resolved

sfc-gh-madkins added the do-not-merge/work-in-progress label Jul 16, 2022

kevjumba added ok-to-test and removed ok-to-test labels Jul 22, 2022

kevjumba reviewed Jul 22, 2022

View reviewed changes

feast-ci-bot added size/XL and removed size/L labels Jul 27, 2022

sfc-gh-madkins force-pushed the snowflake_online branch from b11d002 to 9e2af86 Compare July 28, 2022 04:01

sfc-gh-madkins removed the do-not-merge/work-in-progress label Jul 28, 2022

achals reviewed Jul 28, 2022

View reviewed changes

sfc-gh-madkins and others added 15 commits July 28, 2022 16:57

feat: Add snowflake online store

14fd591

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

lint/format

40507ef

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

removing missing testing env variables

6c76b5f

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

test offline store first

6933e34

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

snowflake online test fixes

98e3a5c

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

format

41ec30d

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

fix snowflake testing (feast-dev#2903)

49286d0

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

change to transient table

fb534ec

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

remove extra lines

1ddb4ee

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

docs + nits

56feaea

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

docs + nits2

c38fd29

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

final docs + nits

2309d4e

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

Update sdk/python/feast/infra/online_stores/snowflake.py

7a03683

Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

Update sdk/python/feast/infra/online_stores/snowflake.py

e1bda4f

Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

Update sdk/python/feast/infra/online_stores/snowflake.py

cce1402

Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

sfc-gh-madkins and others added 2 commits July 28, 2022 16:57

Update sdk/python/feast/infra/online_stores/snowflake.py

48ae284

Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

fix serialization

53aa4f7

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

sfc-gh-madkins force-pushed the snowflake_online branch from 0191302 to 53aa4f7 Compare July 28, 2022 20:57

fix serialization2

d7d0c04

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

achals approved these changes Jul 29, 2022

View reviewed changes

feast-ci-bot assigned achals Jul 29, 2022

feast-ci-bot added the lgtm label Jul 29, 2022

feast-ci-bot added the approved label Jul 29, 2022

feast-ci-bot merged commit 2ef71fc into feast-dev:master Jul 29, 2022

sfc-gh-madkins deleted the snowflake_online branch July 29, 2022 23:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Snowflake online store (again) #2922

feat: Add Snowflake online store (again) #2922

sfc-gh-madkins commented Jul 8, 2022

codecov-commenter commented Jul 8, 2022 •

edited

kevjumba left a comment

kevjumba commented Jul 22, 2022

kevjumba left a comment

kevjumba commented Jul 26, 2022

sfc-gh-madkins commented Jul 28, 2022

achals left a comment

feast-ci-bot commented Jul 29, 2022

feat: Add Snowflake online store (again) #2922

feat: Add Snowflake online store (again) #2922

Conversation

sfc-gh-madkins commented Jul 8, 2022

codecov-commenter commented Jul 8, 2022 • edited

Codecov Report

kevjumba left a comment

Choose a reason for hiding this comment

kevjumba commented Jul 22, 2022

kevjumba left a comment

Choose a reason for hiding this comment

kevjumba commented Jul 26, 2022

sfc-gh-madkins commented Jul 28, 2022

achals left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Jul 29, 2022

codecov-commenter commented Jul 8, 2022 •

edited