feat: Add Snowflake online store (again) (#2922)

* feat: Add snowflake online store Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * lint/format Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * removing missing testing env variables Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * test offline store first Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * snowflake online test fixes Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * format Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * fix snowflake testing (#2903) Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * change to transient table Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * remove extra lines Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * docs + nits Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * docs + nits2 Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * final docs + nits Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * Update sdk/python/feast/infra/online_stores/snowflake.py Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * Update sdk/python/feast/infra/online_stores/snowflake.py Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * Update sdk/python/feast/infra/online_stores/snowflake.py Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * Update sdk/python/feast/infra/online_stores/snowflake.py Co-authored-by: Achal Shah <achals@gmail.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * fix serialization Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * fix serialization2 Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> Co-authored-by: Achal Shah <achals@gmail.com>
feast-dev · Jul 29, 2022 · 2ef71fc · 2ef71fc
1 parent 61a194c
commit 2ef71fc
Show file tree

Hide file tree

Showing 15 changed files with 542 additions and 37 deletions.
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -79,6 +79,7 @@
   * [PostgreSQL (contrib)](reference/offline-stores/postgres.md)
 * [Online stores](reference/online-stores/README.md)
   * [SQLite](reference/online-stores/sqlite.md)
+  * [Snowflake](reference/online-stores/snowflake.md)
   * [Redis](reference/online-stores/redis.md)
   * [Datastore](reference/online-stores/datastore.md)
   * [DynamoDB](reference/online-stores/dynamodb.md)

diff --git a/docs/reference/offline-stores/snowflake.md b/docs/reference/offline-stores/snowflake.md
@@ -2,7 +2,7 @@
 
 ## Description
 
-The Snowflake offline store provides support for reading [SnowflakeSources](../data-sources/snowflake.md).
+The [Snowflake](https://trial.snowflake.com) offline store provides support for reading [SnowflakeSources](../data-sources/snowflake.md).
 
 * Snowflake tables and views are allowed as sources.
 * All joins happen within Snowflake.
@@ -11,7 +11,7 @@ The Snowflake offline store provides support for reading [SnowflakeSources](../d
   * This allows you to call
      * `to_snowflake` to save the dataset into Snowflake
      * `to_sql` to get the SQL query that would execute on `to_df`
-     * `to_arrow_chunks` to get the result in batches ([Snowflake python connector docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#get_result_batches)) 
+     * `to_arrow_chunks` to get the result in batches ([Snowflake python connector docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#get_result_batches))
 
 ## Example
 

diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md
@@ -4,6 +4,8 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
 
 {% page-ref page="sqlite.md" %}
 
+{% page-ref page="snowflake.md" %}
+
 {% page-ref page="redis.md" %}
 
 {% page-ref page="datastore.md" %}

diff --git a/docs/reference/online-stores/snowflake.md b/docs/reference/online-stores/snowflake.md
@@ -0,0 +1,35 @@
+# Snowflake
+
+## Description
+
+The [Snowflake](https://trial.snowflake.com) online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.
+
+* Only the latest feature values are persisted
+
+The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):
+* "entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination
+* "entity_key" (BINARY) -- repeated key currently unused for reading entity_combination
+* "feature_name" (VARCHAR)
+* "value" (BINARY)
+* "event_ts" (TIMESTAMP)
+* "created_ts" (TIMESTAMP)
+
+ (This model may be subject to change when Snowflake Hybrid Tables are released)
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: snowflake.online
+    account: SNOWFLAKE_DEPLOYMENT_URL
+    user: SNOWFLAKE_USER
+    password: SNOWFLAKE_PASSWORD
+    role: SNOWFLAKE_ROLE
+    warehouse: SNOWFLAKE_WAREHOUSE
+    database: SNOWFLAKE_DATABASE
+```
+{% endcode %}
diff --git a/docs/roadmap.md b/docs/roadmap.md
@@ -27,6 +27,7 @@ The list below contains the functionality that contributors are planning to deve
   * [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
   * [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
 * **Online Stores**
+  * [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake)
   * [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
   * [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)
   * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
@@ -59,4 +60,4 @@ The list below contains the functionality that contributors are planning to deve
   * [x] Model-centric feature tracking (feature services)
   * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
   * [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/))
-  * [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
+  * [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
diff --git a/docs/tutorials/driver-stats-on-snowflake.md b/docs/tutorials/driver-stats-on-snowflake.md
@@ -1,6 +1,6 @@
 ---
 description: >-
-  Initial demonstration of Snowflake as an offline store with Feast, using the Snowflake demo template.
+  Initial demonstration of Snowflake as an offline+online store with Feast, using the Snowflake demo template.
 ---
 
 # Drivers stats on Snowflake
@@ -61,6 +61,14 @@ offline_store:
     role: ROLE_NAME #case sensitive
     warehouse: WAREHOUSE_NAME #case sensitive
     database: DATABASE_NAME #case cap sensitive
+online_store:
+    type: snowflake.online
+    account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
+    user: USERNAME
+    password: PASSWORD
+    role: ROLE_NAME #case sensitive
+    warehouse: WAREHOUSE_NAME #case sensitive
+    database: DATABASE_NAME #case cap sensitive
 ```
 {% endcode %}
 

diff --git a/sdk/python/docs/source/conf.py b/sdk/python/docs/source/conf.py
@@ -115,7 +115,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = []
+html_static_path = ["_static"]
 
 
 # -- Options for HTMLHelp output ------------------------------------------

diff --git a/sdk/python/docs/source/feast.infra.online_stores.rst b/sdk/python/docs/source/feast.infra.online_stores.rst
@@ -52,6 +52,14 @@ feast.infra.online\_stores.redis module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.online\_stores.snowflake module
+-------------------------------------------
+
+.. automodule:: feast.infra.online_stores.snowflake
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 feast.infra.online\_stores.sqlite module
 ----------------------------------------
 

diff --git a/sdk/python/docs/source/index.rst b/sdk/python/docs/source/index.rst
@@ -250,18 +250,21 @@ Sqlite Online Store
 
 .. automodule:: feast.infra.online_stores.sqlite
     :members:
+    :noindex:
 
 Datastore Online Store
 ----------------------
 
 .. automodule:: feast.infra.online_stores.datastore
     :members:
+    :noindex:
 
 DynamoDB Online Store
 ---------------------
 
 .. automodule:: feast.infra.online_stores.dynamodb
     :members:
+    :noindex:
 
 Redis Online Store
 ------------------