feat: Add Snowflake materialization engine (#2948)

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
feast-dev · Aug 18, 2022 · f3b522b · f3b522b
1 parent 83c5efb
commit f3b522b
Show file tree

Hide file tree

Showing 25 changed files with 1,376 additions and 102 deletions.
diff --git a/.github/workflows/pr_local_integration_tests.yml b/.github/workflows/pr_local_integration_tests.yml
@@ -75,4 +75,4 @@ jobs:
           IS_TEST: "True"
           FEAST_LOCAL_ONLINE_CONTAINER: "True"
           FEAST_IS_LOCAL_TEST: "True"
-        run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization" sdk/python/tests
+        run: pytest -n 8 --cov=./ --cov-report=xml --color=yes --integration -k "not gcs_registry and not s3_registry and not test_lambda_materialization and not test_snowflake_materialization" sdk/python/tests
diff --git a/docs/reference/batch-materialization/README.md b/docs/reference/batch-materialization/README.md
@@ -2,4 +2,6 @@
 
 Please see [Batch Materialization Engine](../../getting-started/architecture-and-components/batch-materialization-engine.md) for an explanation of batch materialization engines.
 
+{% page-ref page="snowflake.md" %}
+
 {% page-ref page="bytewax.md" %}
diff --git a/docs/reference/batch-materialization/snowflake.md b/docs/reference/batch-materialization/snowflake.md
@@ -0,0 +1,28 @@
+# Snowflake
+
+## Description
+
+The [Snowflake](https://trial.snowflake.com) batch materialization engine provides a highly scalable and parallel execution engine using a Snowflake Warehouse for batch materializations operations (`materialize` and `materialize-incremental`) when using a `SnowflakeSource`.
+
+The engine requires no additional configuration other than for you to supply Snowflake's standard login and context details. The engine leverages custom (automatically deployed for you) Python UDFs to do the proper serialization of your offline store data to your online serving tables.
+
+When using all three options together, `snowflake.offline`, `snowflake.engine`, and `snowflake.online`, you get the most unique experience of unlimited scale and performance + governance and data security.
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+...
+offline_store:
+  type: snowflake.offline
+...
+batch_engine:
+  type: snowflake.engine
+  account: snowflake_deployment.us-east-1
+  user: user_login
+  password: user_password
+  role: sysadmin
+  warehouse: demo_wh
+  database: FEAST
+```
+{% endcode %}
diff --git a/docs/reference/offline-stores/snowflake.md b/docs/reference/offline-stores/snowflake.md
@@ -46,7 +46,7 @@ Below is a matrix indicating which functionality is supported by `SnowflakeRetri
 | export to dataframe                                   | yes |
 | export to arrow table                                 | yes |
 | export to arrow batches                               | no  |
-| export to SQL                                         | no  |
+| export to SQL                                         | yes  |
 | export to data lake (S3, GCS, etc.)                   | yes |
 | export to data warehouse                              | yes |
 | export as Spark dataframe                             | no  |

diff --git a/docs/tutorials/tutorials-overview/driver-stats-on-snowflake.md b/docs/tutorials/tutorials-overview/driver-stats-on-snowflake.md
@@ -6,7 +6,7 @@ description: >-
 # Drivers stats on Snowflake
 
 In the steps below, we will set up a sample Feast project that leverages Snowflake
-as an offline store.
+as an offline store + materialization engine + online store.
 
 Starting with data in a Snowflake table, we will register that table to the feature store and define features associated with the columns in that table. From there, we will generate historical training data based on those feature definitions and then materialize the latest feature values into the online store. Lastly, we will retrieve the materialized feature values.
 
@@ -46,7 +46,7 @@ The following files will automatically be created in your project folder:
 
 #### Inspect `feature_store.yaml`
 
-Here you will see the information that you entered. This template will use Snowflake as an offline store and SQLite as the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.
+Here you will see the information that you entered. This template will use Snowflake as the offline store, materialization engine, and the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.
 
 {% code title="feature_store.yaml" %}
 ```yaml
@@ -61,6 +61,14 @@ offline_store:
     role: ROLE_NAME #case sensitive
     warehouse: WAREHOUSE_NAME #case sensitive
     database: DATABASE_NAME #case cap sensitive
+batch_engine:
+    type: snowflake.engine
+    account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
+    user: USERNAME
+    password: PASSWORD
+    role: ROLE_NAME #case sensitive
+    warehouse: WAREHOUSE_NAME #case sensitive
+    database: DATABASE_NAME #case cap sensitive
 online_store:
     type: snowflake.online
     account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com

diff --git a/sdk/python/docs/source/feast.infra.materialization.rst b/sdk/python/docs/source/feast.infra.materialization.rst
@@ -28,6 +28,14 @@ feast.infra.materialization.local\_engine module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.materialization.snowflake\_engine module
+----------------------------------------------------
+
+.. automodule:: feast.infra.materialization.snowflake_engine
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

diff --git a/sdk/python/docs/source/feast.infra.utils.rst b/sdk/python/docs/source/feast.infra.utils.rst
@@ -28,14 +28,6 @@ feast.infra.utils.hbase\_utils module
    :undoc-members:
    :show-inheritance:
 
-feast.infra.utils.snowflake\_utils module
------------------------------------------
-
-.. automodule:: feast.infra.utils.snowflake_utils
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 Module contents
 ---------------