fix: Fix Java helm charts to work with refactored logic. Fix FTS image (

#3105) * fix: Fix Java helm charts to work with refactored logic. Also include gcp/aws in FTS image Signed-off-by: Danny Chiao <danny@tecton.ai> * fix helm docs and accidental issues Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * fix Signed-off-by: Danny Chiao <danny@tecton.ai> * fix issues with templates Signed-off-by: Danny Chiao <danny@tecton.ai> * fix issues with templates Signed-off-by: Danny Chiao <danny@tecton.ai> * fix serving README Signed-off-by: Danny Chiao <danny@tecton.ai> * fix go Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Danny Chiao <danny@tecton.ai>
feast-dev · Aug 19, 2022 · 2b493e0 · 2b493e0
1 parent 0a0c974
commit 2b493e0
Show file tree

Hide file tree

Showing 22 changed files with 266 additions and 201 deletions.
diff --git a/Makefile b/Makefile
@@ -363,6 +363,15 @@ build-feature-server-java-docker:
 		-t $(REGISTRY)/feature-server-java:$(VERSION) \
 		-f java/infra/docker/feature-server/Dockerfile --load .
 
+build-java-docker-dev:
+	make build-java-no-tests REVISION=dev
+	docker buildx build --build-arg VERSION=dev \
+		-t feastdev/feature-transformation-server:dev \
+		-f sdk/python/feast/infra/transformation_servers/Dockerfile --load .
+	docker buildx build --build-arg VERSION=dev \
+		-t feastdev/feature-server-java:dev \
+		-f java/infra/docker/feature-server/Dockerfile.dev --load .
+
 # Documentation
 
 install-dependencies-proto-docs:

diff --git a/examples/java-demo/README.md b/examples/java-demo/README.md
@@ -52,12 +52,12 @@ For this tutorial, we setup Feast with Redis, using the Feast CLI to register an
     ```bash
    minikube addons enable gcp-auth
    ```
-3. Add Feast's Java feature server chart repo
+2. Add Feast's Java feature server chart repo
     ```bash
     helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
     helm repo update
     ```
-4. Modify the application-override.yaml file to have your credentials + bucket location:
+3. Modify the application-override.yaml file to have your credentials + bucket location:
     ```yaml
     feature-server:
       application-override.yaml:
@@ -77,16 +77,21 @@ For this tutorial, we setup Feast with Redis, using the Feast CLI to register an
         cache_ttl_seconds: 60
       project: feast_java_demo
     ```
-5. Install the Feast helm chart: `helm install feast-release feast-charts/feast --values application-override.yaml`
-6. (Optional): check logs of the server to make sure it’s working
+4. Install the Feast helm chart: `helm install feast-release feast-charts/feast --values application-override.yaml`
+   > **Dev instructions**: if you're changing the java logic or chart, you can do 
+   >1. `eval $(minikube docker-env)`
+   >2. `make build-java-docker-dev` 
+   >3. In the `application-override.yaml`, uncomment the two `image: tag: dev` blocks 
+   >4. `helm install feast-release ../../../infra/charts/feast --values application-override.yaml`
+5. (Optional): check logs of the server to make sure it’s working
    ```bash
    kubectl logs svc/feast-release-feature-server
    ```
-7. Port forward to expose the grpc endpoint:
+6. Port forward to expose the grpc endpoint:
    ```bash
    kubectl port-forward svc/feast-release-feature-server 6566:6566
    ```
-8. Make a gRPC call:
+7. Make a gRPC call:
     - Python example
       ```bash
         python test.py

diff --git a/examples/java-demo/feature_repo/application-override.yaml b/examples/java-demo/feature_repo/application-override.yaml
@@ -11,8 +11,17 @@ feature-server:
                 port: 6379
                 password: [YOUR PASSWORD]
             entityKeySerializationVersion: 2
+    # Uncomment below for dev
+#    image:
+#      tag: dev
+
+# Uncomment below for dev
+#transformation-service:
+#  image:
+#    tag: dev
+
 global:
   registry:
-    path: gs://[YOUR BUCKET]/demo-repo/registry.db
+    path: gs://[YOUR GCS BUCKET]/demo-repo/registry.db
     cache_ttl_seconds: 60
   project: feast_java_demo
diff --git a/examples/java-demo/feature_repo/feature_store.yaml b/examples/java-demo/feature_repo/feature_store.yaml
@@ -1,4 +1,4 @@
-registry: gs://[YOUR BUCKET]/demo-repo/registry.db
+registry: gs://[YOUR GCS BUCKET]/demo-repo/registry.db
 project: feast_java_demo
 provider: gcp
 online_store:

diff --git a/infra/charts/feast-feature-server/README.md b/infra/charts/feast-feature-server/README.md
@@ -1,44 +1,9 @@
-# feast-feature-server
-
-![Version: 0.23.0](https://img.shields.io/badge/Version-0.23.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
-
-Feast Feature Server in Go or Python
-
-**Homepage:** <https://feast.dev/>
-
-## Source Code
-
-* <https://github.com/feast-dev/feast>
-
-## Values
-
-| Key                                | Type   | Default          | Description |
-| ---------------------------------- | ------ | ---------------- | ----------- |
-| affinity                           | object | `{}`             |             |
-| fullnameOverride                   | string | `""`             |             |
-| image.pullPolicy                   | string | `"IfNotPresent"` |             |
-| image.repository                   | string | `""`             |             |
-| image.tag                          | string | `""`             |             |
-| imagePullSecrets                   | list   | `[]`             |             |
-| livenessProbe.initialDelaySeconds  | int    | `30`             |             |
-| livenessProbe.periodSeconds        | int    | `30`             |             |
-| nameOverride                       | string | `""`             |             |
-| nodeSelector                       | object | `{}`             |             |
-| podAnnotations                     | object | `{}`             |             |
-| podSecurityContext                 | object | `{}`             |             |
-| readinessProbe.initialDelaySeconds | int    | `20`             |             |
-| readinessProbe.periodSeconds       | int    | `10`             |             |
-| replicaCount                       | int    | `1`              |             |
-| resources                          | object | `{}`             |             |
-| securityContext                    | object | `{}`             |             |
-| service.port                       | int    | `80`             |             |
-| service.type                       | string | `"ClusterIP"`    |             |
-| tolerations                        | list   | `[]`             |             |
-
-----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.11.0](https://github.com/norwoodj/helm-docs/releases/v1.11.0)
+# Feast Python / Go Feature Server Helm Charts
 
+Current chart version is `0.23.0`
 
+## Installation
+### Python feature server
 Docker repository and tag are required. Helm install example:
 ```
 helm install feast-feature-server . --set image.repository=REPO --set image.tag=TAG
@@ -58,25 +23,51 @@ RUN pip install feast
 COPY feature_store.yaml /feature_store.yaml
 ```
 
+### Go feature server
+> Warning: this is experimental, and only supports a local file registry + Redis
+
 Furthermore, if you wish to use the Go feature server, then you must install the Apache Arrow C++ libraries, and your `feature_store.yaml` should include `go_feature_server: True`.
 For more details, see the [docs](https://docs.feast.dev/reference/feature-servers/go-feature-server).
+
 The docker image might look like:
 ```
-FROM python:3.8
-
-RUN apt update && \
-    apt install -y jq
-
-RUN pip install pip --upgrade
-
-RUN pip install feast
+ FROM python:3.8
+
+ RUN apt update && \
+     apt install -y jq
+ RUN pip install pip --upgrade
+ RUN pip install feast
+ RUN apt update
+ RUN apt install -y -V ca-certificates lsb-release wget
+ RUN wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+ RUN apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+ RUN apt update
+ RUN apt -y install libarrow-dev
+
+ COPY feature_store.yaml /feature_store.yaml
+ ```
 
-RUN apt update
-RUN apt install -y -V ca-certificates lsb-release wget
-RUN wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
-RUN apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
-RUN apt update
-RUN apt -y install libarrow-dev
+## Values
 
-COPY feature_store.yaml /feature_store.yaml
-```
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| affinity | object | `{}` |  |
+| fullnameOverride | string | `""` |  |
+| image.pullPolicy | string | `"IfNotPresent"` |  |
+| image.repository | string | `""` | [required] The repository for the Docker image |
+| image.tag | string | `""` | [required] The Docker image tag |
+| imagePullSecrets | list | `[]` |  |
+| livenessProbe.initialDelaySeconds | int | `30` |  |
+| livenessProbe.periodSeconds | int | `30` |  |
+| nameOverride | string | `""` |  |
+| nodeSelector | object | `{}` |  |
+| podAnnotations | object | `{}` |  |
+| podSecurityContext | object | `{}` |  |
+| readinessProbe.initialDelaySeconds | int | `20` |  |
+| readinessProbe.periodSeconds | int | `10` |  |
+| replicaCount | int | `1` |  |
+| resources | object | `{}` |  |
+| securityContext | object | `{}` |  |
+| service.port | int | `80` |  |
+| service.type | string | `"ClusterIP"` |  |
+| tolerations | list | `[]` |  |
diff --git a/infra/charts/feast-feature-server/README.md.gotmpl b/infra/charts/feast-feature-server/README.md.gotmpl
@@ -0,0 +1,52 @@
+# Feast Python / Go Feature Server Helm Charts
+
+Current chart version is `{{ template "chart.version" . }}`
+
+## Installation
+### Python feature server
+Docker repository and tag are required. Helm install example:
+```
+helm install feast-feature-server . --set image.repository=REPO --set image.tag=TAG
+```
+
+Deployment assumes that `feature_store.yaml` exists on docker image. Example docker image:
+```
+FROM python:3.8
+
+RUN apt update && \
+    apt install -y jq
+
+RUN pip install pip --upgrade
+
+RUN pip install feast
+
+COPY feature_store.yaml /feature_store.yaml
+```
+
+### Go feature server
+> Warning: this is experimental, and only supports a local file registry + Redis
+
+Furthermore, if you wish to use the Go feature server, then you must install the Apache Arrow C++ libraries, and your `feature_store.yaml` should include `go_feature_server: True`.
+For more details, see the [docs](https://docs.feast.dev/reference/feature-servers/go-feature-server).
+
+The docker image might look like:
+```
+ FROM python:3.8
+
+ RUN apt update && \
+     apt install -y jq
+ RUN pip install pip --upgrade
+ RUN pip install feast
+ RUN apt update
+ RUN apt install -y -V ca-certificates lsb-release wget
+ RUN wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+ RUN apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+ RUN apt update
+ RUN apt -y install libarrow-dev
+
+ COPY feature_store.yaml /feature_store.yaml
+ ```
+
+{{ template "chart.requirementsSection" . }}
+
+{{ template "chart.valuesSection" . }}
diff --git a/infra/charts/feast-feature-server/values.yaml b/infra/charts/feast-feature-server/values.yaml
@@ -5,8 +5,10 @@
 replicaCount: 1
 
 image:
+  # image.repository -- [required] The repository for the Docker image
   repository: ""
   pullPolicy: IfNotPresent
+  # image.tag -- [required] The Docker image tag
   tag: ""
 
 imagePullSecrets: []

diff --git a/infra/charts/feast-python-server/README.md b/infra/charts/feast-python-server/README.md
@@ -1,14 +1,26 @@
-# feast-python-server
+# Feast Python Feature Server Helm Charts
 
-![Version: 0.23.0](https://img.shields.io/badge/Version-0.23.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
+Current chart version is `0.23.0`
 
-Feast Feature Server in Python
+## Installation
+Docker repository and tag are required. Helm install example:
+```
+helm install feast-python-server . --set image.repository=REPO --set image.tag=TAG
+```
+
+Deployment assumes that `feature_store.yaml` exists on docker image. Example docker image:
+```
+FROM python:3.8
+
+RUN apt update && \
+    apt install -y jq
 
-**Homepage:** <https://feast.dev/>
+RUN pip install pip --upgrade
 
-## Source Code
+RUN pip install feast
 
-* <https://github.com/feast-dev/feast>
+COPY feature_store.yaml /feature_store.yaml
+```
 
 ## Values
 
@@ -33,27 +45,4 @@ Feast Feature Server in Python
 | securityContext | object | `{}` |  |
 | service.port | int | `80` |  |
 | service.type | string | `"ClusterIP"` |  |
-| tolerations | list | `[]` |  |
-
-----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.5.0](https://github.com/norwoodj/helm-docs/releases/v1.5.0)
-
-
-Docker repository and tag are required. Helm install example:
-```
-helm install feast-python-server . --set image.repository=REPO --set image.tag=TAG
-```
-
-Deployment assumes that `feature_store.yaml` exists on docker image. Example docker image:
-```
-FROM python:3.8
-
-RUN apt update && \
-    apt install -y jq
-
-RUN pip install pip --upgrade
-
-RUN pip install feast
-
-COPY feature_store.yaml /feature_store.yaml
-```
+| tolerations | list | `[]` |  |
diff --git a/infra/charts/feast-python-server/README.md.gotmpl b/infra/charts/feast-python-server/README.md.gotmpl
@@ -0,0 +1,27 @@
+# Feast Python Feature Server Helm Charts
+
+Current chart version is `{{ template "chart.version" . }}`
+
+## Installation
+Docker repository and tag are required. Helm install example:
+```
+helm install feast-python-server . --set image.repository=REPO --set image.tag=TAG
+```
+
+Deployment assumes that `feature_store.yaml` exists on docker image. Example docker image:
+```
+FROM python:3.8
+
+RUN apt update && \
+    apt install -y jq
+
+RUN pip install pip --upgrade
+
+RUN pip install feast
+
+COPY feature_store.yaml /feature_store.yaml
+```
+
+{{ template "chart.requirementsSection" . }}
+
+{{ template "chart.valuesSection" . }}
diff --git a/infra/charts/feast-python-server/values.yaml b/infra/charts/feast-python-server/values.yaml
@@ -5,8 +5,10 @@
 replicaCount: 1
 
 image:
+  # image.repository -- [required] The repository for the Docker image
   repository: ""
   pullPolicy: IfNotPresent
+  # image.tag -- [required] The Docker image tag
   tag: ""
 
 imagePullSecrets: []

diff --git a/infra/charts/feast/README.md b/infra/charts/feast/README.md
@@ -54,9 +54,9 @@ For more details, please see: https://docs.feast.dev/how-to-guides/running-feast
 
 | Repository | Name | Version |
 |------------|------|---------|
-| https://charts.helm.sh/stable | redis | 10.5.6  |
-| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.23.0  |
-| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.23.0  |
+| https://charts.helm.sh/stable | redis | 10.5.6 |
+| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.23.0 |
+| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.23.0 |
 
 ## Values
 

diff --git a/infra/charts/feast/README.md.gotmpl b/infra/charts/feast/README.md.gotmpl
@@ -1,7 +1,5 @@
 # Feast Helm Charts
 
-> :warning: **Disclaimer**: Since Feast 0.10 our vision is to manage all infrastructure for feature store from one place - Feast SDK. But while this new paradigm is still in development, we are planning to support the installation of some Feast components (like Java feature server) through Helm chart presented in this repository. However, we do not expect helm chart to become a long-term solution for deploying Feast components to production, and some frictions still might exist. For example, you will need to manually sync some configurations from [feature_store.yaml](https://docs.feast.dev/reference/feature-repository/feature-store-yaml) into the chart context (like path to the registry file or project name).
-
 This repo contains Helm charts for Feast components that are being installed on Kubernetes:
 * Feast (root chart): The complete Helm chart containing all Feast components and dependencies. Most users will use this chart, but can selectively enable/disable subcharts using the values.yaml file.
     * [Feature Server](charts/feature-server): High performant JVM-based implementation of feature server.