Skip to content

Files

Latest commit

 

History

History

python-helm-demo

Running Feast Python / Go Feature Server with Redis on Kubernetes

For this tutorial, we set up Feast with Redis.

We use the Feast CLI to register and materialize features from the current machine, and then retrieving via a Feast Python feature server deployed in Kubernetes

First, let's set up a Redis cluster

  1. Start minikube (minikube start)

  2. Use helm to install a default Redis cluster

    helm repo add bitnami https://charts.bitnami.com/bitnami 
    helm repo update 
    helm install my-redis bitnami/redis

  3. Port forward Redis so we can materialize features to it

    kubectl port-forward --namespace default svc/my-redis-master 6379:6379
  4. Get your Redis password using the command (pasted below for convenience). We'll need this to tell Feast how to communicate with the cluster.

     export REDIS_PASSWORD=$(kubectl get secret --namespace default my-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
     echo $REDIS_PASSWORD

Then, let's set up a MinIO S3 store

Manifests have been taken from Deploy Minio in your project.

  1. Deploy MinIO instance:

    kubectl apply -f minio-dev.yaml
    
  2. Forward the UI port:

    kubectl port-forward svc/minio-service 9090:9090
  3. Login to (localhost:9090)[http://localhost:9090] as minio/minio123 and create bucket called feast-demo.

  4. Stop previous port forwarding and forward the API port instead:

    kubectl port-forward svc/minio-service 9000:9000

Next, we setup a local Feast repo

  1. Install Feast with Redis dependencies pip install "feast[redis,aws]"

  2. The feature repo is already setup here, so you just need to swap in your Redis credentials. We need to modify the feature_store.yaml, which has one field for you to replace:

    sed "s/_REDIS_PASSWORD_/${REDIS_PASSWORD}/" feature_repo/feature_store.yaml.template > feature_repo/feature_store.yaml
    cat feature_repo/feature_store.yaml

    Example repo:

    registry: s3://localhost:9000/feast-demo/registry.db
    project: feast_python_demo
    provider: local
    online_store:
      type: redis
      connection_string: localhost:6379,password=****
    offline_store:
      type: file
    entity_key_serialization_version: 2
  3. To run feast apply from the current machine we need to define the AWS credentials to connect the MinIO S3 store, which are defined in minio.env:

    source minio.env
    cd  feature_repo
    feast apply
  4. Let's validate the setup by running some queries

    feast entities list
    feast feature-views list
  5. Materialize features to the online store:

    cd  feature_repo
    CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")                                    
    feast materialize-incremental $CURRENT_TIME

Now let's setup the Feast Server

  1. Add Feast's Python feature server chart repo

    helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
    helm repo update
  2. For this tutorial, we'll use a predefined configuration where we just needs to inject the Redis service password:

    sed "s/_REDIS_PASSWORD_/$REDIS_PASSWORD/" online_feature_store.yaml.template > online_feature_store.yaml
    cat online_feature_store.yaml

    As you see, the connection points to my-redis-master:6379 instead of localhost:6379.

  3. Install the Feast helm chart:

    helm upgrade --install feast-online feast-charts/feast-feature-server \
    --set fullnameOverride=online-server --set feast_mode=online \
    --set feature_store_yaml_base64=$(base64 -i 'online_feature_store.yaml')
  4. Patch the deployment to include MinIO settings:

    kubectl patch deployment online-server --type='json' -p='[
      {
        "op": "add",
        "path": "/spec/template/spec/containers/0/env/-",
        "value": {
          "name": "AWS_ACCESS_KEY_ID",
          "value": "minio"
        }
      },
      {
        "op": "add",
        "path": "/spec/template/spec/containers/0/env/-",
        "value": {
          "name": "AWS_SECRET_ACCESS_KEY",
          "value": "minio123"
        }
      },
      {
        "op": "add",
        "path": "/spec/template/spec/containers/0/env/-",
        "value": {
          "name": "AWS_DEFAULT_REGION",
          "value": "default"
        }
      },
      {
        "op": "add",
        "path": "/spec/template/spec/containers/0/env/-",
        "value": {
          "name": "FEAST_S3_ENDPOINT_URL",
          "value": "http://minio-service:9000"
        }
      }
    ]'
    kubectl wait --for=condition=available deployment/online-server --timeout=2m
  5. (Optional): check logs of the server to make sure it’s working

    kubectl logs svc/online-server
  6. Port forward to expose the grpc endpoint:

    kubectl port-forward svc/online-server 6566:80
  7. Run test fetches for online features:8.

    source minio.env
    cd test
    python test_python_fetch.py

    Output example:

    --- Online features with SDK ---
    WARNING:root:_list_feature_views will make breaking changes. Please use _list_batch_feature_views instead. _list_feature_views will behave like _list_all_feature_views in the future.
    conv_rate  :  [0.6799587607383728, 0.9761165976524353]
    driver_id  :  [1001, 1002]
    
    --- Online features with HTTP endpoint ---
    conv_rate  :  [0.67995876 0.9761166 ]
    driver_id  :  [1001 1002]