Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Predict on a KFService with saved model on S3


  1. Your ~/.kube/config should point to a cluster with KFServing installed.
  2. Your cluster's Istio Ingress gateway must be network accessible.
  3. Your cluster's Istio Egresss gateway must allow accessing S3 Storage
  4. The example uses the Kubeflow's Minio setup if you have Kubeflow installed, you can also setup your own Minio server or use other S3 compatible cloud storage.

Train TF mnist model and save on S3

Follow Kubeflow's TF mnist example to train a TF mnist model and save on S3, change following S3 access settings, modelDir and exportDir as needed. If you already have a mnist model saved on S3 you can skip this step.

export S3_USE_HTTPS=0 #set to 0 for default minio installs
export S3_ENDPOINT=minio-service.kubeflow:9000

kustomize edit add configmap mnist-map-training --from-literal=S3_ENDPOINT=${S3_ENDPOINT}
kustomize edit add configmap mnist-map-training --from-literal=AWS_ENDPOINT_URL=${AWS_ENDPOINT_URL}
kustomize edit add configmap mnist-map-training --from-literal=S3_USE_HTTPS=${S3_USE_HTTPS}

kustomize edit add configmap mnist-map-training --from-literal=modelDir=s3://mnist/v1
kustomize edit add configmap mnist-map-training --from-literal=exportDir=s3://mnist/v1/export

Create S3 Secret and attach to Service Account

If you already have a S3 secret created from last step you can skip this step, since KFServing is relying on secret annotations to setup proper S3 environment variables you may still need to add following annotations to your secret to overwrite S3 endpoint or other S3 options.

apiVersion: v1
kind: Secret
  name: mysecret
  annotations: minio-service.kubeflow:9000 # replace with your s3 endpoint "0" # by default 1, for testing with minio you need to set to 0
type: Opaque
  awsAccessKeyID: XXXX
  awsSecretAccessKey: XXXXXXXX

KFServing gets the secrets from your service account, you need to add the above created or existing secret to your service account's secret list. By default KFServing uses default service account, user can use own service account and overwrite on KFService CRD.

apiVersion: v1
kind: ServiceAccount
  name: sa
- name: mysecret

Apply the secret and service account

kubectl apply -f s3_secret.yaml

Create the KFService

Apply the CRD

kubectl apply -f tensorflow_s3.yaml 

Expected Output

$ created

Run a prediction

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -v -H "Host: mnist-s3.default.svc.cluster.local" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

Expected Output

*   Trying
* Connected to ( port 80 (#0)
> POST /v1/models/mnist-s3:predict HTTP/1.1
> Host: mnist-s3.default.svc.cluster.local
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 2052
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
< HTTP/1.1 100 Continue
* We are completely uploaded and fine

< HTTP/1.1 200 OK
< content-length: 218
< content-type: application/json
< date: Thu, 23 May 2019 01:33:08 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 20536
    "predictions": [
            "classes": 2,
            "predictions": [0.28852, 3.02198e-06, 0.484786, 0.123249, 0.000372552, 0.0635331, 0.00168883, 0.00327147, 0.0344911, 8.54185e-05]
* Connection #0 to host left intact
You can’t perform that action at this time.