Issues with KFServing as part of a Kubeflow install #629

amygdala · 2020-01-09T17:41:52Z

/kind bug

I'm seeing some issues with the KFServing install that's part of the 'out-of-the-box' Kubeflow install (0.7.1). As documented, this 'should' work without the need to install additional stuff: the KF 0.7.1 install includes istio, knative-serving, and installs the kfserving-controller-manager statefulset.

Not clear if the following probs are all related, so this bug might need to be factored out into several.
It's possible that some of these issues relate to knative-serving vs kubeflow gateways conflicting, as apparently there can be issues if the transport like http or https could go via either gateway. e.g.: istio/istio#11509.

First, it looks like an inferenceservice can only be deployed into the automatically-created kubeflow-<user> namespace. Is this intended?
Otherwise, there's this error:

Error from server: error when creating "kfserving-tf-flowers.yaml": admission webhook "inferenceservice.kfserving-webhook-server.validator" denied the request: Cannot create the Inferenceservice "flowers-sample" in namespace "kubeflow": the namespace lacks label "serving.kubeflow.org/inferenceservice: enabled"

Once it is deployed, the inferenceservice is showing Ready==False, and giving 'Failed to reconcile predictor' errors:

% kubectl describe inferenceservice flowers-sample -n kubeflow-amyu
Name:         flowers-sample
Namespace:    kubeflow-amyu
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"...
API Version:  serving.kubeflow.org/v1alpha2
Kind:         InferenceService
Metadata:
  Creation Timestamp:  2020-01-09T02:55:25Z
  Generation:          6
  Resource Version:    7257
  Self Link:           /apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow-amyu/inferenceservices/flowers-sample
  UID:                 7c11596e-328b-11ea-a589-42010a800278
Spec:
  Default:
    Predictor:
      Tensorflow:
        Resources:
          Limits:
            Cpu:     1
            Memory:  2Gi
          Requests:
            Cpu:          1
            Memory:       2Gi
        Runtime Version:  1.14.0
        Storage Uri:      gs://kfserving-samples/models/tensorflow/flowers
Status:
  Canary:
  Conditions:
    Last Transition Time:  2020-01-09T02:56:01Z
    Message:               Waiting for VirtualService to be ready
    Reason:                Uninitialized
    Status:                Unknown
    Type:                  DefaultPredictorReady
    Last Transition Time:  2020-01-09T02:55:25Z
    Message:               Failed to reconcile predictor
    Reason:                PredictorHostnameUnknown
    Status:                False
    Type:                  Ready
    Last Transition Time:  2020-01-09T02:55:25Z
    Message:               Failed to reconcile predictor
    Reason:                PredictorHostnameUnknown
    Status:                False
    Type:                  RoutesReady
  Default:
    Predictor:
      Name:  flowers-sample-predictor-default-fjjkn
Events:      <none>

Then, in this section of the instructions: https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction
..this command does not return a value: SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
It’s not finding the status.url. Here’s what the json looks like. What should it be returning? I'm guessing the SERVICE_HOSTNAME should be set to flowers-sample-predictor-default.kubeflow-amyu.svc.cluster.local or similar, right? But perhaps due to the above issue I don't see that string in the json below.

kubectl get inferenceservice ${MODEL_NAME} -o json
{
    "apiVersion": "serving.kubeflow.org/v1alpha2",
    "kind": "InferenceService",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"serving.kubeflow.org/v1alpha2\",\"kind\":\"InferenceService\",\"metadata\":{\"annotations\":{},\"name\":\"flowers-sample\",\"namespace\":\"kubeflow\"},\"spec\":{\"default\":{\"predictor\":{\"tensorflow\":{\"storageUri\":\"gs://kfserving-samples/models/tensorflow/flowers\"}}}}}\n"
        },
        "creationTimestamp": "2020-01-08T21:17:24Z",
        "generation": 5,
        "name": "flowers-sample",
        "namespace": "kubeflow",
        "resourceVersion": "19958",
        "selfLink": "/apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow/inferenceservices/flowers-sample",
        "uid": "43c7f8c0-325c-11ea-bed8-42010a80015f"
    },
    "spec": {
        "default": {
            "predictor": {
                "tensorflow": {
                    "resources": {
                        "limits": {
                            "cpu": "1",
                            "memory": "2Gi"
                        },
                        "requests": {
                            "cpu": "1",
                            "memory": "2Gi"
                        }
                    },
                    "runtimeVersion": "1.14.0",
                    "storageUri": "gs://kfserving-samples/models/tensorflow/flowers"
                }
            }
        }
    },
    "status": {
        "canary": {},
        "conditions": [
            {
                "lastTransitionTime": "2020-01-08T21:17:58Z",
                "message": "Waiting for VirtualService to be ready",
                "reason": "Uninitialized",
                "status": "Unknown",
                "type": "DefaultPredictorReady"
            },
            {
                "lastTransitionTime": "2020-01-08T21:17:24Z",
                "message": "Failed to reconcile predictor",
                "reason": "PredictorHostnameUnknown",
                "status": "False",
                "type": "Ready"
            },
            {
                "lastTransitionTime": "2020-01-08T21:17:24Z",
                "message": "Failed to reconcile predictor",
                "reason": "PredictorHostnameUnknown",
                "status": "False",
                "type": "RoutesReady"
            }
        ],
        "default": {
            "predictor": {
                "name": "flowers-sample-predictor-default-jnp9v"
            }
        }
    }
}

Finally, even if I deploy the KF install so that the istio-ingressgateway is set up with an exernal IP, I can't successfully make an inference request by following the instructions. I get an origin auth failure.

% kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'  
35.192.166.45
% curl -v -H "Host: ${SERVICE_HOSTNAME}" http://35.192.166.45/v1/models/$MODEL_NAME:predict -d $INPUT_PATH 
*   Trying 35.192.166.45...
* TCP_NODELAY set
* Connected to 35.192.166.45 (35.192.166.45) port 80 (#0)
> POST /v1/models/flowers-sample:predict HTTP/1.1
> Host: flowers-sample-predictor-default.kubeflow-amyu.svc.cluster.local
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 16201
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 401 Unauthorized
< content-length: 29
< content-type: text/plain
< date: Thu, 09 Jan 2020 17:30:36 GMT
< server: istio-envoy
< connection: close
<
* we are done reading and this is set to close, stop send
* Closing connection 0
Origin authentication failed.

(cc @jlewi as fyi)

The text was updated successfully, but these errors were encountered:

wronk · 2020-01-10T00:06:59Z

I've also had issues getting the Tensorflow KFServing example up on a vanilla GCP Kubeflow deployment (v0.7.1 and v0.7.0). As @amygdala also mentioned above, I can't get the SERVICE_HOSTNAME (or CLUSTER_IP for that matter). Both are blank because the specified part of the json doesn't appear in the kubectl get responses.

I try getting KFServing into the cluster as suggested here. Relevant lines from the kfserving TF guide:

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
SERVICE_HOSTNAME=$(kubectl -n kubeflow get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)

yuzisun · 2020-01-10T00:12:02Z

@wronk did you get the same error amy is getting “waiting for virtual service to be ready”?

wronk · 2020-01-10T01:02:27Z

Here's my output after re-running these lines on the kubeflow deployment. I don't see that error about virtual service now, but I'm not sure it didn't appear before... I'm also not sure which namespace is the correct one to use (in case it's supposed to differ from the docs).

# After Kubeflow deployment:
TAG=0.2.2
CONFIG_URI=https://raw.githubusercontent.com/kubeflow/kfserving/master/install/$TAG/kfserving.yaml
kubectl apply -f ${CONFIG_URI} 

kubectl apply -f kfserving_flowers.yaml

$ MODEL_NAME=flowers-sample
$ kubectl get inferenceservice ${MODEL_NAME} -o json
{
    "apiVersion": "serving.kubeflow.org/v1alpha2",
    "kind": "InferenceService",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"serving.kubeflow.org/v1alpha2\",\"kind\":\"InferenceService\",\"metadata\":{\"annotations\":{},\"name\":\"flowers-sample\",\"namespace\":\"default\"},\"spec\":{\"default\":{\"predictor\":{\"minReplicas\":1,\"tensorflow\":{\"storageUri\":\"gs://kfserving-samples/models/tensorflow/flowers\"}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"mlUseOnly\",\"operator\":\"Equal\",\"value\":\"true\"}]}}\n"
        },
        "creationTimestamp": "2020-01-10T00:44:41Z",
        "generation": 2,
        "name": "flowers-sample",
        "namespace": "default",
        "resourceVersion": "855937",
        "selfLink": "/apis/serving.kubeflow.org/v1alpha2/namespaces/default/inferenceservices/flowers-sample",
        "uid": "638d82cf-3342-11ea-803c-42010a8e000e"
    },
    "spec": {
        "default": {
            "predictor": {
                "minReplicas": 1,
                "tensorflow": {
                    "resources": {
                        "limits": {
                            "cpu": "1",
                            "memory": "2Gi"
                        },
                        "requests": {
                            "cpu": "1",
                            "memory": "2Gi"
                        }
                    },
                    "runtimeVersion": "1.14.0",
                    "storageUri": "gs://kfserving-samples/models/tensorflow/flowers"
                }
            }
        }
    },
    "status": {
        "canary": {},
        "conditions": [
            {
                "lastTransitionTime": "2020-01-10T00:44:42Z",
                "message": "Failed to reconcile predictor",
                "reason": "PredictorHostnameUnknown",
                "status": "False",
                "type": "Ready"
            },
            {
                "lastTransitionTime": "2020-01-10T00:44:42Z",
                "message": "Failed to reconcile predictor",
                "reason": "PredictorHostnameUnknown",
                "status": "False",
                "type": "RoutesReady"
            }
        ],
        "default": {
            "predictor": {}
        }
    }
}

yuzisun · 2020-01-10T01:12:48Z

So you already installed kubeflow with kfserving and apply the kfserving 0.2.2 release yaml again ? With kubeflow installation kfserving gets deployed to kubeflow namespace while the standalone installation deploys to kfserving-system namespace.

krishnadurai · 2020-01-10T01:23:47Z

Edited

I'm facing a similar issue while I'm trying to setup Kubeflow ~~0.7.0~~ master branch KFServing and trying the flowers example.

The revisions 'flowers-sample-predictor-default' is stuck in status 'Deploying':

kubectl describe revision flowers-sample-predictor-default

Outputs

Name:         flowers-sample-predictor-default-w6qr4
Namespace:    kubeflow
Labels:       serving.knative.dev/configuration=flowers-sample-predictor-default
              serving.knative.dev/configurationGeneration=1
              serving.knative.dev/service=flowers-sample-predictor-default
              serving.kubeflow.org/inferenceservice=flowers-sample
Annotations:  autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
              autoscaling.knative.dev/target: 1
              internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
              queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
              serving.knative.dev/creator: system:serviceaccount:kubeflow:default
API Version:  serving.knative.dev/v1alpha1
Kind:         Revision
Metadata:
  Creation Timestamp:  2020-01-10T00:13:26Z
  Generate Name:       flowers-sample-predictor-default-
  Generation:          1
  Owner References:
    API Version:           serving.knative.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Configuration
    Name:                  flowers-sample-predictor-default
    UID:                   059f78ec-333e-11ea-93a5-42010a8e0070
  Resource Version:        674261
  Self Link:               /apis/serving.knative.dev/v1alpha1/namespaces/kubeflow/revisions/flowers-sample-predictor-default-w6qr4
  UID:                     05a62ebe-333e-11ea-93a5-42010a8e0070
Spec:
  Containers:
    Args:
      --port=9000
      --rest_api_port=8080
      --model_name=flowers-sample
      --model_base_path=/mnt/models
    Command:
      /usr/bin/tensorflow_model_server
    Image:  tensorflow/serving:
    Name:   kfserving-container
    Readiness Probe:
      Success Threshold:  1
      Tcp Socket:
        Port:  0
    Resources:
  Timeout Seconds:  60
Status:
  Conditions:
    Last Transition Time:  2020-01-10T00:15:37Z
    Message:               The target could not be activated.
    Reason:                TimedOut
    Severity:              Info
    Status:                False
    Type:                  Active
    Last Transition Time:  2020-01-10T00:13:26Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  ContainerHealthy
    Last Transition Time:  2020-01-10T00:13:26Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  Ready
    Last Transition Time:  2020-01-10T00:13:26Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  ResourcesAvailable
  Image Digest:            index.docker.io/tensorflow/serving@sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d
  Log URL:                 http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'05a62ebe-333e-11ea-93a5-42010a8e0070',type:phrase))))
  Observed Generation:     1
  Service Name:            flowers-sample-predictor-default-w6qr4
Events:                    <none>

Deployment seems to have scaled down to 0 replica-sets from 1 initially.

kubectl describe deployment flowers-sample-predictor-default-w6qr4-deployment

Name:                   flowers-sample-predictor-default-w6qr4-deployment
Namespace:              kubeflow
CreationTimestamp:      Thu, 09 Jan 2020 16:13:26 -0800
Labels:                 app=flowers-sample-predictor-default-w6qr4
                        serving.knative.dev/configuration=flowers-sample-predictor-default
                        serving.knative.dev/configurationGeneration=1
                        serving.knative.dev/revision=flowers-sample-predictor-default-w6qr4
                        serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
                        serving.knative.dev/service=flowers-sample-predictor-default
                        serving.kubeflow.org/inferenceservice=flowers-sample
Annotations:            autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
                        autoscaling.knative.dev/target: 1
                        deployment.kubernetes.io/revision: 1
                        internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
                        queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
                        serving.knative.dev/creator: system:serviceaccount:kubeflow:default
Selector:               serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
Replicas:               0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:       app=flowers-sample-predictor-default-w6qr4
                serving.knative.dev/configuration=flowers-sample-predictor-default
                serving.knative.dev/configurationGeneration=1
                serving.knative.dev/revision=flowers-sample-predictor-default-w6qr4
                serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
                serving.knative.dev/service=flowers-sample-predictor-default
                serving.kubeflow.org/inferenceservice=flowers-sample
  Annotations:  autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
                autoscaling.knative.dev/target: 1
                internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
                queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
                serving.knative.dev/creator: system:serviceaccount:kubeflow:default
                sidecar.istio.io/inject: true
                traffic.sidecar.istio.io/includeOutboundIPRanges: *
  Containers:
   kfserving-container:
    Image:      index.docker.io/tensorflow/serving@sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /usr/bin/tensorflow_model_server
    Args:
      --port=9000
      --rest_api_port=8080
      --model_name=flowers-sample
      --model_base_path=/mnt/models
    Environment:
      PORT:             8080
      K_REVISION:       flowers-sample-predictor-default-w6qr4
      K_CONFIGURATION:  flowers-sample-predictor-default
      K_SERVICE:        flowers-sample-predictor-default
    Mounts:
      /var/log from knative-var-log (rw)
   queue-proxy:
    Image:       gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:e0654305370cf3bbbd0f56f97789c92cf5215f752b70902eba5d5fc0e88c5aca
    Ports:       8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Requests:
      cpu:      25m
    Readiness:  exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
    Environment:
      SERVING_NAMESPACE:                kubeflow
      SERVING_SERVICE:                  flowers-sample-predictor-default
      SERVING_CONFIGURATION:            flowers-sample-predictor-default
      SERVING_REVISION:                 flowers-sample-predictor-default-w6qr4
      QUEUE_SERVING_PORT:               8012
      CONTAINER_CONCURRENCY:            0
      REVISION_TIMEOUT_SECONDS:         60
      SERVING_POD:                       (v1:metadata.name)
      SERVING_POD_IP:                    (v1:status.podIP)
      SERVING_LOGGING_CONFIG:           {
                                          "level": "info",
                                          "development": false,
                                          "outputPaths": ["stdout"],
                                          "errorOutputPaths": ["stderr"],
                                          "encoding": "json",
                                          "encoderConfig": {
                                            "timeKey": "ts",
                                            "levelKey": "level",
                                            "nameKey": "logger",
                                            "callerKey": "caller",
                                            "messageKey": "msg",
                                            "stacktraceKey": "stacktrace",
                                            "lineEnding": "",
                                            "levelEncoder": "",
                                            "timeEncoder": "iso8601",
                                            "durationEncoder": "",
                                            "callerEncoder": ""
                                          }
                                        }
      SERVING_LOGGING_LEVEL:
      SERVING_REQUEST_LOG_TEMPLATE:
      SERVING_REQUEST_METRICS_BACKEND:
      TRACING_CONFIG_ENABLE:            false
      TRACING_CONFIG_ZIPKIN_ENDPOINT:
      TRACING_CONFIG_DEBUG:             false
      TRACING_CONFIG_SAMPLE_RATE:       0.100000
      USER_PORT:                        8080
      SYSTEM_NAMESPACE:                 knative-serving
      METRICS_DOMAIN:                   knative.dev/serving
      USER_CONTAINER_NAME:              kfserving-container
      ENABLE_VAR_LOG_COLLECTION:        false
      VAR_LOG_VOLUME_NAME:              knative-var-log
      INTERNAL_VOLUME_PATH:             /var/knative-internal
      SERVING_READINESS_PROBE:          {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
    Mounts:                             <none>
  Volumes:
   knative-var-log:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   flowers-sample-predictor-default-w6qr4-deployment-556fdb96b7 (0/0 replicas created)
Events:          <none>

ChanYiLin · 2020-01-10T05:04:49Z

@amygdala and for others who faced the last issue you mentioned which related to istio authorization
I had discussion with the team who built the istio authorization solution ( @yanniszark )

So the temporary solution will be either

disable the auth by using kfctl_k8s_istio deployment
copy the token from the cookie when you login to the dashboard from browser to the X-Auth-Token in the curl command

and as a reminder, before adding X-Auth-Token to the curl command you have to edit envoyfilter to allow X-Auth-Token in the header (reference to this PR)

yuzisun · 2020-01-11T14:45:28Z

One option here is to configure knative to use kubeflow gateway https://github.com/knative/serving/blob/master/config/istio-ingress/config.yaml#L49, based on my testing this only works with knative 0.11 and previous versions do not support configuring the gateway in a different namespace other than knative-serving. We also need to change kfserving to point to the kubeflow gateway https://github.com/kubeflow/kfserving/blob/master/config/default/configmap/inferenceservice.yaml#L93. After that all the virtual services created by knative or kfserving will start to use kubeflow’s gateway to avoid conflicts.

amygdala · 2020-01-14T19:15:32Z

Just a note that to do the above, seems that Kubeflow's version of knative needs to be upgraded (from 0.8 to 0.11)

alexwennerberg · 2020-01-14T21:37:37Z

@yuzisun What values should those configuration variables be set to?

yuzisun · 2020-01-15T15:11:09Z

In kubeflow installation we have two istio gateway installed, they both serve on http port 80 and route to the same istio ingress gateway, so one way to solve the issue is to configure knative to use the kubeflow gateway instead.

kubectl get gateways --all-namespaces
NAMESPACE         NAME                      AGE
knative-serving   knative-ingress-gateway   4d4h
kubeflow          kubeflow-gateway          5d5h

Currently kubeflow installs knative 0.8 and we need to get onto 0.11 due to an issue fixed in 0.11 which allows configuring the gateway in custom namespace (Support custom namespaces for gateways in the config-istio configmap knative/serving#5859)
Edit knative config-istio configmap
kubectl edit cm config-istio -n knative-serving
change
gateway.knative-serving.knative-ingress-gateway: "istio-ingressgateway.istio-system.svc.cluster.local" to
gateway.kubeflow.kubeflow-gateway: "istio-ingressgateway.istio-system.svc.cluster.local
Edit kfserving configmap
kubectl edit cm inferenceservice-config -n kfserving-system
change
"ingressGateway" : "knative-ingress-gateway.knative-serving" to
"ingressGateway" : "kubeflow-gateway.kubeflow"

I'd love to see if someone can try out this and confirm this can fix the issue.

krishnadurai · 2020-01-17T07:05:15Z

@yuzisun there's a correction here:

3. Edit kfserving configmap
   `kubectl edit cm inferenceservice-config -n kfserving-system`

The namespace should be 'kubeflow'.

I'd love to see if someone can try out this and confirm this can fix the issue.

Unfortunately, this does not yet solve this issue.

kubectl describe service.serving.knative.dev flowers-sample-predictor-default

Status:
  Address:
    URL:  http://flowers-sample-predictor-default.kubeflow.svc.cluster.local
  Conditions:
    Last Transition Time:        2020-01-17T06:53:33Z
    Status:                      True
    Type:                        ConfigurationsReady
    Last Transition Time:        2020-01-17T06:53:33Z
    Message:                     Waiting for VirtualService to be ready
    Reason:                      Uninitialized
    Status:                      Unknown
    Type:                        Ready
    Last Transition Time:        2020-01-17T06:53:33Z
    Message:                     Waiting for VirtualService to be ready
    Reason:                      Uninitialized
    Status:                      Unknown
    Type:                        RoutesReady

janeman98 · 2020-01-25T00:58:12Z

I have a similar issue, the output for the following is BLANK
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

I believe this is because my environment don't have an external load balancer (using default Kubeflow installation)
kubectl -n istio-system get service istio-ingressgateway -o json

...
    },
    "status": {
        "loadBalancer": {}
    }
}

kubectl get svc istio-ingressgateway -n istio-system

NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
istio-ingressgateway   NodePort   10.98.65.35   <none>        15020:31369/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32341/TCP,15030:31286/TCP,15031:31518/TCP,15032:30609/TCP,15443:32557/TCP   126d

My istio-ingressgateway's type is NodePort. If I use CLUSTER-IP(10.98.65.35), the following work fine for me:
curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

Question: is an external load balancer required for KFServing?

yuzisun · 2020-01-25T12:17:56Z

@janeman98 load balancer is not required, there is a PR to fix the doc #618

wronk · 2020-02-03T19:14:35Z

@janeman98, I'm still having problems getting the TF flowers example to work from a KF deployment. Did you have to do anything special to make the example work after manually setting the CLUSTER_IP?

Perhaps I'm not setting SERVICE_HOSTNAME correctly? Or maybe there's a special cluster setting I need to define on GCP to enable incoming prediction requests?

cc @yuzisun

Steps to reproduce

Set up KF with GCP IAP
Used the instructions here. Modified to use the kfctl version v1.0-rc.1-0-g963c787 and the master version of CONFIG_URI that addresses #596 ☝️

Applied the example

kubectl -n kubeflow apply -f <my_path_to_kfserving>/kfserving_flowers.yaml
> inferenceservice.serving.kubeflow.org/flowers-sample created

Set the necessary env variables

MODEL_NAME=flowers-sample
INPUT_PATH=<my_path_to_kfserving>/input.json  # Same results with or without '@' symbol proceeding path
# Switched this to grab the cluster IP since the load balancer isn't setup (the specified code [here](https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction) returns empty string for CLUSTER_IP).
CLUSTER_IP=$(kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.clusterIP}')

echo $CLUSTER_IP
10.39.249.51

# The specified SERVICE_HOSTNAME script in the example also returns empty string, so setting it manually. Is this right? I don't know the exact proper format or how to check if it's valid.
SERVICE_HOSTNAME=flowers-sample-predictor-default.kubeflow.example.com

Curl times out

curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

*   Trying 10.39.249.51...
* TCP_NODELAY set
* Connection failed
* connect to 10.39.249.51 port 80 failed: Network is unreachable
* Failed to connect to 10.39.249.51 port 80: Network is unreachable
* Closing connection 0

stomplee · 2020-03-12T18:47:44Z

I'm experiencing issues with this as well. Used the https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_istio_dex.v1.0.0.yaml config first and tried this and it failed as it seems no configuration was made that sets up the ingress for it so instead I tried this one: https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_aws.v1.0.1.yaml and it doesn't even create an inferenceservice.

[ec2-user@ip-10-0-0-170 tfserving-test]$ kubectl apply -f tensorflow.yaml
error: unable to recognize "tensorflow.yaml": no matches for kind "InferenceService" in version "serving.kubeflow.org/v1alpha2"

janeman98 · 2020-03-12T23:34:27Z

@wronk
Sorry for my late response. I don't see your question until stomplee comment on this issue.

I install Kubeflow in minikube using
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml"
Since I don't have load balancer in my env, I use
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')
I don't do anything special on SERVICE_HOSTNAME
SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
--> this will set SERVICE_HOSTNAME=flowers-sample.default.example.com

yantriks-edi-bice · 2020-03-19T21:35:31Z

I'm facing this issue as well (even after upgrading to 1.0.1). On GCP with IAP. Used pip install in TF2-CPU notebook to upgrade to latest Kfserving package.

Seems this is the culprit:

Error detected in taxi-sample-predictor-default version taxi-sample-predictor-default-s5kvs

google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/kf-poc-edi/o?projection=noAcl&prefix=tfx_pipeline_output%2Fmy_tfx_on_kf_pipeline%2Fserving_model%2F1584479473%2F: Primary: /namespaces/saas-ml-dev.svc.id.goog with additional claims does not have storage.objects.list access to kf-poc-edi.

Tried following to no avail

KFServing = KFServingClient()
KFServing.set_credentials(storage_type='GCS',
#namespace='kubeflow-edi-bice',
#credentials_file='/secret/gcp/user-gcp-sa.json')#,
service_account='default-editor')
KFServing.create(isvc)

yantriks-edi-bice · 2020-03-20T18:24:54Z

3. SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)

Thanks @janeman98 - those last two bullets helped out as the sample instructions did not work as is

pvgbabu · 2020-05-14T21:57:41Z

Hi All, I need your help on fixing this Kfserving issue
I am using KFserving installed using PKS(1.15) Kubflow 1.0
When i try > kubectl apply -f tensorflow.yaml -n kubeflow
inferenceservice.serving.kubeflow.org/flowers-sample created
but

kubectl get inferenceservices -n kubeflow
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
flowers-sample False 2m43s
kubectl describe inferenceservices -n kubeflow
Message: Configuration "flowers-sample-predictor-default" is waiting for a Revision to become ready
kubectl get pods -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-6dc4884-g5h82 2/2 Running 1 23d
autoscaler-69bcc99c79-xnlvp 2/2 Running 2 23d
autoscaler-hpa-68cc87bfb9-c5cht 1/1 Running 0 23d
controller-95dc7f8bd-ptqq2 1/1 Running 0 23d
networking-istio-5b8c5c6cff-x4746 1/1 Running 0 23d
webhook-67847fb4b5-mgwsb 1/1 Running 0 23d

kubectl logs -l app=networking-istio -n knative-serving
W0514 20:27:32.108954 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18995786 (18999379)
W0514 20:35:11.115562 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18999993 (19003682)

Appreciate all your help in this.

issue-label-bot · 2020-05-14T21:57:48Z

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/inference	0.85

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

Manvi07 · 2020-06-26T10:53:43Z

@janeman98, I am facing exactly the same issue, as mentioned by @wronk. I tried this solution by you, but it again says connection timed out. Do I need to change the kubeflow version? (Currently I am using the configuration kfctl_k8s_istio.v1.0.2)

Also, the SERVICE_HOSTNAME is blank.

@janeman98, I'm still having problems getting the TF flowers example to work from a KF deployment. Did you have to do anything special to make the example work after manually setting the CLUSTER_IP?

Perhaps I'm not setting SERVICE_HOSTNAME correctly? Or maybe there's a special cluster setting I need to define on GCP to enable incoming prediction requests?

cc @yuzisun

Steps to reproduce

Set up KF with GCP IAP
Used the instructions here. Modified to use the kfctl version v1.0-rc.1-0-g963c787 and the master version of CONFIG_URI that addresses #596

Applied the example
kubectl -n kubeflow apply -f <my_path_to_kfserving>/kfserving_flowers.yaml
> inferenceservice.serving.kubeflow.org/flowers-sample created
Set the necessary env variables
MODEL_NAME=flowers-sample
INPUT_PATH=<my_path_to_kfserving>/input.json  # Same results with or without '@' symbol proceeding path
# Switched this to grab the cluster IP since the load balancer isn't setup (the specified code [here](https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction) returns empty string for CLUSTER_IP).
CLUSTER_IP=$(kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.clusterIP}')

echo $CLUSTER_IP
10.39.249.51

# The specified SERVICE_HOSTNAME script in the example also returns empty string, so setting it manually. Is this right? I don't know the exact proper format or how to check if it's valid.
SERVICE_HOSTNAME=flowers-sample-predictor-default.kubeflow.example.com
Curl times out
curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

*   Trying 10.39.249.51...
* TCP_NODELAY set
* Connection failed
* connect to 10.39.249.51 port 80 failed: Network is unreachable
* Failed to connect to 10.39.249.51 port 80: Network is unreachable
* Closing connection 0

jlewi · 2020-06-26T14:30:31Z

Hi Folks,

This issue was closed on January 29. If you are still having trouble I would suggest opening new issues. It also seems like multiple issues and platforms might be being discussed; I would suggest trying to create one issue for each specific problem.

added second change for ingress issue for The RC yaml has a bug for gateway kubectl edit cm inferenceservice-config -n kfserving-system kserve#629. Line 829

k8s-ci-robot added the kind/bug label Jan 9, 2020

wronk mentioned this issue Jan 10, 2020

All the examples inference service status stays at False #527

Closed

amygdala mentioned this issue Jan 14, 2020

update the version of knative bundled with KF kubeflow/kubeflow#4651

Closed

animeshsingh mentioned this issue Jan 18, 2020

Upgrading KFServing to KNative 0.11 #596

Closed

k8s-ci-robot assigned yuzisun Jan 18, 2020

yuzisun mentioned this issue Jan 25, 2020

Upgrade Knative serving to 0.11.1 kubeflow/manifests#651

Merged

k8s-ci-robot closed this as completed in kubeflow/manifests#651 Jan 30, 2020

janeman98 mentioned this issue Jan 30, 2020

Enabled test_sklearn.py, etc. to run in an env without ext load balancer #649

Merged

yuzisun mentioned this issue Apr 13, 2020

Protect KFServing endpoint #760

Closed

prashanthharshangi mentioned this issue Apr 22, 2020

Unable to connect to port : Operation timed out #793

Closed

issue-label-bot bot added the area/inference label May 14, 2020

MATRIX4284 added a commit to MATRIX4284/kfserving that referenced this issue Jul 9, 2020

Update kfserving.yaml

7defdab

added second change for ingress issue for The RC yaml has a bug for gateway kubectl edit cm inferenceservice-config -n kfserving-system kserve#629. Line 829

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues with KFServing as part of a Kubeflow install #629

Issues with KFServing as part of a Kubeflow install #629

amygdala commented Jan 9, 2020

wronk commented Jan 10, 2020

yuzisun commented Jan 10, 2020

wronk commented Jan 10, 2020

yuzisun commented Jan 10, 2020

krishnadurai commented Jan 10, 2020 •

edited

ChanYiLin commented Jan 10, 2020 •

edited

yuzisun commented Jan 11, 2020 •

edited

amygdala commented Jan 14, 2020

alexwennerberg commented Jan 14, 2020 •

edited

yuzisun commented Jan 15, 2020

krishnadurai commented Jan 17, 2020

janeman98 commented Jan 25, 2020

yuzisun commented Jan 25, 2020

wronk commented Feb 3, 2020 •

edited

stomplee commented Mar 12, 2020

janeman98 commented Mar 12, 2020

yantriks-edi-bice commented Mar 19, 2020 •

edited

yantriks-edi-bice commented Mar 20, 2020 •

edited

pvgbabu commented May 14, 2020

issue-label-bot bot commented May 14, 2020

Manvi07 commented Jun 26, 2020 •

edited

Steps to reproduce

jlewi commented Jun 26, 2020

Issues with KFServing as part of a Kubeflow install #629

Issues with KFServing as part of a Kubeflow install #629

Comments

amygdala commented Jan 9, 2020

wronk commented Jan 10, 2020

yuzisun commented Jan 10, 2020

wronk commented Jan 10, 2020

yuzisun commented Jan 10, 2020

krishnadurai commented Jan 10, 2020 • edited

ChanYiLin commented Jan 10, 2020 • edited

yuzisun commented Jan 11, 2020 • edited

amygdala commented Jan 14, 2020

alexwennerberg commented Jan 14, 2020 • edited

yuzisun commented Jan 15, 2020

krishnadurai commented Jan 17, 2020

janeman98 commented Jan 25, 2020

yuzisun commented Jan 25, 2020

wronk commented Feb 3, 2020 • edited

Steps to reproduce

stomplee commented Mar 12, 2020

janeman98 commented Mar 12, 2020

yantriks-edi-bice commented Mar 19, 2020 • edited

yantriks-edi-bice commented Mar 20, 2020 • edited

pvgbabu commented May 14, 2020

issue-label-bot bot commented May 14, 2020

Manvi07 commented Jun 26, 2020 • edited

Steps to reproduce

jlewi commented Jun 26, 2020

krishnadurai commented Jan 10, 2020 •

edited

ChanYiLin commented Jan 10, 2020 •

edited

yuzisun commented Jan 11, 2020 •

edited

alexwennerberg commented Jan 14, 2020 •

edited

wronk commented Feb 3, 2020 •

edited

yantriks-edi-bice commented Mar 19, 2020 •

edited

yantriks-edi-bice commented Mar 20, 2020 •

edited

Manvi07 commented Jun 26, 2020 •

edited