Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClusterServingRuntime can't pull image from my custom harbor #2681

Closed
631068264 opened this issue Feb 7, 2023 · 1 comment
Closed

ClusterServingRuntime can't pull image from my custom harbor #2681

631068264 opened this issue Feb 7, 2023 · 1 comment
Labels

Comments

@631068264
Copy link

631068264 commented Feb 7, 2023

/kind bug

What steps did you take and what happened:
apply a InferenceService

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "iris-test"
spec:
  predictor:
    tensorflow:
        storageUri: "http://xxx/model/tensorflow_iris.tar.gz"

get error

Status:
  Components:
    Predictor:
      Latest Created Revision:  iris-test-predictor-default-00001
  Conditions:
    Last Transition Time:  2023-02-07T09:45:34Z
    Message:               Revision "iris-test-predictor-default-00001" failed with message: Unable to fetch image "library/tensorflow/serving:2.6.2": failed to resolve image to digest: Get "https://index.docker.io/v2/": context deadline exceeded.
    Reason:                RevisionFailed
    Severity:              Info
    Status:                False
    Type:                  PredictorConfigurationReady
    Last Transition Time:  2023-02-07T09:39:59Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  PredictorReady
    Last Transition Time:  2023-02-07T09:39:59Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Severity:              Info
    Status:                False
    Type:                  PredictorRouteReady
    Last Transition Time:  2023-02-07T09:39:59Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  Ready
Events:
  Type     Reason         Age                    From                Message
  ----     ------         ----                   ----                -------
  Warning  InternalError  5m47s (x2 over 5m56s)  v1beta1Controllers  fails to reconcile predictor: fails to update knative service: Operation cannot be fulfilled on services.serving.knative.dev "iris-test-predictor-default": the object has been modified; please apply your changes to the latest version and try again

But I can pull this image on my node

docker pull library/tensorflow/serving:2.6.2

2.6.2: Pulling from library/tensorflow/serving
Digest: sha256:2f0b1213de372be6ddb310f9d5eecc6d6787a595b2466f1911b2b2a557bf9b8b
Status: Image is up to date for library/tensorflow/serving:2.6.2
docker.io/library/tensorflow/serving:2.6.2

The ClusterServingRuntime image look like not control by docker config
Then I use the whole repo name , I can pull on my node

docker pull harbor.xxx.cn/library/tensorflow/serving:2.6.2
2.6.2: Pulling from library/tensorflow/serving
Digest: sha256:2f0b1213de372be6ddb310f9d5eecc6d6787a595b2466f1911b2b2a557bf9b8b
Status: Downloaded newer image for harbor.xxx.cn/library/tensorflow/serving:2.6.2
harbor.xxx.cn/library/tensorflow/serving:2.6.2

But get error in InferenceService

Status:
  Components:
    Predictor:
      Latest Created Revision:  iris-test-predictor-default-00001
  Conditions:
    Last Transition Time:  2023-02-07T10:01:16Z
    Message:               Revision "iris-test-predictor-default-00001" failed with message: Unable to fetch image "harbor.xxx.cn/library/tensorflow/serving:2.6.2": failed to resolve image to digest: Get "https://harbor.xxx.cn/v2/": x509: certificate is not valid for any names, but wanted to match harbor.xxx.cn.
    Reason:                RevisionFailed
    Severity:              Info
    Status:                False
    Type:                  PredictorConfigurationReady
    Last Transition Time:  2023-02-07T10:01:16Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  PredictorReady
    Last Transition Time:  2023-02-07T10:01:16Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Severity:              Info
    Status:                False
    Type:                  PredictorRouteReady
    Last Transition Time:  2023-02-07T10:01:16Z
    Message:               Configuration "iris-test-predictor-default" does not have any ready Revision.
    Reason:                RevisionMissing
    Status:                False
    Type:                  Ready
Events:
  Type     Reason         Age   From                Message
  ----     ------         ----  ----                -------
  Warning  InternalError  15s   v1beta1Controllers  fails to reconcile predictor: fails to update knative service: Operation cannot be fulfilled on services.serving.knative.dev "iris-test-predictor-default": the object has been modified; please apply your changes to the latest version and try again

What did you expect to happen:

What's the InferenceService yaml:
[To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:
https://github.com/kubeflow/manifests/tree/v1.6-branch#kubeflow-components-versions

  • Istio Version: 1.14.1
  • Knative Version: 1.2.5
  • KServe Version: [release-0.8
  • Kubeflow version: 1.6.1
  • Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
  • Minikube/Kind version:
  • Kubernetes version: (use kubectl version): 1.22
  • OS (e.g. from /etc/os-release): Ubuntu 18.04.2
@631068264
Copy link
Author

631068264 commented Feb 7, 2023

#1928 save my life

https://knative.dev/docs/serving/configuration/deployment/#skipping-tag-resolution

Why not follow the container running time config ?

@631068264 631068264 reopened this Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants