Error Pulling Image s2i-minimal-notebook

[dymurray]

[dymurray@dymurray-desktop sbcli]$ oc get is                          
NAME                      DOCKER REPO                                     TAGS      UPDATED                                                 
jupyterhub                172.30.1.1:5000/dylan/jupyterhub                latest    21 minutes ago                                          
s2i-minimal-notebook      172.30.1.1:5000/dylan/s2i-minimal-notebook      3.5       26 minutes ago                                          
s2i-scipy-notebook        172.30.1.1:5000/dylan/s2i-scipy-notebook        3.5       23 minutes ago                                          
s2i-tensorflow-notebook   172.30.1.1:5000/dylan/s2i-tensorflow-notebook 

$ oc describe pod jupyterhub-nb-272442c5-2de4d4-2d40fd-2d8054-2dfcc1349e9c42
  FirstSeen     LastSeen        Count   From                    SubObjectPath                   Type            Reason          Message
  ---------     --------        -----   ----                    -------------                   --------        ------          -------
  11s           11s             1       default-scheduler                                       Normal          Scheduled       Successfully assigned jupyterhub-nb-272442c5-2de4d4-2d40fd-2d8054-2dfcc1349e9c42 to localhost
  10s           10s             1       kubelet, localhost      spec.containers{notebook}       Normal          Pulling         pulling image "s2i-minimal-notebook:3.5"
  5s            5s              1       kubelet, localhost      spec.containers{notebook}       Warning         Failed          Failed to pull image "s2i-minimal-notebook:3.5": rpc error: code = Unknown desc = repository docker.io/s2i-minimal-notebook not found: does not exist or no pull access
  5s            5s              1       kubelet, localhost      spec.containers{notebook}       Warning         Failed          Error: ErrImagePull
  4s            4s              1       kubelet, localhost      spec.containers{notebook}       Normal          BackOff         Back-off pulling image "s2i-minimal-notebook:3.5"
  4s            4s              1       kubelet, localhost      spec.containers{notebook}       Warning         Failed          Error: ImagePullBackOff

Any idea why I am seeing this?

[dymurray]

Just to clarify... the jupyterhub-nb pod is created when I do "start server". This is what fails. I don't get why the jupyterhub imagestream isn't failing to be pulled but the minimal-notebook is.

[GrahamDumpleton]

What OpenShift instance are you using?

The problem is that the OpenShift instance is missing a default configuration for what is called the ImagePolicy plugin and so that plugin is not enabled as it should be. The configuration should be present, but if the cluster was upgraded from an older OpenShift version it may be missing. If using recent Minishift it is also missing because adding it was removed from oc cluster up and Minishift didn't compensate and add it back.

For Minishift you can see a discussion about it in:

• ImagePolicy plugin for local image lookup is not configured properly. minishift/minishift#2392

For Minishift you can fix the issue by running:

minishift openshift config set --patch '{"admissionConfig": {"pluginConfig": {"openshift.io/ImagePolicy": {"configuration": {"resolutionRules": null, "kind": "ImagePolicyConfig", "executionRules": [{"skipOnResolutionFailure": true, "invertMatch": false, "name": "execution-denied", "onResources": [{"resource": "pods", "group": ""}, {"resource": "builds", "group": ""}], "matchRegistries": null, "matchIntegratedRegistry": false, "matchImageLabels": null, "matchImageAnnotations": [{"set": false, "value": "true", "key": "images.openshift.io/deny-execution"}], "ignoreNamespaceOverride": false, "reject": true, "matchDockerImageLabels": null}], "apiVersion": "v1", "resolveImages": ""}, "location": ""}}}}'

If you are on a full OpenShift installation, then the master config file for the cluster needs to be updated to add the appropriate ImagePolicy plugin configuration. I am not sure what services need to be restarted after that.

As per:

• Provide a empty ImagePolicy to get the default behavior in 3.9 minishift/minishift#2379

the ImagePolicy plugin configuration needs to look something like:

admissionConfig:
  pluginConfig:
    GenericAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig
      location: ""
    openshift.io/ImagePolicy:
      configuration:
        apiVersion: v1
        executionRules:
        - ignoreNamespaceOverride: false
          invertMatch: false
          matchDockerImageLabels: null
          matchImageAnnotations:
          - key: images.openshift.io/deny-execution
            set: false
            value: "true"
          matchImageLabels: null
          matchIntegratedRegistry: false
          matchRegistries: null
          name: execution-denied
          onResources:
          - group: ""
            resource: pods
          - group: ""
            resource: builds
          reject: true
          skipOnResolutionFailure: true
        kind: ImagePolicyConfig
        resolutionRules: null
        resolveImages: ""
      location: ""

This is taken from another Minishift issue, but should be same for a full OpenShift cluster.

As a temporary workaround until you can resolve the issue, you can change:

s2i-minimal-notebook

in the template parameter to be:

172.30.1.1:5000/dylan/s2i-minimal-notebook

That is, reference it from the internal image registry explicitly.

[dymurray]

Thanks @GrahamDumpleton for that information that's very helpful. I am using openshift v3.10.0-alpha.0+3611001-1427 and doing oc cluster up. I found this issue: openshift/origin#19910 which appears to be relevant to my problems. For now I can make the changes manually. Thanks!

[GrahamDumpleton]

If modifying the master config manually, you will need to use --use-existing-config option and possible also specify a place to save the config so you can get to it to modify it. The environment will then persist, allowing you to modify the config and then restart the instance. Using Minishift may be easier.

[dymurray]

Thanks good point. I will spin up a minishift instance. Appreciate the help!

[dymurray]

@GrahamDumpleton Seeing this error when running your config command:

[dymurray@dymurray sbcli]$ minishift openshift config set --patch '{"admissionConfig": {"plugin
Config": {"openshift.io/ImagePolicy": {"configuration": {"resolutionRules": null, "kind": "Imag
ePolicyConfig", "executionRules": [{"skipOnResolutionFailure": true, "invertMatch": false, "nam
e": "execution-denied", "onResources": [{"resource": "pods", "group": ""}, {"resource": "builds
", "group": ""}], "matchRegistries": null, "matchIntegratedRegistry": false, "matchImageLabels"
: null, "matchImageAnnotations": [{"set": false, "value": "true", "key": "images.openshift.io/d
eny-execution"}], "ignoreNamespaceOverride": false, "reject": true, "matchDockerImageLabels": n
ull}], "apiVersion": "v1", "resolveImages": ""}, "location": ""}}}}'
Patching OpenShift configuration /var/lib/origin/openshift.local.config/master/master-config.yaml with {"admissionConfig": {"pluginConfig": {"openshift.io/ImagePolicy": {"configuration": {"resolutionRules": null, "kind": "ImagePolicyConfig", "executionRules": [{"skipOnResolutionFailure": true, "invertMatch": false, "name": "execution-denied", "onResources": [{"resource": "pods", "group": ""}, {"resource": "builds", "group": ""}], "matchRegistries": null, "matchIntegratedRegistry": false, "matchImageLabels": null, "matchImageAnnotations": [{"set": false, "value": "true", "key": "images.openshift.io/deny-execution"}], "ignoreNamespaceOverride": false, "reject": true, "matchDockerImageLabels": null}], "apiVersion": "v1", "resolveImages": ""}, "location": ""}}}}
E0613 08:01:13.156056   18550 openshift.go:96] Creating patched configuration failed. Not applying the changes.ssh command error:
command : docker exec -t origin /usr/bin/openshift ex config patch /var/lib/origin/openshift.local.config/master/master-config.yaml --patch='{"admissionConfig": {"pluginConfig": {"openshift.io/ImagePolicy": {"configuration": {"resolutionRules": null, "kind": "ImagePolicyConfig", "executionRules": [{"skipOnResolutionFailure": true, "invertMatch": false, "name": "execution-denied", "onResources": [{"resource": "pods", "group": ""}, {"resource": "builds", "group": ""}], "matchRegistries": null, "matchIntegratedRegistry": false, "matchImageLabels": null, "matchImageAnnotations": [{"set": false, "value": "true", "key": "images.openshift.io/deny-execution"}], "ignoreNamespaceOverride": false, "reject": true, "matchDockerImageLabels": null}], "apiVersion": "v1", "resolveImages": ""}, "location": ""}}}}'
err     : exit status 1
output  : Error: unknown command "ex" for "openshift"
Run 'openshift --help' for usage.

[GrahamDumpleton]

What minishift version do you have? Ensure you do a minishift update if this is an old minishift version. If not used for a while also do a minishift delete to delete the VM and start over.

I know the command works with:

minishift v1.17.0+f974f0c

[dymurray]

Thanks @GrahamDumpleton! That worked for me. You can feel free to close this issue.

[karpikpl]

For someone that faces the same issue, following the thread from @GrahamDumpleton forminishift v1.25.0+90fb23e
I had to run:

minishift openshift config set --patch '{"admissionConfig": {"pluginConfig": {"openshift.io/ImagePolicy": {"configuration": {"resolutionRules": null, "kind": "ImagePolicyConfig", "executionRules": [{"skipOnResolutionFailure": true, "invertMatch": false, "name": "execution-denied", "onResources": [{"resource": "pods", "group": ""}, {"resource": "builds", "group": ""}], "matchRegistries": null, "matchIntegratedRegistry": false, "matchImageLabels": null, "matchImageAnnotations": [{"set": false, "value": "true", "key": "images.openshift.io/deny-execution"}], "ignoreNamespaceOverride": false, "reject": true, "matchDockerImageLabels": null}], "apiVersion": "v1", "resolveImages": ""}, "location": ""}}}}' --target kube

[GrahamDumpleton]

That minishift config fix doesn't work as far as I know in latest minishift versions (ODK 3.11) and there is no workaround. I believe it will only be fixed in next ODK version. So may not be possible to use latest minishift versions.

[karpikpl]

I see.
I've been using it on OKD 3.10 and minishift v1.25.0+90fb23e