[Regression] unsupported protocol scheme

[niedbalski]

[Environment]

Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic

[Description]

The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
      endpoint = ["niedbalski-bastion.cloud.sts:5000"]

This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.

The pod definition is:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
    - name: busybox
      image: niedbalski-bastion.cloud.sts:5000/busybox:latest
      command:
        - sleep
        - "3600"
  imagePullSecrets:
    - name: regcred
  restartPolicy: Always

New pods fail with the following error:

" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"

Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"

[Steps to reproduce]

1. Configure a private docker repository repository

2. Modify the containerd registry mirror config as follows:
   ** http://paste.ubuntu.com/p/yP63WMkVT6/

3. Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)

4. Status of the scheduled pod should be ImagePullBackOff
   and the before mentioned error should be raised.

[Possible workaround and solution]

1. As a workaround change the endpoint to support the scheme (https://)
2. Provide a fallback mechanism for URL parsing validation to fallback to http or https.

I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.

[niedbalski]

@dmcgowan or @crosbymichael let me know if you need any additional detail.

[niedbalski]

Reported in Ubuntu https://bugs.launchpad.net/bugs/1867398

[niedbalski]

This commit fixes the issue a022c21 in the 1.3 branch

This should have been imported through commit 37b9a34
niedbalski@alicorn:/canonical/source/containerd/upstream-2$ git show --oneline 37b9a34 | head -n1
37b9a34 Improve host fallback behaviour in docker remote
niedbalski@alicorn:/canonical/source/containerd/upstream-2$ git tag --contains 37b9a34
niedbalski@alicorn:~/canonical/source/containerd/upstream-2$

1.3.4 should be fixed.

[AkihiroSuda]

@dmcgowan @estesp Can we release v1.3.4 soon?

[dmcgowan]

1.3.4 has since been released