Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempting to pull missing image from insecure registry gives wrong message #1201

Closed
ColinChartier opened this issue Jul 18, 2019 · 9 comments

Comments

@ColinChartier
Copy link

commented Jul 18, 2019

Before image is pushed:

crictl pull 172.17.0.3:31653/webserver-doctest:e7376bfde60d
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "172.17.0.3:31653/webserver-doctest:e7376bfde60d": no available registry endpoint: failed to do request: Head https://172.17.0.3:31653/v2/webserver-doctest/manifests/e7376bfde60d: http: server gave HTTP response to HTTPS client

After image is pushed:

crictl pull 172.17.0.3:31653/webserver-doctest:5028ed6cc4ab
Image is up to date for sha256:8866ef28793fc646f5aff3434eb02406831a66ce6efcaac3f34a65642dfb46fb

Insecure registry config:

        [plugins.cri.registry.mirrors."172.17.0.3:31653"]
          endpoint = ["http://172.17.0.3:31653"]

Relevant logs:

INFO[2019-07-18T15:08:31.194164353Z] PullImage "172.17.0.3:31653/webserver-doctest:e7376bfde60d" 
DEBU[2019-07-18T15:08:31.194328492Z] resolving                                    
DEBU[2019-07-18T15:08:31.194360171Z] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="http://172.17.0.3:31653/v2/webserver-doctest/manifests/e7376bfde60d"
DEBU[2019-07-18T15:08:31.196526114Z] fetch response received                       response.headers=map[Content-Type:[application/json; charset=utf-8] Docker-Distribution-Api-Version:[registry/2.0] X-Content-Type-Options:[nosniff] Date:[Thu, 18 Jul 2019 15:08:31 GMT] Content-Length:[102]] status="404 Not Found" url="http://172.17.0.3:31653/v2/webserver-doctest/manifests/e7376bfde60d"
DEBU[2019-07-18T15:08:31.196596998Z] resolving                                    
DEBU[2019-07-18T15:08:31.196612422Z] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://172.17.0.3:31653/v2/webserver-doctest/manifests/e7376bfde60d"
ERRO[2019-07-18T15:08:31.197066387Z] PullImage "172.17.0.3:31653/webserver-doctest:e7376bfde60d" failed  error="failed to resolve image "172.17.0.3:31653/webserver-doctest:e7376bfde60d": no available registry endpoint: failed to do request: Head https://172.17.0.3:31653/v2/webserver-doctest/manifests/e7376bfde60d: http: server gave HTTP response to HTTPS client"

It seems that even with an explicitly insecure registry, it still attempts to use https when it receives a 404, and then tries https, and is surprised at getting an http response.

@mikebrow

This comment has been minimized.

Copy link
Member

commented Jul 18, 2019

please provide:
cat /etc/containerd/config.toml
containerd -v

@ColinChartier

This comment has been minimized.

Copy link
Author

commented Jul 18, 2019

# cat /etc/containerd/config.toml 
disabled_plugins = ["aufs", "btrfs", "zfs"]
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."172.17.0.3:31653"]
          endpoint = ["http://172.17.0.3:31653"]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

# containerd -v
containerd github.com/containerd/containerd 1.2.6-0ubuntu1 
@mikebrow

This comment has been minimized.

Copy link
Member

commented Jul 18, 2019

have to think this over... the code checks the mirrors then the default host which it derives from the reference.. And as you can see the default is to use https://reference...

There is no check in the code for failing from the mirror to the default where the default is derived from whether or not some number of mirrors were http or https and there is no way, to my knowledge, for you to specify what the default should be.

hmm...

@Random-Liu

This comment has been minimized.

Copy link
Member

commented Jul 24, 2019

Hm, there is an issue here.

We blindly try mirrors one by one no matter what errors we hit.
We should only fallback to mirrors when there is a connection problem to the registry. There is a complex logic about this in Docker https://github.com/moby/moby/blob/8e610b2b55bfd1bfa9436ab110d311f5e8a74dcb/distribution/errors.go#L116

@dmcgowan Any suggestion about this? I remember you said that you are working on a library in containerd for the coming Docker integration.

@Random-Liu Random-Liu added the kind/bug label Jul 24, 2019

@Random-Liu

This comment has been minimized.

Copy link
Member

commented Jul 25, 2019

We should use the new registry mirror support in containerd containerd/containerd#3400, which should fix this issue.

@Random-Liu Random-Liu added this to the v1.3 milestone Jul 25, 2019

@mikebrow

This comment has been minimized.

Copy link
Member

commented Jul 25, 2019

That works.

@joedborg

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

I'm seeing the same issue with a private registry that's HTTP(notS) and basic auth.

Failed to pull image "172.31.55.16:5000/nginx:latest": rpc error: code = Unknown desc = failed to resolve image "172.31.55.16:5000/nginx:latest": no available registry endpoint: failed to do request: Head https://172.31.55.16:5000/v2/nginx/manifests/latest: http: server gave HTTP response to HTTPS client`
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."http://172.31.55.16:5000"]
          endpoint = ["http://172.31.55.16:5000"]
      [plugins.cri.registry.auths] 
        [plugins.cri.registry.auths."http://172.31.55.16:5000"]
          username = "testuser"
          password = "testpassword"
@joedborg

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

It is working via ctr.

sudo ctr image pull --skip-verify --plain-http --user testuser:testpassword 172.31.55.16:5000/nginx:latest

@Random-Liu Random-Liu modified the milestones: v1.3, v1.2 Aug 9, 2019

@Random-Liu

This comment has been minimized.

Copy link
Member

commented Aug 9, 2019

Both the original "wrong error message" issue, and the issue mentioned in #1201 (comment) will be fixed in 1.3 by #1227

I changed the milestone to 1.2, because I think we should cherry-pick a small fix for the "wrong error message" in 1.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.