Unable to pull from private repo

[naude-r]

Environmental Info:
K3s Version:
k3s version v1.18.4+k3s1 (97b7a0e

Node(s) CPU architecture, OS, and Version:
redhat 7.7
Linux yy 3.10.0-1062.12.1.el7.x86_64 #1 SMP Thu Dec 12 06:44:49 EST 2019 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
single master, no workers

Describe the bug:
cannot pull from private registry. docker pull does work

Steps To Reproduce:
default install of k3s. attempt to deploy image from local registry

Expected behavior:
pod deploy

Actual behavior:
pod does not deploy

Additional context / logs:

/var/lib/rancher/k3s/agent/etc/containerd/config.toml
registry:
    endpoint:
      - "https://registry"
"configs": {
        "https://registry": {
          "auth": {
            "username": "",
            "password": "",
            "auth": "c....==",
            "identitytoken": ""
          },
          "tls": {
            "caFile": "/etc/rancher/k3s/registry.crt",
            "certFile": "",
            "keyFile": ""
          }
        }
      },

ctr works:

ctr image pull --user user-id registry/image:latest
Password: 
registry/image:latest:               resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:0227c6c9d3cc79210bd015b9c1072faf3d18d30072a6aea9f2f776c9f7095f87: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:b04aeda99a8c8ac85edc3e9d5ed977580fa7badc75cbabbb5d87e18d53db7d24:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e37085983d0020916d82f875ed4ac10f6c721ea97928924a0b5eaa03e93bc000:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:36ed0a30ce606708bdcb4d006719f6d71d77e67a0a37495ef6c71c94faf24de1:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:dbe4bafae083eb867d7f2e2f37b18301c8e11892b71c5fd054bd0a8d3a6399f7:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:28565784d19301b6b8f4ccff5ded5befd6586c47b83cd51babb1f8eccf99ae71:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:2a1589b8dc54d330dff119225e2ae6eaf3e9ac0389fbb48b5c541b66e28c9807:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:64431928fec3f27310e934969d373a24c49023e88bd4e92e088d855dd6fd701a:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:333d6556854f88f659c9117b3b83a1c853930d45bbb3d1a7444fed5594541ce3:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f98253e3200f0ce1fdad3dde2ebdd0b1d5e369417341df6d3effed2781407e4c:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9bea2816ef5d87f97bb76291e2cf6fd8fc8330a13712a350f5b3a8bc88605d9c:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8af0f366a8633405d96ab4610d9e5500128ae967f90790097551b62bc8b64d2f:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:75aadbb659762a3eecb40bac66a47a5741408a6d3dc7157ca694fe3f2f58cccd:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:ed3e50a5961c2b630846e3a0eba938719a79996da771242a03a6973096b0a86d:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:5d03ffb2d922f8b17b5f0042c96b9d11dc822b4c635df3088218ee2d37947d97:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e1aede2e884b2c18a893780e419a2aa46d09b19939433a41727999862efa0f0f:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:fdd8338fd996048158c367979517908cff8de5163501537c0545288b2ef64c58:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e04bff10a5bfd13f6dd90537000e5cec07d99bc1ddad2d09af7966780a924a19:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:54fec2fa59d0a0de9cd2dec9850b36c43de451f1fd1c0a5bf8f1cf26a61a5da4:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:793cbc6f8a5918a32dc216bfcded294b39b466b134d11995092baf42f55bcdb2:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8b8f1a73da66c3cf92a5e307fbe6b53dbf51c33edf4914c9a469ac24fdc966cb:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9a08b87df7124cef7fdfded574ff69981116bb28c293e281dc50a63a0c4a6268:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:27d20d68eb689259f681237ff2f20db38ff8d1dc13f44600401ce1bf51fe9c47:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:57e3fe1f08a528bf2f8964485e30eeff1aee7d573d2e0f4b30836798cb26f2ab:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:273e054a60f5e89475edc274d546ff498dfb0a5f75f3ca884d4125c2ec632f21:    exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:b7dd01647a92dedec9093f4872d18b5a91e98a48731a7c9bc733d288be336017:    exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.6 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:0227c6c9d3cc79210bd015b9c1072faf3d18d30072a6aea9f2f776c9f7095f87...
done

crictl and k3s does not:

crictl --debug pull --auth c....== registry/image:latest
DEBU[2020-07-02T11:42:07.014821527+02:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:registry/image:latest,},Auth:&AuthConfig{Username:,Password:,Auth:c....==,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,} 
DEBU[2020-07-02T11:42:07.036475244+02:00] PullImageResponse: nil                       
FATA[2020-07-02T11:42:07.036546332+02:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry/image:latest": failed to resolve reference "registry/image:latest": failed to do request: Head https://registry/image/manifests/latest: EOF

was hoping to get away without going the docker route.

[tly1980]

I've found the similar issues on this one:

k3s version: k3s version v1.18.4+k3s1 (97b7a0e9)

On my server:

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

With my regististries configuration setup

/etc/rancher/k3s/registries.yaml

configs:
  "docker.pkg.github.com":
    auth:
      username: what-ever-user
      password: what-ever-password

And I was able to use the same username and password login to github registry and pull my docker images.

I suspect this is related to the authentications, because when I'm running a docker registry (in my k8s cluster) without requiring authentications, and the images can be pulled

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[naude-r]

closing issue. this is resolved in newer versions of k3s.