"run" fails to pull image with authorization failure

[stefanb2]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Pull operation from company internal docker repository with podman run IMAGE fails whereas podman pull IMAGE works fine.

Steps to reproduce the issue:

1. podman image rm docker.COMPANY.COM:5004/PATH/mongo:3.4.7
2. podman run docker.COMPANY.COM:5004/PATH/mongo:3.4.7
3. podman pull docker.COMPANY.COM:5004/PATH/mongo:3.4.7

Describe the results you received:

$ podman image rm docker.COMPANY.COM:5004/PATH/mongo:3.4.7
ac18aa9255fd8023d1eb6e08d2fbc7b95c0c92166db040cb12b2ac19842e6c6c

$ podman run docker.COMPANY.COM:5004/PATH/mongo:3.4.7
Trying to pull docker.COMPANY.COM:5004/PATH/mongo:3.4.7...ERRO[0000] Error pulling image ref //docker.COMPANY.COM:5004/PATH/mongo:3.4.7: Error initializing source docker://docker.COMPANY.COM:5004/PATH/mongo:3.4.7: Error reading manifest 3.4.7 in docker.COMPANY.COM:5004/PATH/mongo: unauthorized: access to the requested resource is not authorized 
Failed
Error: unable to pull docker.COMPANY.COM:5004/PATH/mongo:3.4.7: unable to pull image: Error initializing source docker://docker.COMPANY.COM:5004/PATH/mongo:3.4.7: Error reading manifest 3.4.7 in docker.COMPANY.COM:5004/PATH/mongo: unauthorized: access to the requested resource is not authorized

$ podman pull docker.COMPANY.COM:5004/PATH/mongo:3.4.7
Trying to pull docker.COMPANY.COM:5004/PATH/mongo:3.4.7...Getting image source signatures
Copying blob 9cb9d47c5d80 done
...
Copying config ac18aa9255 done
Writing manifest to image destination
Storing signatures
ac18aa9255fd8023d1eb6e08d2fbc7b95c0c92166db040cb12b2ac19842e6c6c

Describe the results you expected:

According to man podman-run

If the IMAGE is not already loaded then podman run will pull the IMAGE, and all image dependencies, from the repository in the same way running podman pull IMAGE, before it starts the container from that image.

Output of podman version:

Version:            1.4.5-dev
RemoteAPI Version:  1
Go Version:         go1.12.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.6
  podman version: 1.4.5-dev
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-1.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.2.0, commit: 41010e63c287618b1dc34ee11d10d268e2feeefe'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 18037665792
  MemTotal: 33400737792
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 32002535424
  SwapTotal: 32002535424
  arch: amd64
  cpus: 12
  hostname: beckst-lnx
  kernel: 5.1.16-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 38h 36m 26.96s (Approximately 1.58 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/stefanb/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/stefanb/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /tmp/1000
  VolumePath: /home/stefanb/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

[stefanb2]

@muayyad-alsadi while not really fatal this does cause an annoying regression for users of podman-compose, i.e. users have to manually pull images beforehand.

[giuseppe]

could it have been a temporary failure? I am not able to reproduce it locally using a local registry using a httpasswd file

[stefanb2]

No, this happens for any docker image from our local docker repo.

I'm not sure if this is relevant: our local docker repo is

Nexus Repository Manager
OSS 3.10.0-04

That server is connected to our local AD, i.e. I have to use my AD account name and password to authenticate.

My authorization tokens are stored in ~/.podman-auth.json, apparently for BASIC authentication, i.e. Base64 encoded <user>:<password> strings

[giuseppe]

could you try if #3528 solve the issue you are seeing?

[cybernet]

does anyone have a fix for this ??

[mheon]

Can you open a new issue if you're still seeing this? This issue is over a year old.

[cybernet]

Can you open a new issue if you're still seeing this? This issue is over a year old.

exactly, OVER 1 YEAR old and no one confirmed that is fixed, however the issue was closed ...

[vrothberg]

Actually it was fixed in #3528 (July '19).

[vrothberg]

Just tested on 2.1.0 and it's still working

[cybernet]

#7780 - just opened