Would be nice to have a different 'podman pull' exit code for authorization failures

[debarshiray]

/kind feature

Description

This is a lot like #6190 but for authorization failures which is possibly more narrowly defined than network errors.

We are receiving feature requests against Toolbox from downstream distributors to detect when a registry requires the user to log in and handle the authorization by prompting the user as part of the container creation process (ie., toolbox create) without requiring users to invoke podman login on their own.

This is motivated to a great extent by coreos/toolbox, which does the same.

However, on a closer look, this isn't easy to implement in a Podman wrapper like Toolbox because it's difficult to know when podman pull failed due to lack of authorization.

The coreos/toolbox implementation assumes that every podman pull failure is an authorization issue. This is sub-optimal because the vast majority of Fedora (or Arch, or Debian, or ...) users won't be using private registries, and they are way more likely to encounter passing network or registry issues.

We have an initial implementation for Toolbox that tries to screen scrape the standard error stream of podman pull to guess the nature of a failure. However, this sort of screen scraping is very fragile and is a bad idea.

First, the messages that get dumped in the error stream don't have any fixed structure or format, and can change from one Podman version to another. Moreover, the error stream is also used to show things like progress bars, so those need to be skipped when parsing it. This makes it very fragile to test whether the parsing works as expected across several Podman versions in several operating systems.

Second, since the error stream is also used to show progress bars for podman pull, redirecting it away from a terminal device (to intercept its contents) prevents the progress bar from being shown. We show the progress bar when toolbox --verbose ... is used because they are useful for checking the status of the pulls. Ultimately, we are downloading hundreds of megabytes as part of a critical operation, so having access to a working progress bar is important.

So, long story short, it would be good if podman pull had a dedicated exit code to denote authorization failures.

Output of podman version:

Version:      3.2.1
API Version:  3.2.1
Go Version:   go1.15.12
Built:        Mon Jun 14 21:17:18 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: kolache
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.19-200.fc33.x86_64
  linkmode: dynamic
  memFree: 435675136
  memTotal: 8102621184
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 7061364736
  swapTotal: 8184127488
  uptime: 157h 9m 52.5s (Approximately 6.54 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/rishi/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/rishi/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/rishi/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.1
  Built: 1623698238
  BuiltTime: Mon Jun 14 21:17:18 2021
  GitCommit: ""
  GoVersion: go1.15.12
  OsArch: linux/amd64
  Version: 3.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.1-1.fc33.x86_64

[rhatdan]

I don't believe this will be easy to implement, since containers/image, as I understand it, has a hard time differentiating between "authorization required" and image does not exist.

@vrothberg @mtrmac WDYT?

[vrothberg]

Thanks for reaching out, @debarshiray!

I don't believe this will be easy to implement, since containers/image, as I understand it, has a hard time differentiating between "authorization required" and image does not exist.

That's right (same error even when logged in):

~ $ podman pull docker.io/idonot/exist:latest
Trying to pull docker.io/idonot/exist:latest...
Error: Error initializing source docker://idonot/exist:latest: Error reading manifest latest in docker.io/idonot/exist: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Many registries (not sure if all) return auth errors when trying to pull images that do not exist. That would lead to false positives when prompting users to login.

[baude]

Indeed, this has been asked for several times before but my understand is that until registries send better messages/response codes we cannot differentiate.

[mtrmac]

There are at least four major cases, with variants:

• Server rejects credentials, when converting them into a bearer token, with a 401 . This should show up as c/image/docker/ErrUnauthorizedForCredentials .
• Server accepts credentials (for a specific operation) and later explicitly refuses to perform it. Perhaps docker/distribution/registry/api/errcode.ErrorCodeUnauthorized or docker/distribution/registry/api/errcode.ErrorCodeDenied. Note that this is the failure case in @vrothberg ’s example above, and it is also the case for an anonymous attempt to access a missing repo . Surprisingly no permission check happens when asking for a bearer token, at least with this server right now.
• Server accepts valid credentials (or no credentials) and then pretends not to know about the image (to not disclose its existence), typically docker/distribution/registry/api/errcode.ErrorCodeManifestUnknown or IIRC others docker/distribution/registry/api/errcode.ErrorCodeUnknown. IIRC this used to be the case, now at least docker.io is returning the “unauthorized” status above, it seems.
• (Server returns a HTTP {404,401}? without a JSON response as defined in the API; IIRC that used to happen with Quay.)

So the first question is what do you want to do in all of these cases. (That decision could be easier when accessing a hard-coded image, e.g. registry.redhat.io/always-intended-to-be-public:foo ; but with user-specified input it just looks hard.) At the very least note that “provided but invalid credentials” and “missing credentials” are different use cases; “authorization failure” is too vague. Which one do you want to handle, and how?

Another question is how to expose this via Podman, and the difficulties from #6190 (comment) / #6190 (comment) still very much apply. (Is the libimage work such that it would make sense to call it directly? I guess not, I’m asking at least for my own education.)

[debarshiray]

Ok, thanks for all the prompt and thoughtful responses, everybody. Much appreciated.

I have forwarded a summary of this discussion to the downstream distributors pushing for this feature. ;)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[debarshiray]

I don't mind closing this issue because I managed to convince the relevant downstream distributor that this can't be achieved. :)