podman -r manifest push does not work unless I log into quay within the machine vm first

[jmontleon]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I've set up a podman machine, done a multiarch build, and tried to push it using the machine. It seems to work up until I run podman -r manifest push.

Steps to reproduce the issue:

1. podman machine init --cpus 2 -m 8192
2. podman machine start
3. podman machine ssh sudo rpm-ostree install qemu-user-static
4. podman machine ssh sudo systemctl reboot
5. git clone https://github.com/jwmatthews/case_watcher.git
6. cd case_watcher
7. podman -r build -f Dockerfile --manifest quay.io/jmontleon/case_watcher:latest --platform linux/amd64,linux/arm64 .
8. podman -r manifest push quay.io/jmontleon/case_watcher:latest docker://quay.io/jmontleon/case_watcher:latest

Describe the results you received:

$ podman -r --log-level=debug manifest push quay.io/jmontleon/case_watcher:latest docker://quay.io/jmontleon/case_watcher:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman -r --log-level=debug manifest push quay.io/jmontleon/case_watcher:latest docker://quay.io/jmontleon/case_watcher:latest) 
DEBU[0000] SSH Ident Key "/home/jason/.ssh/podman-machine-default" SHA256:S4G1wRA6yAbG7O/Tq15fzTnWekJmCdB0tu4BAz8Q5zQ ssh-ed25519 
DEBU[0000] Found SSH_AUTH_SOCK "/run/user/1000/ssh-agent.socket", ssh-agent signer(s) enabled 
DEBU[0000] SSH Agent Key SHA256:i13XTbtJgcZoJn2oXAYp7FJvBHqqJL9nPC3n1WdyEsA ssh-rsa 
DEBU[0000] SSH Agent Key SHA256:rJsMx7sPGxhckIxUh4mjhDBgQwAhr0QG6F9eBoDXUZo ssh-dss 
DEBU[0000] SSH Agent Key SHA256:DoYGRPSmnKydjI5kfbFmZDkgq9Df/V++1QUrmEdlwsg ecdsa-sha2-nistp521 
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.4/libpod/_ping 
DEBU[0000] SSH Ident Key "/home/jason/.ssh/podman-machine-default" SHA256:S4G1wRA6yAbG7O/Tq15fzTnWekJmCdB0tu4BAz8Q5zQ ssh-ed25519 
DEBU[0000] Found SSH_AUTH_SOCK "/run/user/1000/ssh-agent.socket", ssh-agent signer(s) enabled 
DEBU[0000] SSH Agent Key SHA256:i13XTbtJgcZoJn2oXAYp7FJvBHqqJL9nPC3n1WdyEsA ssh-rsa 
DEBU[0000] SSH Agent Key SHA256:rJsMx7sPGxhckIxUh4mjhDBgQwAhr0QG6F9eBoDXUZo ssh-dss 
DEBU[0000] SSH Agent Key SHA256:DoYGRPSmnKydjI5kfbFmZDkgq9Df/V++1QUrmEdlwsg ecdsa-sha2-nistp521 
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.4/libpod/_ping 
DEBU[0000] DoRequest Method: POST URI: http://d/v3.4.4/libpod/manifests/quay.io%2Fjmontleon%2Fcase_watcher:latest/push 
DEBU[0001] Called push.PersistentPostRunE(podman -r --log-level=debug manifest push quay.io/jmontleon/case_watcher:latest docker://quay.io/jmontleon/case_watcher:latest) 

This returns very fast and nothing is pushed.

Describe the results you expected:
The manifest actually gets pushed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Wed Dec  8 16:45:07 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "35"
  eventLogger: journald
  hostname: pixelbook.montleon.intra
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.16-201.pixelbook.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 271175680
  memTotal: 16659902464
  ociRuntime:
    name: crun
    package: crun-1.4.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.3
      commit: 61c9600d1335127eba65632731e2d72bc3f0b9e8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 25768484864
  swapTotal: 25769795584
  uptime: 11h 54m 12.23s (Approximately 0.46 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/jason/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jason/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/jason/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1638999907
  BuiltTime: Wed Dec  8 16:45:07 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.4-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshing Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No. I reproduced this trying to help @jwmatthews experiencing the same problem on a Mac using 4.0.2.

Additional environment details (AWS, VirtualBox, physical, etc.):
If I do a normal build podman -r build -f Dockerfile . -t .... I can run podman -r push ... and it will push without first SSHing into the VM and running podman login.

If I do a multiarch build it seems like it will only work if I ssh into the vm log in to quay, exit, and then run podman -r manifest push ...

[jmontleon]

Looking at push it looks like auth headers might be missing. This seems to get it working:

diff --git a/pkg/bindings/manifests/manifests.go b/pkg/bindings/manifests/manifests.go
index 458cb913a..4237f9fc7 100644
--- a/pkg/bindings/manifests/manifests.go
+++ b/pkg/bindings/manifests/manifests.go
@@ -11,7 +11,9 @@ import (
 
        "github.com/blang/semver"
        "github.com/containers/image/v5/manifest"
+       imageTypes "github.com/containers/image/v5/types"
        "github.com/containers/podman/v4/pkg/api/handlers"
+       "github.com/containers/podman/v4/pkg/auth"
        "github.com/containers/podman/v4/pkg/bindings"
        "github.com/containers/podman/v4/pkg/bindings/images"
        "github.com/containers/podman/v4/version"
@@ -179,6 +181,11 @@ func Push(ctx context.Context, name, destination string, options *images.PushOpt
                return "", err
        }
 
+       header, err := auth.MakeXRegistryAuthHeader(&imageTypes.SystemContext{AuthFilePath: options.GetAuthfile()}, options.GetUsername(), options.GetPassword())
+       if err != nil {
+               return "", err
+       }
+
        params, err := options.ToParams()
        if err != nil {
                return "", err
@@ -192,11 +199,11 @@ func Push(ctx context.Context, name, destination string, options *images.PushOpt
 
        var response *bindings.APIResponse
        if bindings.ServiceVersion(ctx).GTE(semver.MustParse("4.0.0")) {
-               response, err = conn.DoRequest(ctx, nil, http.MethodPost, "/manifests/%s/registry/%s", params, nil, name, destination)
+               response, err = conn.DoRequest(ctx, nil, http.MethodPost, "/manifests/%s/registry/%s", params, header, name, destination)
        } else {
                params.Set("image", name)
                params.Set("destination", destination)
-               response, err = conn.DoRequest(ctx, nil, http.MethodPost, "/manifests/%s/push", params, nil, name)
+               response, err = conn.DoRequest(ctx, nil, http.MethodPost, "/manifests/%s/push", params, header, name)
        }
        if err != nil {
                return "", err

[rhatdan]

Interested in opening a PR?

[jmontleon]

Sure!