podman save --format=oci-archive fails to write to a pipe

[hexagonrecursion]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I wanted a quick and dirty way to copy an image from one computer to another. The manual for podman save says that it writes to stdout by default and the manual for podman load says that it reads from stdin by default so I assumed that piping the output of podman save to the destination via ssh would work, but podman save --format=oci-archive fails to write to a pipe.

Steps to reproduce the issue:

1. C="$(buildah from scratch)"

2. buildah commit "$C" bug

3. podman save --format=oci-archive bug | cat >/dev/null

Describe the results you received:

Error: error getting OCI archive ImageReference for ("/dev/stdout", "localhost/bug"): lstat /proc/2102/fd/pipe:[34531]: no such file or directory

Describe the results you expected:

The image archive is written to stdout.

Additional information you deem important (e.g. issue happens only occasionally):

• This bug happens every time
• Without --format=oci-archive the command does not error

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 03:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 2
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: hostname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.19.132-1.pvops.qubes.x86_64
  linkmode: dynamic
  memFree: 106500096
  memTotal: 479236096
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 997453824
  swapTotal: 1073737728
  uptime: 14m 16.04s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 03:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.2-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
QubesOS https://github.com/QubesOS

[rhatdan]

@QiWang19 PTAL

[QiWang19]

This can work without using | cat like examples from podman-save.1.md, since the default output has been set as /dev/stdout

podman save --format=oci-archive docker.io/library/alpine >/dev/null
@hexagonrecursion have you tried with this?

[hexagonrecursion]

@QiWang19 podman save --format=oci-archive bug | cat >/dev/null is just a stripped down "minimal complete verifiable example" that triggers the bug. As I said above the actual command I'm struggling with pipes the output of podman save through ssh into podman load on another machine like this:

podman save --format=oci-archive bug | ssh remotehost 'podman load'

My goal is to copy a container without having to create and the delete a temporary file.

For now I am using the following command instead as a workaround

podman save --format=docker-archive bug | ssh remotehost 'podman load'

[QiWang19]

@mtrmac is this a bug from containers/image?

[mtrmac]

Yes, in a way. c/image resolves the input file path into an absolute no-symlinks path (so that it can use the path for policy.json, which we can’t very well just remove now), and there just isn’t an absolute no-symlinks path for unnamed pipes.

(Go’s filepath.EvalSymlinks fails in a not particularly helpful way, not returning the last path that worked. OTOH it’s quite understandable that they didn’t anticipate this situation, and honestly even if they did return that path, I’m not exactly sure what to do with it.)

As an immediate workaround, using named pipes would probably work. But, ouch.

[hexagonrecursion]

@mtrmac

c/image resolves the input file path into an absolute no-symlinks path (so that it can use the path for policy.json, which we can’t very well just remove now), and there just isn’t an absolute no-symlinks path for unnamed pipes.

I am not sure how this is related to this bug. This bug is specifically about writing to an unnamed pipe failing. In the command podman save --format=oci-archive bug | cat >/dev/null from the reproduction steps listed above the opuput file path of podman save is an unnamed pipe.

[rhatdan]

Yes this is about the output path.

[mtrmac]

The relationship is that internally the code does not work with file descriptors, only with paths. So, Podman calls the code with /dev/stdout, and that an “input” to the path resolution that fails.

[rhatdan]

So podman needs to be special cased to not do this if the path is /dev/stdout.

[mtrmac]

Well, it’s already a Podman special case to say /dev/stdout — but there isn’t any easy file name to use instead. Named pipes would work, but, ugh. Extra code, need to clean up the file…

It’s almost tempting to allow the reference to carry a file descriptor. Yet, that breaks the concept of stateless reusable references pretty badly.

[rhatdan]

Well I am willing to go with whatever you think is best.

[mtrmac]

I’m very tempted to sing la la la and pretend this bug doesn’t exist :)

A named pipe would probably be least disruptive, when taking into account both c/image and the Podman call stack.

[QiWang19]

@mtrmac Can you help me with some details how to fix this with named pipe in podman and c/image?

can I sing with you together :) 🎤 🎤

[mtrmac]

In the place that currently decides to use /dev/stdout:

• Create a named pipe somewhere in /tmp; arrange for it to be deleted on termination.
• Open a write file descriptor for the pipe? (*)
• In a goroutine, read from the named pipe and copy everything to standard output
• Pass the path to the named pipe to c/image instead of /dev/stdout
• After c/image is done, close the named pipe file descriptor (*) and collect errors, if any, from the goroutine.

(*) FIFO behavior WRT EOF is a bit non-obvious, this is based on man 3p read.

[rhatdan]

Wasn't there a way to create a named pipe in the virtual address space. Something like opening a file @/tmp/fifo?

[hexagonrecursion]

How does podman save --format=docker-archive avoid this?

[QiWang19]

By only looking at the code I didn't see it resolving the symlink of /dev/stdout when saving as docker-archive.

[mtrmac]

Wasn't there a way to create a named pipe in the virtual address space. Something like opening a file @/tmp/fifo?

There is an “abstract namespace” for AF_UNIX sockets, but those are explicitly not paths (start with NUL).

[mtrmac]

How does podman save --format=docker-archive avoid this?

Yes, by that time we have given up on signature policies based on path names. But the feature still exists in the OCI transport, which is older.

[rhatdan]

There is an “abstract namespace” for AF_UNIX sockets, but those are explicitly not paths (start with NUL).
Yes this was what I was thinking of.