Tarballs produced by buildkit are treated as OCI archives

[SquidDev]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman image load on a docker-archive tarball produced by buildkit treats it as an OCI archive rather than a docker archive, meaning tag information is discarded.

Steps to reproduce the issue:

1. Build an image and export it to a docker tarball using buildkit: buildctl build --frontend=dockerfile.v0 --progress=plain --local dockerfile=. --local context=. --output type=docker,name=test,dest=../image.tar
2. Import it into Podman: podman image load --input ../image2.tar.

Describe the results you received:
The image is imported into localhost/latest:latest.

Describe the results you expected:
The image is imported as localhost/test:latest.

Additional information you deem important (e.g. issue happens only occasionally):
I believe this is because buildkit produces tarballs containing an index.json file, and containers prefers the OCI format:

podman/vendor/github.com/containers/common/libimage/load.go

Line 40 in c260cbd

for _, f := range []func() ([]string, error){

While I appreciate buildkit is doing something weird here, Docker does correctly import such a tarball, and so I felt it was worth reporting the discrepancy.

Output of podman version:

Version:      3.2.2
API Version:  3.2.2
Go Version:   go1.16.5
Git Commit:   d577c44e359f9f8284b38cf984f939b3020badc3
Built:        Mon Jun 28 20:13:34 2021
OS/Arch:      linux/amd64

Output of `podman info --debug`

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.29-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b'
  cpus: 6
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: autoclave.squiddev.cc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 970
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 970
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.39-1-lts
  linkmode: dynamic
  memFree: 942088192
  memTotal: 16784269312
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.20.1-1
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 38271d1c8d9641a2cdc70acfa3dcb6996d124b3d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/970/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.11-1
    version: |-
      slirp4netns version 1.1.11
      commit: 368e69ccc074628d17a9bb9a35b8f4b9f74db4c6
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 2703339520
  swapTotal: 4199542784
  uptime: 856h 15m 26.14s (Approximately 35.67 days)
registries: {}
store:
  configFile: /var/vampyroteuthis/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.6-1
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.6
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /var/vampyroteuthis/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 8
  runRoot: /tmp/podman-run-970/containers
  volumePath: /var/vampyroteuthis/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1624911214
  BuiltTime: Mon Jun 28 20:13:34 2021
  GitCommit: d577c44e359f9f8284b38cf984f939b3020badc3
  GoVersion: go1.16.5
  OsArch: linux/amd64
  Version: 3.2.2

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Qi podman
Name            : podman
Version         : 3.2.2-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables  libseccomp  runc  slirp4netns  libsystemd  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2 [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 72.37 MiB
Packager        : Morten Linderud <foxboron@archlinux.org>
Build Date      : Mon 28 Jun 2021 08:13:34 PM UTC
Install Date    : Tue 29 Jun 2021 10:13:26 AM UTC
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

[vrothberg]

Thanks for opening the issue!

The produced tarball looks like a mix of an oci-archive (i.e., oci-layout plus index.json plus blobs) and a docker-archive (i.e., manifest.json).

$ ls
blobs  index.json  manifest.json  oci-layout

index.json set "org.opencontainers.image.ref.name":"latest" which what Podman is looking at. Docker seems to read "RepoTags":["test:latest"] from the manifest.json. Note that the manifest.json is not part of OCI image spec (see https://github.com/opencontainers/image-spec/blob/master/image-layout.md).

I fear there's not much we can do as buildkit seems to produce a tarball of an OCI layout plus custom logic in the manifest.json.

@mtrmac WDYT?

[mtrmac]

Does loading this as a docker-archive: work? If so, we could just switch the order of the attempts easily enough, at the cost of making OCI-archive users slower.

Or, more correctly, I think Podman should let the caller specify the input format; the caller almost always knows, so we shouldn’t have to guess when the guesses are so costly and the caller should just tell us. That would also let the caller deal with this ambiguity.

Ideally, we would detect a tarfile, decompress it to a staging file, apply a heuristic to decide which format it is, and then submit the already-uncompressed data to c/image. Most of that just doesn’t exist and it is a fair amount of code. There are some hints at such an API in c/image/docker/archive.NewReader (with the caller still having to deal with decompression to a seekable file, for this to be efficient), and we don’t have a similar OCI archive reader (though some work in that direction was at least discussed).

---

Or is this some new format different from docker-archive:?

[SquidDev]

Does loading this as a docker-archive: work?

Yes, podman pull docker-image:image2.tar correctly imports it. It looks like it's some hybrid which is both OCI and Docker compatible, hence the ambiguity.

Or, more correctly, I think Podman should let the caller specify the input format

A flag to specify which input type the file is would be perfect.

It occurs to me that I could get away with podman pull + podman tag. As long as the image is imported, and I have some way to tag it I'm not worried, especially given buildkit is doing weird things you may not want to have to work around.

[vrothberg]

I am surprised (but can confirm) that docker-archive loads it. Mostly because docker-archive usually does not use a ./blobs dir.

[mtrmac]

The new version of docker-archive is manifest-driven, manifest.json contains paths to other files, rather than relying on conventions.

[vrothberg]

@mtrmac, maybe we should massage oci-archive to return an error if there's a file or dir that doesn't comply with the OCI layout?

[mtrmac]

If such mixed-format archives exist in the wild, a failure seems rather undesirable: if both oci-archive and docker-archive had such a check, there wouldn’t be any way to load the archive at all.

---

To an extent this is a bug in the producer, using a different tag for the two formats; if that difference weren’t there, wouldn’t we be quite happy with loading the OCI format? And again, even if we did detect the inconsistency, just failing on that archive is unlikely to be what users want.

Actually

index.json set "org.opencontainers.image.ref.name":"latest"

vs.

--output type=docker,name=test,dest=../image.tar

isn’t this just a straightforward bug in buildkit that could be fixed much easier than Podman worrying about heuristics to choose one of two conflicting values? I don’t know anything about buildkit at all.

[vrothberg]

I concur with @mtrmac. Closing the issue but we continue the conversation.