podman save always compresses layers in OCI-formatted archives

[vlk-charles]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Probably.

Description

Layers in archives produced by podman save --format docker-archive (default) are saved as uncompressed tarballs. Archives produced by podman save --format oci-archive have the layers gzip-compressed. As documented in man podman-save, the --compress flag is not applicable for archives, only dirs.

Steps to reproduce the issue:

# podman save hello-world | tar -tv
-r--r--r-- 0/0           15360 1970-01-01 01:00 f22b99068db93900abe17f7f5e09ec775c2826ecfe9db961fea68293744144bd.tar
-r--r--r-- 0/0            1470 1970-01-01 01:00 d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726.json
l--------- 0/0               0 1970-01-01 01:00 a803ca04f00ecd1c10fb4978f7a963d1461f4225e568901bc6990cb46a3e04be/layer.tar -> ../f22b99068db93900abe17f7f5e09ec775c2826ecfe9db961fea68293744144bd.tar
-r--r--r-- 0/0               3 1970-01-01 01:00 a803ca04f00ecd1c10fb4978f7a963d1461f4225e568901bc6990cb46a3e04be/VERSION
-r--r--r-- 0/0            1142 1970-01-01 01:00 a803ca04f00ecd1c10fb4978f7a963d1461f4225e568901bc6990cb46a3e04be/json
-r--r--r-- 0/0             210 1970-01-01 01:00 manifest.json
-r--r--r-- 0/0             103 1970-01-01 01:00 repositories
# podman save --format oci-archive hello-world | tar -tv
drwxr-xr-x root/root         0 2021-09-16 18:32 blobs/
drwxr-xr-x root/root         0 2021-09-16 18:32 blobs/sha256/
-rw-r--r-- root/root       584 2021-09-16 18:32 blobs/sha256/1189c90721f97a81243e82b81a28b4b47876d5bd0fe3b48194261744b70bfc17
-rw-r--r-- root/root       345 2021-09-16 18:32 blobs/sha256/2cf6251bbf296029043a4ba3e1caaf7dd22825c554512621b77c66cff366cd5e
-rw-r--r-- root/root      2607 2021-09-16 18:32 blobs/sha256/63050ae8454a6515291dd4b87f793ec8106ca6054ff0350475dcead380ffeef7
-rw-r--r-- root/root       269 2021-09-16 18:32 index.json
-rw-r--r-- root/root        31 2021-09-16 18:32 oci-layout
# podman save hello-world | tar -xO f22b99068db93900abe17f7f5e09ec775c2826ecfe9db961fea68293744144bd.tar | sha256sum 
f22b99068db93900abe17f7f5e09ec775c2826ecfe9db961fea68293744144bd  -
# podman save --format oci-archive hello-world | tar -xO blobs/sha256/63050ae8454a6515291dd4b87f793ec8106ca6054ff0350475dcead380ffeef7 | gzip -d | sha256sum 
f22b99068db93900abe17f7f5e09ec775c2826ecfe9db961fea68293744144bd  -

Describe the results you received:
OCI archives are smaller. But further compression applied over the whole archive (podman save --format oci-archive hello-world | xz) is not as efficient.

Describe the results you expected:
I would expect there to be a way to produce uncompressed OCI archives.

Output of podman version:

Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 22:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 322105344
  memTotal: 2061930496
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 405h 38m 21.44s (Approximately 16.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 28
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 22:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.3.1-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes.

[mheon]

@vrothberg PTAL

[vlk-charles]

podman save always uses the following media type:

# podman save --format oci-archive hello-world | tar -xO "blobs/$(podman save --format oci-archive hello-world | tar -xO index.json | jq -r .manifests[0].digest | tr : /)" | jq -r .layers[0].mediaType
application/vnd.oci.image.layer.v1.tar+gzip

A type without +gzip and others are defined as well in the specification.

[rhatdan]

@mtrmac PTAL

[mtrmac]

c/image allows that as types.SystemContext.OCIAcceptUncompressedLayers; Podman could expose it as a CLI option.

[rhatdan]

Now that zstd is merged into the Moby Project, we should probably support this as well.

[vlk-charles]

I see a new --uncompressed option was implemented. Was adapting the existing --compress option not feasible?

[vrothberg]

I see a new --uncompressed option was implemented. Was adapting the existing --compress option not feasible?

No, that would have changed the default behavior and potentially broke existing users who desire to have compressed layers.