ZIP support

[frafra]

Feature Request

Support zip compression.

Desired Feature

Configuration Specification v2.3.0 says that a file content can be compressed or not.

compression (string): the type of compression used on the contents (null or gzip). Compression cannot be used with S3.

It would be nice to support zip too, as it is quite common.

[lucab]

Thanks for the report.
However zip is an archive format (i.e. akin to tar), while the field you reference is only meant for compression algorithms and it does not support picking a specific file from within an archive (on purpose).
From the point of view of compression, to the best of my knowledge zip archives are commonly compressed with the DEFLATE algorithm which is also the one used by gzip.

[ajeddeloh]

I'm not sure how I feel about archive support. DEFLATE support is fine by me if it's useful to people. Archive support can be used as a "poor-man's-package-manager" which isn't ideal, but also has legitimate uses. Then there's the question of which formats to support. .tar arguably makes more sense as a higher priority for a Linux tool, but there's also many flavors of tar which makes it more complicated.

Is there a particular use case you had in mind?

[bgilbert]

To be clear, gzip is DEFLATE plus headers, so we wouldn't support DEFLATE standalone.

There also isn't a universal interchange tar format or zip format; there are different flavors. Zip in particular is underspecified and has redundant headers (central directory and individual file headers), so there's plenty of room for incompatibilities. I've seen multiple writer/reader pairs where a particular reader can't consume the output of a particular writer.

[Snaipe]

I'd be interested in this (though more generally speaking, supporting directory creation from archive types). The reason why is that it makes it incredibly easier to provision things like CNI plugins directly from release tarballs (e.g. https://github.com/containernetworking/plugins/releases/). Right now the alternative is to have automation (terraform) download the archive, extract it locally, then populate the entire tree inline, then use the result; or deploy some service in the middle that provides the extracted files individually and populate the tree from their URLs. This is incredibly tedious, and wastes resources generating a large json file that then needs to ultimately be uploaded, fetched, and parsed. That said, seeing #541, I'm not exactly sure I should be working on this if it's going to end up rejected.

@lucab what would it take, if I were to implement this, to get it accepted/merged?

My general plan for this would be:

1. add an optional contents section for directories, which would have the same keys as in the files section, but with an extra mandatory archive key to not have to do any sort of filetype detection. e.g:

{
  "directories": [
    {
      "path": "/foo",
      "contents": {
        "archive": "tar"
        "source": "http://example.com/archive.tar",
        "verification": {
          "hash": "sha256-98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4"
        }
      }
    },
    {
      "path": "/bar",
      "contents": {
        "archive": "tar"
        "compression": "gzip"
        "source": "http://example.com/archive.tar.gz"
      }
    },
    {
      "path": "/baz",
      "contents": {
        "archive": "zip"
        "compression": "deflate"
        "source": "http://example.com/archive.zip"
      }
    }
  ]
}

2. Use archive/tar (and archive/zip) to extract files to their destination, using openat2 with RESOLVE_IN_ROOT|RESOLVE_NO_SYMLINKS|RESOLVE_NO_MAGICLINKS (chroot is not sufficient if /proc is available). This avoids the issues around extracting random content to destinations with symlinks, and given that most entries to the "directories" section is for creating initially empty dirs, I think it makes sense to just never follow symlinks and error out. We could even require "overwrite": true to avoid people cutting themselves on a sharp edge if the tarball itself contains symlinks and the archive gets applied again.

Note that I'm mainly interested in tarballs, so if zip is causing issues with its format as noted in earlier comments, I'm happy to just drop it for now, and just do tar.

[travier]

I've had the need to get binaries into FCOS instances before and another option I used was to create a container image with the content that I need and then to use podman to fetch and copy the content on first boot: https://github.com/travier/fedora-coreos-nomad/blob/main/common.bu#L31.

[Snaipe]

Right; as far as alternatives go the simplest way to pull and expand a tarball would be to use curl and tar in a systemd oneshot service, though this has the same caveats around symlinks. I ended up going that route (which is simpler than expanding each file in the ignition config), but it still feels out of place.