Image import #315

This implements one half of an image cache -- creating a filesystem from the cache, given an image name. (The other half is to download and put the files into the cache in the first place.) In the process, I moved the overlay filesystem to its own package.

Rough-and-ready code for downloading images into the cache. Manifests referred to by digest get written out; otherwise, they get tagged.

- factor out the symlink of tag manifests - factor out writing a layer - write each layer to a tmp dir, then move it, so failures don't leave partial layers - test that the cached image can be accessed using its digest, without using the image registry

This gives the vm (and therefore any command that runs the VM) a flag `--lib`, which will cache and use the mentioned image in the module search path. At present the module search path only uses a FileResolver; that is, it expects modules in a flat structure. Ths involves a bit of extra test scaffolding, so that invocations of `jk` when testing can use `--lib` (and `--cache` to set the cache to a fresh, temporary directory). In particular, an image registry is started, and images from tarballs uploaded to it. This is described in `tests/testfiles/README-images.md`.

In the OCI image format, the layer filesystems can contain "whiteout" files (think liquid paper or tipp-ex). This suppress any like-named file in the layers below. https://github.com/opencontainers/image-spec/blob/master/layer.md#whiteouts This commit accounts for whiteouts in the overlay VFS when opening a file or enumerating files in a directory. It's a fairly naive implementation, which checks each possible whiteout file, when traversing layers (since whiteouts hide things in the _next_ layer).

This is important because images may contain symlinks to save repeating files. A notable example is alpine, in which everything in /bin/ is a symlink to the busybox binary.

This adds tests for present and missing files in a library image that contains whiteout files. The test surfaced a mistake: the opaque whiteout file name is `.wh..wh..opq`, not `.wh..wh.opq`. Owps.

Instead of expecting modules to be dumped in the root directory, it's friendlier to give jk a "namespace" by putting things in a subdirectory. For modules, I have chosen `/jk/modules`. But to make sure we look for files in the right location (and not outside it!), we have to do the same thing as we do for modules in the "real" filesystem, and treat the chosen directory as the root of the image filesystem. This requires an extra layer, to "chroot" in the image filesystem.

It's much easier to debug if you get the full path (including the image name) for an import candidate, when an import fails.

Starting with foolib, which is the easy one.

Phew! This took some figuring out. Details in README-images.md.

This needs a "plugin" to pflag, which I've put in `pkg/cli`.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image import #315

Image import #315

Commits on Mar 31, 2020