Support Additional Layer Store

[ktock]

This is the subset of #281.

This commit adds "additional layer store" that enables lazy pulling on Podman/CRI-O. Instead of the snapshotter API, this directly exposes layers on the filesystem, following the required directory structure.

The command which provides such directory structure is stargz-store (cmd/stargz-store). This is implemented using fs/layer pkg of this project. This commit comes with scripts to test (script/cri-o) and benchmark (script/benchmark-podman) with Podman and CRI-O.

The following command mounts the store at <mountpoint>:

# stargz-store <mountpoint>

This store exposes the following structure required as an additional layer store:

<mountpoint>/base64(imageref)/<layerdigest>/
- diff  : exposes the extracted layer
- info  : contains JSON-formatted "Layer" structure of containers/storage
- use  : files to notify the use of this layer (used for management of reference count)

There are also other APIs for debugging:

<mountpoint>/base64(imageref)/
- manifest       : contains manifest of this image
- config         : contains config blob of this image

The following configuration is required on /etc/containers/storage.conf.
Here, /var/lib/stargz-store/store is an example of <mountpoint>.

[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"

[storage.options]
additionallayerstores = ["/var/lib/stargz-store/store:ref"]

• TODOs
  • add demo environment: we should have script/demo for additional layer store as well.
  • dedupe scripts: script/benchmark-containerd and script/benchmark-podman are very similar. will address in Switch from shellscripts-based CI to Go-based CI #310
  • add docs: will address in following up PRs

[ktock]

containers/storage#795 and containers/image#1109 are merged to Podman/CRI-O so eStargz-based lazy pulling is now available on them without patches.

Please note that stargz-store needs to run as a separete process and there are some limitations as of now (lazily pulled images can't be save/push-ed, runtime cannot verify chunks).