Support loading artifacts into a Docker Engine

[aluzzardi]

This adds support to loading artifacts (e.g. docker.#Build, os.#Container, ...) into any arbitrary docker engine (through a
dagger.#Stream for UNIX sockets or SSH for a remote engine).

Serves as basis for supporting arbitrary artifacts in #593.

Implementation:

• Add op.#SaveImage which serializes an artifact into an arbitrary path
  (docker tarball format)
• Add docker.#Load which uses op.#SaveImage to serialize to disk and
  executes docker load to load it back

Caveats: Because we're doing this in userspace rather than letting
dagger itself load the image, the performance is pretty bad.

The buildkit API is meant for streaming (get a stream of a docker image
pipe it into docker load). Because of userspace, we have to load the
entire docker image into memory, then serialize it in a single WriteFile
LLB operation.

Example:

package main

import (
	"alpha.dagger.io/dagger"
	"alpha.dagger.io/docker"
)

source: dagger.#Input & dagger.#Artifact

dockersocket: dagger.#Input & dagger.#Stream

build: docker.#Build & {
	"source": source
}

load: docker.#Load & {
	source: build
	tag:    "testimage"
	socket: dockersocket
}

Signed-off-by: Andrea Luzzardi aluzzardi@gmail.com

[netlify]

✔️ Deploy Preview for devel-docs-dagger-io ready!

🔨 Explore the source changes: 322997b

🔍 Inspect the deploy log: https://app.netlify.com/sites/devel-docs-dagger-io/deploys/6166240a6aa137000814c452

😎 Browse the preview: https://deploy-preview-1056--devel-docs-dagger-io.netlify.app

[shykes]

Wouldn’t it be more efficient to save the image to a local directory in the buildkit state? Then it can be injected into a docker engine in userland (using docker load and/or docker import).

[aluzzardi]

That's what this PR does.

But to save the image to local state, dagger has to receive it from buildkit and write it back to a directory in the state.

There's no LLB operation to do this, the image is streamed using the gRPC API and then written back in a LLB WriteFile op.

[shykes]

That's what this PR does.

But to save the image to local state, dagger has to receive it from buildkit and write it back to a directory in the state.

There's no LLB operation to do this, the image is streamed using the gRPC API and then written back in a LLB WriteFile op.

I see. op.#DockerBuild does write the image to local state, but only the unpacked filesystem contents, so other metadata would be lost.

Maybe the docker2llb frontend could be modified or extended to write not just the unpacked contents, but the whole image as well? Patching a frontend would be better than patching buildkit itself..

[aluzzardi]

I see. op.#DockerBuild does write the image to local state, but only the unpacked filesystem contents, so other metadata would be lost.

It only translates a Dockerfile to LLB instructions for buildkit, it does not write the unpacked contents. The only way to extract content from buildkit is through the gRPC API, either by pushing it somewhere (e.g. registry) or streaming it back (what this PR does).

This PR is agnostic to docker.#Build, doing a docker.#Load & { source: os.#Container & { image: alpine } } would work just as well.

[shykes]

It only translates a Dockerfile to LLB instructions for buildkit, it does not write the unpacked contents. The only way to extract content from buildkit is through the gRPC API, either by pushing it somewhere (e.g. registry) or streaming it back (what this PR does).

I can copy or mount the output of docker.#Build just like any other llb state. That gives me the unpacked contents of the image but not its metadata.

This PR is agnostic to docker.#Build, doing a docker.#Load & { source: os.#Container & { image: alpine } } would work just as well.

Good point.