Europa: new docker package

[shykes]

Overview

The new docker package, to be introduced in the Europa release at universe.dagger.io/docker, will be the primary API for using Docker containers, and as such will play an important role in our overall developer experience. Its overall design is complete, but implementation is not. No doubt there will be some last minute questions on the design along the way, too.

This issue tracks remaining work to call the docker package complete.

Use cases

Below is a list of use cases for Docker containers in Dagger, the relevant package, and progress status.

Use case	Package	Design status	Tests status
Build images with a Dockerfile	docker	🚧	❌
Build images with CUE	docker	🚧	❌
Push and pull images	docker	✅	❌
Run ephemeral containers as part of a CICD pipeline (with or without external engine)	docker	🚧	❌
Run Docker Compose commands (with or without external engine)	docker/compose	❌	❌
Run persistent containers on an external Docker engine	docker/client	❌	❌
Run arbitrary commands on an external Docker engine	docker/client	❌	❌

High-level design

docker

The purpose of universe.dagger.io/docker is to offer a modern API for the most common uses of Docker containers in the context of CICD:

• Build images, with or without Dockerfiles
• Push and pull images
• Run ephemeral containers for the duration of the pipeline

An external Docker engine is not required by default, but it should be supported as an opt-in whenever useful and practical. When supporting an external engine for an operation, remember that full Docker engine API compatibility is not a goal: there is a separate package, docker/client for that purpose.

docker/client

The purpose of universe.dagger.io/docker/client is to implement a client to the Docker engine API with maximum compatibility, and maximum familiarity for users of the docker CLI.

docker/compose

The purpose of universe.dagger.io/docker/compose is to support running a Docker Compose project, as well as other common Docker Compose operations.

[shykes]

@aluzzardi should this go in “next” since it fits in the “finish porting universe” theme?

[TomChv]

It's maybe a silly question but what do you mean by external Docker engine?

Since Dagger need a docker engine to run buildkit daemon in it so how could we run ephemeral containers without docker engine? Is it about tools like podman who do not require daemons to run containers?

[shykes]

It's maybe a silly question but what do you mean by external Docker engine?

Running containers “on an external Docker engine” means “without relying on the built-in engine API”. It means that the end user must provide access to a Docker engine API endpoint, and Dagger will run a client container to connect to that external endpoint, just like any external service.

Since Dagger need a docker engine to run buildkit daemon in it so how could we run ephemeral containers without docker engine? Is it about tools like podman who do not require daemons to run containers?

Dagger doesn’t actually need a Docker engine. It needs a functioning buildkit daemon.

• The buildkit daemon may of course run on Docker, but it may not.
• The buildkit daemon may use a Docker engine as an execution backend, but it may not. For example it could be using a naked containerd daemon, or shell out to runc, or a fancy integration with kubernetes.

[shykes]

The docker package is not feature-complete, but it is functional, and most urgently needed improvements are tracked in #1466 and #1650 . If needed, feel free to open new issues for some of the unfinished items in the table above.