Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerfile: add run --mount support #442

Merged
merged 3 commits into from Jun 8, 2018
Merged

Conversation

@tonistiigi
Copy link
Member

@tonistiigi tonistiigi commented Jun 7, 2018

Update: this feature was added to the experimental Dockerfile syntax

Read more about experimental Dockerfile features in the experimental.md file


This implements moby/moby#32507 (missing tmpfs support atm) as an experimental feature of Dockerfile frontend. The main dockerfile.v0 frontend is unaffected.

It can be used by building the frontend with a dfrunmount build tag and then loading it with either the gateway frontend or syntax directive (or gateway-devel to run from source on development).

The keys from the --mount options are taken from the ones supported by docker run. I'm not sure if we want to keep the ambiguous ones.

I also ran into a case where a scratch mount would error if binded. This was explicitly validated with a test but I don't think that is required. Even though there is no use case for this we can just create an empty temporary readonly directory instead of failing the build. @ijc

example:1

# syntax=docker/dockerfile:1.0-experimental
FROM busybox
RUN --mount=target=/foo cat /foo/Dockerfile

example2:
https://gist.github.com/tonistiigi/0e0ab30ebf0eb6e4b82e1786d8b4dda1#file-test-dockerfile-L28
repeated builds reuse go cache

Note: The examples do not require updating to this PR. They run in master buildkit or moby integration PR.

Follow-up: need to figure out how to run tests with these features. Currently only checks that it builds.

@AkihiroSuda @tiborvass @thaJeztah @vdemeester

tonistiigi added 3 commits Jun 6, 2018
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Supports binds from images and context and cache mounts.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@tonistiigi tonistiigi force-pushed the tonistiigi:runmount branch from a5c0d32 to 479419a Jun 7, 2018
mounts := instructions.GetMounts(c)

for i, mount := range mounts {
if mount.From == "" && mount.Type == "cache" {

This comment has been minimized.

@vdemeester

vdemeester Jun 8, 2018
Member

Is cache a mount type support by moby ? I guess it's for directory like .m2, … so 👍

@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Jun 8, 2018

How Go is configured to use /root/.cache?

case "type":
allowedTypes := map[string]struct{}{
"cache": {},
"bind": {},

This comment has been minimized.

@AkihiroSuda

AkihiroSuda Jun 8, 2018
Member

Enabling bind-mount by default may cause security issue?
Maybe better to require daemon-side configuration?

@tonistiigi
Copy link
Member Author

@tonistiigi tonistiigi commented Jun 8, 2018

Is cache a mount type support by moby ? I guess it's for directory like

Enabling bind-mount by default may cause security issue?

As described in moby/moby#32507 these are custom mount types with builder specific meaning. Cache is builder specific but of course, it works with moby as well (with no updates required for the current integration branch). The bind (default) is bind between stages or build context not a directory on the host.

How Go is configured to use /root/.cache?

Default go env GOCACHE is /root/.cache/go-build

@AkihiroSuda AkihiroSuda merged commit 60344aa into moby:master Jun 8, 2018
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bryanhuntesl
Copy link

@bryanhuntesl bryanhuntesl commented Jun 13, 2018

OMG - does this finally mean I'll be able to mount a SSH socket file during docker build ?

@tonistiigi
Copy link
Member Author

@tonistiigi tonistiigi commented Jun 18, 2018

moby/moby#32507 defines mounts from context, stages and images, tmpfs mounts and persistent cache mounts that can be used for application specific cache. ssh sharing is covered in #262

@voltechs
Copy link

@voltechs voltechs commented Sep 19, 2018

So... this was merged 4 months ago. I followed about 16 (closed) pull requests and 23 issues all referencing each other like a game of "whodunnit" in 2nd grade, finally arriving here.

I switched to edge (Docker version 18.09.0-ce-beta1, build 78a6bdb) but was confronted with Error response from daemon: Dockerfile parse error line 111: Unknown flag: mount. Also, I see no reference of this mysterious version that I seem to be on... I'm sure I did something wrong.

Generally specking though... Is this out yet?

Edit: Fun fact. When I check for updates in Docker for Mac, I get Docker 2.0.0.0-beta1-mac75 is currently the newest version available. even though docker --version returns 18.09.0 etc 🤔
Edit2: Ok, that's the version of Docker for Mac, which is different from the docker engine. 🙄

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 19, 2018

@voltechs you need to enable buildkit as a builder, and use the syntax directive (as --mount is still a preview/custom feature, and hasn't been included in the official Dockerfile syntax yet);

mkdir foo && cd foo

cat <<EOF > Dockerfile
# syntax=docker/dockerfile:1.0-experimental
FROM busybox
RUN --mount=target=/foo cat /foo/Dockerfile
EOF

DOCKER_BUILDKIT=1 docker build -t foo .
[+] Building 3.2s (11/11) FINISHED                                                                                                                                                                               
 => [internal] load Dockerfile                                                                                                                                                                              0.0s
 => => transferring dockerfile: 144B                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => resolve image config for docker.io/docker/dockerfile:1.0-experimental                                                                                                                              0.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1.0-experimental@sha256:85308b90e426508ed13b338b091a4f159b45c2efdbfdabe419477f92d256f744                                                          0.0s
 => local://dockerfile (Dockerfile)                                                                                                                                                                         0.0s
 => local://context (.dockerignore)                                                                                                                                                                         0.0s
 => resolve image config for docker.io/library/busybox:latest                                                                                                                                               0.0s
 => local://context                                                                                                                                                                                         0.0s
 => => transferring context: 144B                                                                                                                                                                           0.0s
 => CACHED docker-image://docker.io/library/busybox:latest                                                                                                                                                  0.0s
 => /bin/sh -c cat /foo/Dockerfile                                                                                                                                                                          1.3s
 => exporting to image                                                                                                                                                                                      0.0s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:a2f52971817e7f39fa45e54f35235d48f95d2263a63dc8327ff614d8419d504b                                                                                                                0.0s
 => => naming to docker.io/library/foo                                                                                                                                                                      0.0s
@bufdev
Copy link

@bufdev bufdev commented Oct 1, 2018

I've read through #262 and I've read through this, and I still can't figure out how to do ssh sharing, I'm on 18.09 edge as well - since volume isn't a supported mount type, you can't do e.g. https://github.com/avsm/docker-ssh-agent-forward. Is there an example somewhere?

@simonvanderveldt
Copy link

@simonvanderveldt simonvanderveldt commented Jan 14, 2019

@tonistiigi just to check is it still possible to use this to mount a location on the host's filesystem with the current code on master? So something like this:

# syntax=docker/dockerfile:1.0-experimental
FROM busybox
RUN --mount=target=/home/me/somefiles cat /foo/Dockerfile
EOF

DOCKER_BUILDKIT=1 docker build -t foo .

Looking at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#experimental-syntaxes it seems to suggest it doesn't work with host volume mounts which was something that this PR did seem to implement.

Also it seems like it's not possible to choose where to mount it to, is that correct?

@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Jan 14, 2019

Host-mount is unlikely to be implemented.

Why do you need to mount the host filesystem?

@rcjsuen
Copy link

@rcjsuen rcjsuen commented Jan 14, 2019

When will --mount be available for use in regular Docker?

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 15, 2019

@rcjsuen you mean without the # syntax in the Dockerfile, or you mean with docker build?

You can use it already with docker build if you have DOCKER_BUILDKIT=1 set as environment variable, and use a Dockerfile with the # syntax header to use the experimental Dockerfile syntax

@rcjsuen
Copy link

@rcjsuen rcjsuen commented Jan 15, 2019

Hi, @thaJeztah. Thanks for getting back to me.

I meant as an official feature of Docker without the need of the special # syntax header.

@andig
Copy link

@andig andig commented Feb 28, 2019

@thaJeztah I'm still unable to get this to work beyond your example (which is working). I have

# syntax = tonistiigi/dockerfile:runmount20180607
FROM golang:alpine as builder
RUN --mount=target=/root/.cache,type=cache go build -ldflags="-w -s" -a -installsuffix cgo -o /go/bin/ingress github.com/andig/ingress/cmd/ingress

and it still errors with Unknown flag: mount:

❯ DOCKER_BUILDKIT=1 docker build -t ingress:lastest .
[+] Building 0.1s (2/2) FINISHED
=> [internal] load build definition from Dockerfile                                                                                                    0.1s
=> => transferring dockerfile: 1.50kB                                                                                                                  0.0s
=> [internal] load .dockerignore                                                                                                                       0.1s
=> => transferring context: 2B                                                                                                                         0.0s
failed to create LLB definition: Dockerfile parse error line 6: Unknown flag: mount
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Feb 28, 2019

@andig that's odd; what version of docker are you running? I just tried your exact example, and it works (obviously it fails building because the source isn't there, but it doesn't fail on the --mount option);

DOCKER_BUILDKIT=1 docker build -<<'EOF'
# syntax=docker/dockerfile:1.0-experimental
FROM golang:alpine as builder
RUN --mount=target=/root/.cache,type=cache go build -ldflags="-w -s" -a -installsuffix cgo -o /go/bin/ingress github.com/andig/ingress/cmd/ingress
EOF
 => ERROR /bin/sh -c go build -ldflags="-w -s" -a -installsuffix cgo -o /go/bin/ingress github.com/andig/ingress/cmd/ingress                                                                             1.4s
------
 > /bin/sh -c go build -ldflags="-w -s" -a -installsuffix cgo -o /go/bin/ingress github.com/andig/ingress/cmd/ingress:
#8 1.174 can't load package: package github.com/andig/ingress/cmd/ingress: cannot find package "github.com/andig/ingress/cmd/ingress" in any of:
#8 1.174 	/usr/local/go/src/github.com/andig/ingress/cmd/ingress (from $GOROOT)
#8 1.174 	/go/src/github.com/andig/ingress/cmd/ingress (from $GOPATH)
------
exit code: 2
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Feb 28, 2019

Also note that the tonistiigi/dockerfile:runmount20180607 frontend is quite outdated by now, and there's now official "experimental" frontends (https://hub.docker.com/r/docker/dockerfile/tags);

#syntax=<see below>:

  • docker/dockerfile:1-experimental (latest 1.x.x experimental syntax; currently 1.0.2)
  • docker/dockerfile:1.0-experimental (latest 1.0.x experimental syntax; currently 1.0.2)
  • docker/dockerfile:1.0.2-experimental (1.0.2 experimental syntax)
  • docker/dockerfile:experimental (latest experimental; currently 1.0.2)

I'll update the examples in this thread

@andig
Copy link

@andig andig commented Feb 28, 2019

Thanks for the fast reply and the updates. For sake of other readers here's why it failed for me- additional comments above the # syntax don't work:

❯ DOCKER_BUILDKIT=1 docker build -<<'EOF'
############################
# STEP 1 build executable binary
############################
# syntax=docker/dockerfile:1.0-experimental
FROM golang:alpine as builder
RUN --mount=target=/root/.cache,type=cache go build -ldflags="-w -s" -a -installsuffix cgo -o /go/bin/ingress github.com/andig/ingress/cmd/ingress
EOF
[+] Building 0.1s (2/2) FINISHED
=> [internal] load build definition from Dockerfile                                                                                                    0.0s
=> => transferring dockerfile: 356B                                                                                                                    0.0s
=> [internal] load .dockerignore                                                                                                                       0.0s
=> => transferring context: 2B                                                                                                                         0.0s
failed to create LLB definition: Dockerfile parse error line 6: Unknown flag: mount

Without those other comments its fine...

@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Feb 28, 2019

The # syntax = .. needs to be on the first line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants