New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerfile: add run --mount support #442

Merged
merged 3 commits into from Jun 8, 2018

Conversation

Projects
None yet
9 participants
@tonistiigi
Copy link
Member

tonistiigi commented Jun 7, 2018

This implements moby/moby#32507 (missing tmpfs support atm) as an experimental feature of Dockerfile frontend. The main dockerfile.v0 frontend is unaffected.

It can be used by building the frontend with a dfrunmount build tag and then loading it with either the gateway frontend or syntax directive (or gateway-devel to run from source on development).

The keys from the --mount options are taken from the ones supported by docker run. I'm not sure if we want to keep the ambiguous ones.

I also ran into a case where a scratch mount would error if binded. This was explicitly validated with a test but I don't think that is required. Even though there is no use case for this we can just create an empty temporary readonly directory instead of failing the build. @ijc

example:1

# syntax = tonistiigi/dockerfile:runmount20180607
from busybox
run --mount=target=/foo cat /foo/Dockerfile

example2:
https://gist.github.com/tonistiigi/0e0ab30ebf0eb6e4b82e1786d8b4dda1#file-test-dockerfile-L28
repeated builds reuse go cache

Note: The examples do not require updating to this PR. They run in master buildkit or moby integration PR.

Follow-up: need to figure out how to run tests with these features. Currently only checks that it builds.

@AkihiroSuda @tiborvass @thaJeztah @vdemeester

tonistiigi added some commits Jun 6, 2018

dockerfile: add run mount parsing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
dockerfile: run -—mount initial support
Supports binds from images and context and cache mounts.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
llbsolver: don’t error on readonly scratch mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

@tonistiigi tonistiigi force-pushed the tonistiigi:runmount branch from a5c0d32 to 479419a Jun 7, 2018

mounts := instructions.GetMounts(c)

for i, mount := range mounts {
if mount.From == "" && mount.Type == "cache" {

This comment has been minimized.

@vdemeester

vdemeester Jun 8, 2018

Member

Is cache a mount type support by moby ? I guess it's for directory like .m2, … so 👍

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Jun 8, 2018

How Go is configured to use /root/.cache?

case "type":
allowedTypes := map[string]struct{}{
"cache": {},
"bind": {},

This comment has been minimized.

@AkihiroSuda

AkihiroSuda Jun 8, 2018

Member

Enabling bind-mount by default may cause security issue?
Maybe better to require daemon-side configuration?

@tonistiigi

This comment has been minimized.

Copy link
Member

tonistiigi commented Jun 8, 2018

Is cache a mount type support by moby ? I guess it's for directory like

Enabling bind-mount by default may cause security issue?

As described in moby/moby#32507 these are custom mount types with builder specific meaning. Cache is builder specific but of course, it works with moby as well (with no updates required for the current integration branch). The bind (default) is bind between stages or build context not a directory on the host.

How Go is configured to use /root/.cache?

Default go env GOCACHE is /root/.cache/go-build

@AkihiroSuda AkihiroSuda merged commit 60344aa into moby:master Jun 8, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bryanhuntesl

This comment has been minimized.

Copy link

bryanhuntesl commented Jun 13, 2018

OMG - does this finally mean I'll be able to mount a SSH socket file during docker build ?

@tonistiigi

This comment has been minimized.

Copy link
Member

tonistiigi commented Jun 18, 2018

moby/moby#32507 defines mounts from context, stages and images, tmpfs mounts and persistent cache mounts that can be used for application specific cache. ssh sharing is covered in #262

@voltechs

This comment has been minimized.

Copy link

voltechs commented Sep 19, 2018

So... this was merged 4 months ago. I followed about 16 (closed) pull requests and 23 issues all referencing each other like a game of "whodunnit" in 2nd grade, finally arriving here.

I switched to edge (Docker version 18.09.0-ce-beta1, build 78a6bdb) but was confronted with Error response from daemon: Dockerfile parse error line 111: Unknown flag: mount. Also, I see no reference of this mysterious version that I seem to be on... I'm sure I did something wrong.

Generally specking though... Is this out yet?

Edit: Fun fact. When I check for updates in Docker for Mac, I get Docker 2.0.0.0-beta1-mac75 is currently the newest version available. even though docker --version returns 18.09.0 etc 🤔
Edit2: Ok, that's the version of Docker for Mac, which is different from the docker engine. 🙄

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Sep 19, 2018

@voltechs you need to enable buildkit as a builder, and use the syntax directive (as --mount is still a preview/custom feature, and hasn't been included in the official Dockerfile syntax yet);

mkdir foo && cd foo

cat <<EOF > Dockerfile
# syntax = tonistiigi/dockerfile:runmount20180607
FROM busybox
RUN --mount=target=/foo cat /foo/Dockerfile
EOF

DOCKER_BUILDKIT=1 docker build -t foo .
[+] Building 3.2s (11/11) FINISHED                                                                                                                                                                               
 => [internal] load Dockerfile                                                                                                                                                                              0.0s
 => => transferring dockerfile: 144B                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => resolve image config for docker.io/tonistiigi/dockerfile:runmount20180607                                                                                                                               0.6s
 => CACHED docker-image://docker.io/tonistiigi/dockerfile:runmount20180607@sha256:85308b90e426508ed13b338b091a4f159b45c2efdbfdabe419477f92d256f744                                                          0.0s
 => local://dockerfile (Dockerfile)                                                                                                                                                                         0.0s
 => local://context (.dockerignore)                                                                                                                                                                         0.0s
 => resolve image config for docker.io/library/busybox:latest                                                                                                                                               0.0s
 => local://context                                                                                                                                                                                         0.0s
 => => transferring context: 144B                                                                                                                                                                           0.0s
 => CACHED docker-image://docker.io/library/busybox:latest                                                                                                                                                  0.0s
 => /bin/sh -c cat /foo/Dockerfile                                                                                                                                                                          1.3s
 => exporting to image                                                                                                                                                                                      0.0s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:a2f52971817e7f39fa45e54f35235d48f95d2263a63dc8327ff614d8419d504b                                                                                                                0.0s
 => => naming to docker.io/library/foo                                                                                                                                                                      0.0s
@peter-edge

This comment has been minimized.

Copy link

peter-edge commented Oct 1, 2018

I've read through #262 and I've read through this, and I still can't figure out how to do ssh sharing, I'm on 18.09 edge as well - since volume isn't a supported mount type, you can't do e.g. https://github.com/avsm/docker-ssh-agent-forward. Is there an example somewhere?

@simonvanderveldt

This comment has been minimized.

Copy link

simonvanderveldt commented Jan 14, 2019

@tonistiigi just to check is it still possible to use this to mount a location on the host's filesystem with the current code on master? So something like this:

# syntax = tonistiigi/dockerfile:runmount20180607
FROM busybox
RUN --mount=target=/home/me/somefiles cat /foo/Dockerfile
EOF

DOCKER_BUILDKIT=1 docker build -t foo .

Looking at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#experimental-syntaxes it seems to suggest it doesn't work with host volume mounts which was something that this PR did seem to implement.

Also it seems like it's not possible to choose where to mount it to, is that correct?

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Jan 14, 2019

Host-mount is unlikely to be implemented.

Why do you need to mount the host filesystem?

@rcjsuen

This comment has been minimized.

Copy link

rcjsuen commented Jan 14, 2019

When will --mount be available for use in regular Docker?

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 15, 2019

@rcjsuen you mean without the # syntax in the Dockerfile, or you mean with docker build?

You can use it already with docker build if you have DOCKER_BUILDKIT=1 set as environment variable, and use a Dockerfile with the # syntax header to use the experimental Dockerfile syntax

@rcjsuen

This comment has been minimized.

Copy link

rcjsuen commented Jan 15, 2019

Hi, @thaJeztah. Thanks for getting back to me.

I meant as an official feature of Docker without the need of the special # syntax header.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment