Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: docker build read Dockerfile from stdin #27393

Closed
dsheets opened this issue Oct 14, 2016 · 41 comments · Fixed by #31236
Closed

Proposal: docker build read Dockerfile from stdin #27393

dsheets opened this issue Oct 14, 2016 · 41 comments · Fixed by #31236
Labels
area/builder kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny
Milestone

Comments

@dsheets
Copy link
Contributor

dsheets commented Oct 14, 2016

I propose that docker build accept -f - to indicate that the Dockerfile to be used is read from the STDIN of the docker client process.

Use cases

  • Shared engine CI systems can now stack anonymous image builds easily without requiring intermediate directories or tar archives
  • Dockerfiles in local directories, tar archives, and git repos can now be overridden
  • docker build is now a versatile shell scripting command as Dockerfiles can participate in shell pipelines decoupled from file system contexts

New capabilities

  • Uncooperative remote tar archives and git repos can now have out-of-band Dockerfiles applied to them
  • Local directory builds can now have Dockerfiles imported for just the build via I/O redirection

Design considerations

  • A - stdin reader interface already exists in the context path argument so any -f - implementation will have to accommodate that
  • Accepting - in place of file paths is a common UNIX idiom to indicate that what would have been read from the path specified should instead be read from stdin; this makes having docker participate in UNIX pipelines much easier
  • This feature could be abused to avoid Best Practice so all advice regarding application/system design should be maintained and users should be encouraged to place Dockerfiles in repositories and tarballs so builds are straightforward and unambiguous when possible

Cases

  • builder.GetContextFromReader (builder/context.go) is used to read a context (or Dockerfile only) from stdin: nothing will change, a file called - will still be sourced from the stdin tarball (or still ignored in the case of Dockerfile only on stdin)
  • builder.GetContextFromGitURL (builder/context.go) is used to read a context from a git URL and the resultant context will have its Dockerfile rewritten and used for the build
  • builder.GetContextFromURL (builder/context.go) is used to read a context from a remote URL and the resultant context will have its Dockerfile rewritten and used for the build (or ignored in the case of a Dockerfile only context)
  • builder.GetContextFromLocalDir (builder/context.go) is used to read a context from a local directory and the resultant context will have its Dockerfile rewritten and used for the build

Implementation

I've prototyped this functionality by rewriting the tar stream of the context before the trusted pull rewriting. If there is interest in this functionality, I will clean up my prototype and submit a PR. I welcome your ideas regarding this or similar features. Thanks!

@mlaventure mlaventure added the kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny label Oct 14, 2016
@thaJeztah
Copy link
Member

@dsheets perhaps I misunderstood, but docker build already allows building from stdin;

Note: If you build by passing a Dockerfile through STDIN (docker build - < somefile), there is no build context, so the Dockerfile can only contain a URL based ADD instruction. You can also pass a compressed archive through STDIN: (docker build - < archive.tar.gz), the Dockerfile at the root of the archive and the rest of the archive will get used at the context of the build.

Or is this to build using the local build context, but getting only the dockerfile from stdin?

@dsheets
Copy link
Contributor Author

dsheets commented Oct 15, 2016

@thaJeztah sorry I wasn't clearer. Your second explanation is correct. This proposal is to have the build context indicated by local path, git repository, or URL and the Dockerfile only to be read from stdin and supply the build instructions (or override existing build instructions) from the context.

The use case that prompted this proposal is the first in the above use case list -- stacking anonymous images in a shared engine CI environment. It is much easier and more natural to pipe around text than tar files and there are many common tools to source/modify/compile text compared to tar files without creating temporary files or directories.

@thaJeztah
Copy link
Member

@dsheets clear! Sounds reasonable to me (although the documentation may need a rewrite as all possible combinations become quite complex)

@justincormack
Copy link
Contributor

Actually I am not sure what the current behaviour is, is -f supported at all when using a git url at present?

@duglin
Copy link
Contributor

duglin commented Oct 17, 2016

I believe it should still work - as long as the -f points to a file in the context/git-repo. It should allow for you to build from a Dockerfile that’s not in the root.
-Doug

On Oct 16, 2016, at 9:03 PM, Justin Cormack notifications@github.com wrote:

Actually I am not sure what the current behaviour is, is -f supported at all when using a git url at present?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #27393 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AB2sX2f_SbTC6Yi2ydmIB5kailTz_Wziks5q0iCIgaJpZM4KXOEm.

@justincormack
Copy link
Contributor

I think this is a bit inconsistent, as I believe -f always just points to a name in the context, and -f - does not make sense like that.

I think it would be better to allow multiple contexts which are appended so you could do docker build git://... Dockerfile which would append the Dockerfile to the git bundle, overwriting any Dockerfile in it, and then you could also allow docker build git:// - as a file (vs tarball) on stdin is assumed to be a file called Dockerfile. I have other use cases for multiple appended bundles as a build context as well.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 18, 2016

For the local directory case, -f path will resolve path relative to the cwd of the docker client process which is inconsistent with the stdin, git, and URL cases. It is a widespread convention that path arguments can be - to read from stdin which is consistent with the local directory case (relative to the invocation).

Currently, a local context can only be a directory (not a tarball or a Dockerfile). This isn't currently consistent but the suggestion to stack contexts would have to expand that functionality as well. If you have use cases for stacked contexts, I recommend a separate proposal outlining them. I don't see any reason why the build subcommand shouldn't support both -f - and stacked contexts but the latter seems like a separate matter and a significantly larger extension of the CLI.

@thaJeztah
Copy link
Member

@justincormack -f allows picking Dockerfile from anywhere, including outside of the build-context, but it's treated as if it's at the root of the build-context

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

I think currently -f takes an arbitrary path and, when the context is a local directory, uses that path relative to the cwd of the docker client process but the Dockerfile path must be within the build context otherwise:

unable to prepare context: The Dockerfile (/Users/dsheets/Code/test/Dockerfile) must be within the build context (foo)

@duglin
Copy link
Contributor

duglin commented Oct 19, 2016

@thaJeztah's comment about being able to indirectly reference a Dockerfile from outside of the build context is a really interesting usecase - I'd love to hear about some usecase/need for that one.

My initial reaction to this is that it could lead to a bit of confusion since we would support "-" being an option of -f as well as in place of the build context location (last param). This isn't necessary an issue since we would need to restrict the user from specifying it in both places, but just something to think about.

Would still like time to think about this, but I'm leaning towards saying we should add this since it pretty consistent with other tools that support similar options and it could open up some interesting usecases. For example:
docker build -f /path/to/tomcat/Dockerfile .
where the referenced Dockerfile would act like a CloudFoundry buildpack - meaning it takes your app and packages it up w/o you really knowing what's going on - you just know what Dockerfile to use - and you don't need to modify your code/dir to do it. And if the /path/... was allowed to be a URL (e.g. http://docker.com/buildpacks/tomcat) I think that would be pretty neat.

@justincormack
Copy link
Contributor

I would like to be able to use a Docker file outside a tarball build
context, but I am not sure if this can be made backwards compatible...

On 19 Oct 2016 15:28, "Doug Davis" notifications@github.com wrote:

@thaJeztah https://github.com/thaJeztah's comment about being able to
indirectly reference a Dockerfile from outside of the build context is a
really interesting usecase - I'd love to hear about some usecase/need for
that one.

My initial reaction to this is that it could lead to a bit of confusion
since we would support "-" being an option of -f as well as in place of the
build context location (last param). This isn't necessary an issue since we
would need to restrict the user from specifying it in both places, but just
something to think about.

Would still like time to think about this, but I'm leaning towards saying
we should add this since it pretty consistent with other tools that support
similar options and it could open up some interesting usecases. For example:
docker build -f /path/to/tomcat/Dockerfile .
where the referenced Dockerfile would act like a CloudFoundry buildpack -
meaning it takes your app and packages it up w/o you really knowing what's
going on - you just know what Dockerfile to use - and you don't need to
modify your code/dir to do it. And if the /path/... was allowed to be a URL
(e.g. http://docker.com/buildpacks/tomcat) I think that would be pretty
neat.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27393 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPEgS5nQHgLEF6uuNEz02PYm7Byn_ks5q1ikZgaJpZM4KXOEm
.

@thaJeztah
Copy link
Member

You mean multiple -f flags, one for tar, one for Dockerfile?

@duglin
Copy link
Contributor

duglin commented Oct 19, 2016

@justincormack where do you see the incompatibility?

Actually, the more I think about this... @dsheets how do you see this working? Right now the Dockerfile is passed as part of the context, not as an independent entity to the REST API.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

@duglin your /path/to/tomcat/Dockerfile example is currently explicitly disallowed unless . is a subdirectory (or equal to /path/to/tomcat). To do this under my proposal, you could write docker build -f - . < /path/to/tomcat/Dockerfile. In my prototype patch, the build context tarball is rewritten to include the contents of stdin.

@justincormack using -f and the current local tarball build context via - it is not possible. If the local context path were made consistent with the other context cases, using a Dockerfile from outside of a tarball could be done with docker build -f - context.tar.gz < /path/to/Dockerfile which is backward compatible.

@thaJeztah Do you mean multiple positional arguments to the "context"? I don't know what multiple -f could mean.

@duglin
Copy link
Contributor

duglin commented Oct 19, 2016

@dsheets yea, I know its disallowed today but my point is that once you figure out how to support stdin then there's no reason to disallow it anymore - we can just treat any file outside of the build context as coming from "stdin". In your prototype, did you end up creating a Dockerfile in the context? Not sure you can do that w/o also creating a .dockerignore with Dockerfile in there, or overriding any existing one. This scares me because then you'd be modifying the user's files and we shouldn't do that since we have no idea what they might be doing with them.

@duglin
Copy link
Contributor

duglin commented Oct 19, 2016

I guess one more safe option, but a bit hacky, would be to put the Dockerfile contents into a well defined, but hidden and pretty unlikely to overlap with a user's, file - such as .dockerfile_which_shall_be_deleted :-) then we can pass this filename to the REST API as the -f option and the builder can ALWAYS delete it w/o it needing to be listed in the .dockerignore. As I said, a bit hacky but might work w/o needing to redesign how we send over the build info - which I know we may want to do one day anyway but probably not today.

@justincormack
Copy link
Contributor

@thaJeztah no, tar cf - *.c *.h | docker build -f Dockerfile - does not work, as @dsheets says it has to be inside the build context if the build context is not the filesystem.

@thaJeztah
Copy link
Member

@justincormack @duglin @dsheets I don't really see a reason why we would disallow a Dockerfile that 's outside of the build-context. As long as we treat the Dockerfile as if it's at the root of the context (so doesn't have access to files outside of it), I don't see why we would reject that.

So basically;

docker build -f /my/dockerfiles/Dockerfile /somewhere/else/build-context/

Sounds like something we could support. The -f /build-context/subdir/some/Dockerfile already treats that Dockerfile as if it's at the root of the build-context (i.e. paths inside the Dockerfile are not relative to the physical location of the Dockerfile), so it looks very similar to me.

@duglin
Copy link
Contributor

duglin commented Oct 19, 2016

Well, I think that fails because using '-' as the build context means that there is no context and stdin is the Dockerfile, not the context. Or am I not folllowing?

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

@thaJeztah I agree that we could definitely support arbitrary Dockerfile paths with the tar stream rewriting I use to put stdin into the context. When I make a PR, I will put that in a secondary commit.

@duglin I don't understand what you are referring to re: .dockerignore as, currently, if .dockerignore matches the referenced Dockerfile then the Dockerfile is included in the build context regardless. Why do we need to mess with .dockerignore? If you use - as the context, it can be either a tar stream or a Dockerfile (by sniffing for magic). We should either disallow -f - - or make it look at the stdin tar stream for a Dockerfile named -. The only collision with user data that I'm aware of is if the user wants to both provide a Dockerfile on stdin and access the original Dockerfile Dockerfile in the container build. This use case seems far-fetched and is easy to work-around/satisfy with renaming, symlinks, or hard links.

@justincormack It would be cool if you could pass a local tarball as a build context to go in the direction that you are thinking. Unfortunately, your tar cf - *.c *.h | docker build -f Dockerfile - pipeline would resolve Dockerfile relative to the root of the tarball and I'm not sure this is something we can change without breaking lots of stuff. You can currently run tar cf - Dockerfile *.c *.h | docker build - to do the same thing, though. There is an inconsistency with named local tarballs, though, as all other tarball contexts have -f relative to the tarball but all local contexts have -f relative to the docker build process.

We could have a method for packing both tarballs and dockerfiles into stdin (multi-part MIME?!) but that is quite an extension and design exercise.

@erikh
Copy link
Contributor

erikh commented Oct 19, 2016

What's wrong with docker build -f /dev/stdin .?

relevant: https://codereview.appspot.com/22170044/#msg2

On Wed, Oct 19, 2016 at 12:23 PM, David Sheets notifications@github.com
wrote:

@thaJeztah https://github.com/thaJeztah I agree that we could
definitely support arbitrary Dockerfile paths with the tar stream rewriting
I use to put stdin into the context. When I make a PR, I will put that in a
secondary commit.

@duglin https://github.com/duglin I don't understand what you are
referring to re: .dockerignore as, currently, if .dockerignore matches
the referenced Dockerfile then the Dockerfile is included in the build
context regardless. Why do we need to mess with .dockerignore? If you
use - as the context, it can be either a tar stream or a Dockerfile (by
sniffing for magic). We should either disallow -f - - or make it look at
the stdin tar stream for a Dockerfile named -. The only collision with
user data that I'm aware of is if the user wants to both provide a
Dockerfile on stdin and access the original Dockerfile Dockerfile in the
container build. This use case seems far-fetched and is easy to
work-around/satisfy with renaming, symlinks, or hard links.

@justincormack https://github.com/justincormack It would be cool if you
could pass a local tarball as a build context to go in the direction that
you are thinking. Unfortunately, your tar cf - *.c *.h | docker build -f
Dockerfile - pipeline would resolve Dockerfile relative to the root of
the tarball and I'm not sure this is something we can change without
breaking lots of stuff. You can currently run tar cf - Dockerfile *.c *.h
| docker build - to do the same thing, though. There is an inconsistency
with named local tarballs, though, as all other tarball contexts have -f
relative to the tarball but all local contexts have -f relative to the docker
build process.

We could have a method for packing both tarballs and dockerfiles into
stdin (multi-part MIME?!) but that is quite an extension and design
exercise.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#27393 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABJ6xaEeHB8COIslUeKl2_5Ib6Kp8zSks5q1m45gaJpZM4KXOEm
.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

@erikh /dev/stdin would have to be special-cased and docker build already has the - means stdin convention.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

The special-casing would not be required if the out-of-context path suggestion by @thaJeztah was used. I think at that point only the - convention and Windows compatibility are relevant. -f /dev/stdin doesn't work right now, though.

@erikh
Copy link
Contributor

erikh commented Oct 19, 2016

right, special casing would be needed for the context, but -f does not
prescribe the context, it only prescribes the dockerfile.

I haven't tested but I'm pretty sure there's no barrier here.

On Wed, Oct 19, 2016 at 1:08 PM, David Sheets notifications@github.com
wrote:

The special-casing would not be required if the out-of-context path
suggestion by @thaJeztah https://github.com/thaJeztah was used. I think
at that point only the - convention and Windows compatibility are
relevant. -f /dev/stdin doesn't work right now, though.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27393 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABJ6_tQgwMXL2Ggp_xQD0FqD3xqINvNks5q1njNgaJpZM4KXOEm
.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

As I said, -f /dev/stdin does not work currently and out-of-context Dockerfiles are explicitly prohibited (because the Dockerfile is carried inside of the build context and not separately). To use /dev/stdin, the context tar stream would still need to be rewritten, Dockerfile would still need to be overwritten, and the out-of-context Dockerfile prohibition would need to be lifted.

With @thaJeztah's suggestion to rewrite the context tar stream for out-of-context Dockerfile references, all of those changes would be made. Portability and convention would be the reasons (and good ones IMO) to support -f - but -f /dev/stdin would work fine after this proposal (+ @thaJeztah's suggestion) is implemented as planned.

@erikh
Copy link
Contributor

erikh commented Oct 19, 2016

this works just fine on 1.12.2:

docker build -f /dev/stdin . < Dockerfile

Just verified. Anyways, I'm tapping out.

On Wed, Oct 19, 2016 at 2:34 PM, David Sheets notifications@github.com
wrote:

As I said, -f /dev/stdin does not work currently and out-of-context
Dockerfiles are explicitly prohibited (because the Dockerfile is carried
inside of the build context and not separately). To use /dev/stdin, the
context tar stream would still need to be rewritten, Dockerfile would
still need to be overwritten, and the out-of-context Dockerfile prohibition
would need to be lifted.

With @thaJeztah https://github.com/thaJeztah's suggestion to rewrite
the context tar stream for out-of-context Dockerfile references, all of
those changes would be made. Portability and convention would be the
reasons (and good ones IMO) to support -f - but -f /dev/stdin would
work fine after this proposal (+ @thaJeztah https://github.com/thaJeztah's
suggestion) is implemented as planned.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27393 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABJ67bZtEfZVUzJqJVQ3ln5IwfUt5o9ks5q1ozWgaJpZM4KXOEm
.

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

@erikh, that's interesting. This works on 1.12.2 on Linux but not on OS X... thanks for the encouragement!

@dsheets
Copy link
Contributor Author

dsheets commented Oct 19, 2016

It doesn't always work on Linux, either --

# docker build -f /dev/stdin . < ../D
unable to prepare context: The Dockerfile (/dev/stdin) must be within the build context (.)
# docker build -f /dev/stdin foo < D
unable to prepare context: The Dockerfile (/dev/stdin) must be within the build context (foo)
# docker build -f /dev/stdin foo < foo/Dockerfile
Sending build context to Docker daemon 2.048 kB
# docker build -f /dev/stdin . < Nockerfile
Sending build context to Docker daemon 3.072 kB

This seems quite brittle and does not actually satisfy the specification in this proposal. I'm currently inclined to believe that this disgusting magic is courtesy of the shell's I/O redirection rewriting /dev/stdin into the path passed via < but I haven't confirmed, yet. Here's some pretty damning evidence (edit: no, it's not, I was too tired when I posted this):

# echo /dev/stdin < wut
-ash: can't open wut: no such file

@erikh, or others, do you have any idea what is going on here? The source code (and my understanding) supports the error case only. I'll go hunting for this bogus magic tomorrow, I suppose...

@dsheets
Copy link
Contributor Author

dsheets commented Oct 20, 2016

Your shell replaces stdin (fd 0) with an open file which /proc changes into a symlink:

# readlink -f /dev/stdin < Dockerfile
/root/foo/Dockerfile

This doesn't work on OS X (or likely other BSDs) because /dev/stdin terminates to a character device like /dev/fd/0 there.

This means that docker -f /dev/stdin . cannot participate in a pipeline as we would like:

# cat Dockerfile | docker build -f /dev/stdin .
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /proc/6106/fd/pipe:[20920]: no such file or directory

Essentially, using /dev/stdin currently will only work in exactly the same circumstances that passing a normal in-context Dockerfile will work in and only on Linux due to its exposure of the stdin file as a symlink back to the open file fd.

@erikh
Copy link
Contributor

erikh commented Oct 21, 2016

I think github's mail system died, so I'll echo my thoughts directly into the form, hopefully they do not deliver duplicates later, sorry in advance.

I think I see part of the problem. I agree, it's confusing behavior at best, but I think it's exposing a breakout of the context so I wanted to explain.

I tried this for giggles:

[16] erikh@b842e9cf3c0b github.com/erikh# cat box/Dockerfile | docker build -f /dev/stdin box
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /proc/1740/fd/pipe:[2629405]: no such file or directory

https://github.com/docker/docker/blob/master/builder/context.go#L219 seems to be the relevant portion of code here. How we qualify -f seems like it has some holes.

https://github.com/docker/docker/blob/master/builder/context.go#L249-L251 specifically doesn't seem to check if, say, the path is ./../../../dev/stdin. I haven't validated this but if the context is . and is not qualified to an absolute path, this may be possible.

Additionally,

I don't know if docker is doing anything fancy but I checked:

package main

import (
        "fmt"
        "os"
)

func main() {
        f, err := os.Open("/dev/stdin")
        if err != nil {
                panic("wut")
        }

        fmt.Println(f.Fd())
}

This prints "3" which is what I expect it to. Maybe docker is seeing stdin first and ignoring the -f option?

@thaJeztah
Copy link
Member

@tonistiigi did some testing, and may be able to add more info there

@erikh
Copy link
Contributor

erikh commented Oct 21, 2016

Unsure if it's useful, but I've been using this in my path handling code on my own projects (could replace 249-251)

paths := filepath.SplitList(rel)
for _, path := range paths {
   if path == ".." {
     return nil, createException(m, fmt.Sprintf("Cannot use relative path %s because it may fall below the root build directory", source))
   }
}

@dsheets
Copy link
Contributor Author

dsheets commented Oct 21, 2016

@erikh symlinks are resolved prior to path relativization and context-escape checking. On Linux, the kernel creates a symlink chain /dev/stdin -> /proc/self/fd/0 -> /path/to/file/redirected/with/shell. ../../../../../../dev/stdin is handled correctly. I don't see any context escape.

@erikh
Copy link
Contributor

erikh commented Oct 25, 2016

I think I see part of the problem. I agree, it's confusing behavior at
best, but I think it's exposing a breakout of the context so I wanted to
explain.

I tried this for giggles with the repo @ https://github.com/erikh/box:

[16] erikh@b842e9cf3c0b github.com/erikh# cat box/Dockerfile | docker build
-f /dev/stdin box
unable to prepare context: unable to evaluate symlinks in Dockerfile path:
lstat /proc/1740/fd/pipe:[2629405]: no such file or directory

https://github.com/docker/docker/blob/master/builder/context.go#L219 seems
to be the relevant portion of code here. How we qualify -f seems like it
has some holes.

https://github.com/docker/docker/blob/master/builder/context.go#L249-L251
specifically doesn't seem to check if, say, the path is
./../../../dev/stdin. I haven't validated this but if the context is .
and is not qualified to an absolute path, this may be possible.

On Wed, Oct 19, 2016 at 4:52 PM, David Sheets notifications@github.com
wrote:

It doesn't always work on Linux, either --

docker build -f /dev/stdin . < ../D

unable to prepare context: The Dockerfile (/dev/stdin) must be within the build context (.)

docker build -f /dev/stdin foo < D

unable to prepare context: The Dockerfile (/dev/stdin) must be within the build context (foo)

docker build -f /dev/stdin foo < foo/Dockerfile

Sending build context to Docker daemon 2.048 kB

docker build -f /dev/stdin . < Nockerfile

Sending build context to Docker daemon 3.072 kB

This seems quite brittle and does not actually satisfy the specification
in this proposal. I'm currently inclined to believe that this disgusting
magic is courtesy of the shell's I/O redirection rewriting /dev/stdin
into the path passed via < but I haven't confirmed, yet. Here's some
pretty damning evidence:

echo /dev/stdin < wut

-ash: can't open wut: no such file

@erikh https://github.com/erikh, or others, do you have any idea what
is going on here? The source code (and my understanding) supports the error
case only. I'll go hunting for this bogus magic tomorrow, I suppose...


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27393 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABJ6y9F3EqAtnAdmaLbp3I_L0rHBIFKks5q1q0ggaJpZM4KXOEm
.

@erikh
Copy link
Contributor

erikh commented Oct 25, 2016

I don't know if docker is doing anything fancy but I checked:

package main

import (
        "fmt"
        "os"
)

func main() {
        f, err := os.Open("/dev/stdin")
        if err != nil {
                panic("wut")
        }

        fmt.Println(f.Fd())
}

This prints "3' which is what I expect it to. Maybe docker is seeing stdin
first and ignoring the -f option?

@Magnitus-
Copy link

Magnitus- commented Dec 22, 2016

I think that would be great. I could finally stick a placeholder in the FROM directive for multiple distros that are closely related, cat & sed the dockerfile with the right distro plugged in to docker build (instead of generating a temporary dockerfile for each distro) and still have access to the context for COPY operations.

Atm, I have a bunch of centos 6 & centos 7 dockerfiles that look exactly the same, except for a differing FROM directive.

@kcmerrill
Copy link

@Magnitus- Sorry ... Not sure if this is still relevant to you but one way I got around the issue of multiple docker files was simply tagging the base image before building

So it'd look like this

docker tag centos:6 mybaseimg
... build
docker tag centos:7 mybaseimg
... build

Dockerfile:
FROM mybaseimg
RUN yum install whatever
etc etc

@tonistiigi
Copy link
Member

I'm ok with both allowing -f outside of context directory(but it shouldn't follow symlink to parent) and -f - if context doesn't also specify -. There's some trickery that needs to happen for content trust for this but shouldn't be a big issue. This should also work on API side when a remote context is specified. I see there was already some work for it(NewBuilder accepts dockerfile readcloser) but I guess it was never finished.

If there are no objections(didn't spot in comments) and nobody has already done work on this, it can be assigned to me.

dsheets added a commit to dsheets/moby that referenced this issue Feb 15, 2017
Implements proposal moby#27393 that describes the meaning of '-f -' in a
`docker build` command. For all context types except reading a context from
stdin, '-f -' will replace any file called 'Dockerfile' in the context's
tar stream with the contents of stdin. If the context itself is read from
stdin, '-f -' continues to read the Dockerfile called '-' from this context.

Fixes moby#27393.

Signed-off-by: David Sheets <sheets@alum.mit.edu>
@dsheets
Copy link
Contributor Author

dsheets commented Feb 15, 2017

@tonistiigi I've pushed the -f - work I had previously done to https://github.com/dsheets/docker/commits/build-dockerfile-stdin which may be of some use to you (or perhaps the approach is not so good). I also had some changes to allow arbitrary files outside of context directory but they seem to have bitrotted due to the removal of builder/context.go.

@tonistiigi
Copy link
Member

tonistiigi commented Feb 15, 2017

A problem with setting -f outside of context dir is that it is not possible to determine if docker build -f Dockerfile foo/ means that Dockerfile is at foo/Dockerfile or ./Dockerfile. I think docker build -f ../Dockerfile foo/ is quite counter-intuitive (as well as absolute paths). Unless there is a solution for that we would need to settle for -f - . Also docker build - <Dockerfile already works but isn't that useful.

@duglin
Copy link
Contributor

duglin commented Feb 15, 2017

just some history... when -f was added it was decided that the value of "-f" was relative to the current dir (unless its an abs path).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/builder kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants