Proposal: Dockerfile add INCLUDE #735

Closed
dysinger opened this Issue May 28, 2013 · 108 comments

Projects

None yet
@dysinger

No description provided.

@shykes
Contributor
shykes commented Jun 21, 2013

+1

@keeb
Contributor
keeb commented Jun 21, 2013

+1

@ptone
ptone commented Jun 21, 2013

Yes this would be great to see +1

@jpfuentes2

I think this would be a great feature as I want to leverage some of my knowledge/experience with systems like Chef whereby you can compose complex builds using smaller/simpler building blocks.

Does anyone have a way they're implementing this now?

@crosbymichael
Member

Can someone give me a few examples on how they would use this? ping @ptone @keeb @dysinger

@ptone
ptone commented Aug 12, 2013

Lets say I have build file snippets for different components of web architecture.

I may want to include 1 or more of them on a single container image - or in general bundle things that always go together.

Simple examples might be nginx and varnish always go on the same container, or redis and pyredis.

I might have a "Scientific Python" list of python packages and linked libs, that I may want to include in other images.

The problem with FROM and base images is that you can't remix things the same way you can with includes. A base image is all or nothing - if you don't want something, your only hope is that you can go back to a 'lower' base image, and re-add stuff manually that you want, skipping the stuff you don't.

It is essentially a case of inheritance vs composition in many ways.

@binocarlos

+1 to this - @crosbymichael @ptone @keeb @dysinger @shykes

I am at this exact point where I have an appserver base image (which is just node.js).

I also have a collection of Dockerfiles representing little bits of services that have deps - so:

ImageMagick:

from appserver
run apt-get install imagemagick

RedisSession:

from appserver
run apt-get install redis-server

To have a container that is both ImageMagick and RedisSession:

from appserver
run apt-get install imagemagick
run apt-get install redis-server

Whereas the following syntax means I can build up a folder of modules and include them by name in the application Dockerfile:

from appserver
include ./modules/imagemagick
include ./modules/redis-server

Now, because Docker is so darn brilliantly good : ) this is currently trivial to do (i.e. read module file and inject here) and so I'm not sure if for me this is a required feature.

However - it would make a user app's Dockerfile (i.e. the last thing in the chain for this scenario) much more about composing modules together than about creating a strict inheritance tree where some combinations are not possible - my 2 cents on what is a joy to work with otherwise :)

@Thermionix
Contributor

+1 would also be good to reference out generic blocks (possibly to a github raw link?)

@frankscholten

+1

@flavio flavio added a commit to flavio/docker that referenced this issue Oct 15, 2013
@flavio flavio docker build: initial work on the include command
Added the 'include' command to dockerfile build as suggested by issue #735.
Right now the include command works only with files in the same directory of
the main Dockerfile or with remote ones.
ec3973c
@wagerlabs

I'll re-implement this on top of #2266.

@prologic
Contributor

+1 Turns Docker and the Dockerfile into a portable rudimentary configuration management system for the constructions of portable Docker images :)

@ghost
ghost commented Feb 14, 2014

This would help a lot. +1

@deeky666

+1

@dmarti dmarti referenced this issue in cloudius-systems/capstan Mar 2, 2014
Open

Includes in Capstanfile #11

@peenuty
peenuty commented Mar 10, 2014

+1

@newhoggy

+1

@jfinkhaeuser

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also get the need for parametrized Dockerfiles, and then for conditionals, etc. Continue down that path, and you'll end up implementing a programming language in Dockerfiles, which may even become turing complete. The cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program your own, you could use m4 (used in autotools for this purpose) or the C preprocessor (used in IMake, which does a similar job as autotools but is pretty much defunct these days).

Makefile:

Dockerfile: Dockerfile.in *.docker
  cpp -o Dockerfile Dockerfile.in

build: Dockerfile
  docker build -rm -t my/image .

Dockerfile:

FROM ubuntu:latest
MAINTAINER me

#include "imagemagick.docker"
#include "redis.docker"

Run make and it'll re-build the Dockerfile if any input file changed. Run make build and it'll re-build the Dockerfile if any input file changed, and continue to build the image.

Look ma, no code!

@prologic
Contributor

On Tue, Mar 11, 2014 at 6:45 PM, Jens Finkhaeuser
notifications@github.comwrote:

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also
get the need for parametrized Dockerfiles, and then for conditionals, etc.
Continue down that path, and you'll end up implementing a programming
language in Dockerfiles, which may even become turing complete. The
cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program
your own, you could use m4 (used in autotools for this purpose) or the C
preprocessor (used in IMake, which does a similar job as autotools but is
pretty much defunct these days).

I'm in agreement with this. Having toyed with design and implemtantions of
a few languages myself over the years
turning Dockerfile(s) into a "scripting" language even if it's not turning
complete sounds like something that Docker should not do.

As Jens clearly points out there are better more appropriate tools for this
job.

cheers
James

James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

@hunterloftis

+1

The slippery-slope argument for a turing-complete scripting language in Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just lets you create a common base image locally so you can reference a file system (for example, in your app's repository) instead of relying on the Docker registry for what should be a self-contained app definition.

@prologic
Contributor
prologic commented Apr 9, 2014

On Wed, Apr 9, 2014 at 12:15 PM, Hunter Loftis notifications@github.comwrote:

The slippery-slope argument for a turing-complete scripting language in
Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just
lets you create a common base image locally so you can reference a file
system (for example, in your app's repository) instead of relying on the
Docker registry for what should be a self-contained app definition.

I don't agree with this. The very notion of referencing and building a base
image is already there
and it only accesses the public registry (or a private onf if you're so
inclined) if you don't have said image.

I'm still -1 on this -- it adds more complexity for little gain. I'd rather
see Docker pick up some YAML-style
configuration format for "configuring one or more containers" ala fig et
all.

cheers
James

James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

@jfinkhaeuser

The slippery-slope argument stems from experience with a bunch of other DSLs. There's a general trend for DSLs to become turing complete over time.

The include statement in itself presents little danger here, but consider that in almost every language, include or import statements are linked to the concept of an include path, quite often set via an environment variable.

There's a reason for that: having include statements means you can collect building blocks into reusable libraries, and having reusable libraries means you'll want to use the same building blocks in multiple Dockerfiles. The best way to do that is to be agnostic to the actual file location in your include statement, but instead have a "canonical" name for a building block, which is usually a file path relative to some externally provided include path.

So what's wrong with that? Nothing, except (and I'm quoting you here):

(...) instead of relying on the Docker registry for what should be a self-contained app definition.

I agree. A Dockerfile should be a self-contained app definition. Once you include stuff from any location that's shared between projects, though, you have anything but that - the needs of one project will lead to modifications of the shared file and those may not reflect the needs of the other project any longer. Worse, any kind of traceability - this version of the Dockerfile should build the same image again - is completely gone.

Besides, once you have re-usable building blocks, parametrizing them is the obvious next step. That's how libraries work.

@hunterloftis

Then follow a common, well-understood example that has certainly not become turing complete:

https://developer.mozilla.org/en-US/docs/Web/CSS/@import

@ryedog
ryedog commented Apr 14, 2014

+1 on include (or even variables that can be declared on the command line on build would be helpful)

@ChristianKniep

+1 on include as well

I have to deal with one site where I have to set http_proxy and one without.

@kennu
kennu commented May 8, 2014

My +1 use case for INCLUDE would be to be able to offer third party components that are easily added to any existing Dockerfile. So if you want your Docker image to utilize a software product or SaaS service, you just add:

INCLUDE https://example.com/dockerfile

And it's ready to go. No need to add complicated preprocessing steps.

@crosbymichael crosbymichael removed this from the 0.7.1 milestone May 15, 2014
@dougmoscrop

+1

I don't think INCLUDE is an affront to what Docker is trying to do at all. The fact that there exist containers that are all FROM ubuntu and install separate things like Ruby, Java, Node, and I have to create a new one that inherits from one, and copy-pastes from the others, just to have those three items available, says something is missing to me.

It would be great if INCLUDE fetched the Dockerfile from the registry that was used to build the referred image (making the registry more powerful), and I think a way of keeping INCLUDE in check might be that when a FROM is encountered in the context of an include, it asserts that the image being built shares a descendant with the included image.

So I can INCLUDE foo/ubuntu-java in my image that is FROM ubuntu but not one FROM centos

@prologic
Contributor

On Fri, May 30, 2014 at 12:26 PM, Doug Moscrop notifications@github.com
wrote:

It would be great if INCLUDE fetched the Dockerfile from the registry
that was used to build the referred image (making the registry more
powerful), and I think a way of keeping INCLUDE in check might be that
when a FROM is encountered in the context of an include, it asserts that
the image being built shares a descendant with the included image.

So I can INCLUDE foo/ubuntu-java in my image that is FROM ubuntu but not
one FROM centos

So this makes the proposed "INCLUDE" feature dependent on images and what
they're based on.
i.e: there's a direct relationship between FROM and INCLUDE where parent
base images have to match in some way.

Doesn't this complicate the implementation and usage a bit?

cheers
James

James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

@dougmoscrop

Could be.. I viewed it as a restriction that might mean more predictability for the result (amid concerns in another thread about 'grafting centos on to ubuntu'). My gripe is looking at the index and seeing a bunch of different things that are useful, and not wanting to just copy and paste their innards. You're right that there's probably (definitely) some hidden gotchas in what I'm suggesting.

I could rephrase my comments to be:

"As a Dockerfile author, I would like to re-use the work other people have put in to their Dockerfiles so that my specific instructions are more unique, and I'm willing to accept some kind of inheritance constraint when doing so".

@dougmoscrop

I'll also add that, without some kind of mixin/include functionality, it's much more difficult to have - or rather, use - "authoritative" or canonical instructions (vs. "trusted builds"):

Imagine joyent/ubuntu-nodejs as a starting place. Now there's also oracle/ubuntu-java. If I want both, I have to create dmoscrop/ubuntu-nodejs-java and inherit from one while adding instructions copied from the other. Now someone else wants Java, Node and Ruby. Inheriting from mine feels "dirty" (because Oracle could easily push new updates to the instructions for installing Java and I've abandoned mine, or simply that image is stale) so they end up inheriting from one and copy-paste in the other two, and it goes on, combinatorially, as different people choose their inherit vs. copy sources.

Really, I don't care about dmoscrop/ubuntu-nodejs-java conceptually or as an artifact. I care about dmoscrop/my-project-builder which is simply FROM ubuntu INCLUDE joyent/nodejs INCLUDE oracle/java and some build steps.

@caffinatedmonkey

@dmoscrop +1

@gzankevich

+1 on include

@richardgill

@dmoscrop +1

@ishanagr

+1 this would be very helpful

@dergachev

+1

@ChristianKniep

To emphasis my usecase (a common one, I guess):

INCLUDE env.dockerfile

On-site:

$ cat env.dockerfile
ENV http_proxy http://theproxy.local:3128
$

Normal case:

$ cat env.dockerfile
$

Currently I have to differnet Dockerfiles.

@kennu
kennu commented Jul 25, 2014

@ChristianKniep I think your use case might be better served with a docker build -e option to specify environment variables when building an image. I myself could use both that and INCLUDE.

@ChristianKniep

@kennu To simple! Oh my god, sometimes I wonder how I get up by myself. :)

@avelino
avelino commented Jul 31, 2014

+1

@mitar
mitar commented Aug 17, 2014

What about allowing multiple FROM to combine things? Or maybe MIXIN? So instead of including Dockerfile, one could combine multiple images?

@erikh
Contributor
erikh commented Aug 17, 2014

Just marking that after #7461 we can tackle this if desired

@tiborvass
Contributor

How is this overlapping with #7115 or #7149 ?

@zeroasterisk

+1 - I would also like this

@thorsummoner

👍

@felixrabe
Contributor

+1 (and also +1 for having a way to vote on issues on GitHub so I don't have to +1-spam issues) :)

@erikh
Contributor
erikh commented Oct 2, 2014

We’re actively addressing this.

On Oct 2, 2014, at 11:48 AM, Felix Rabe notifications@github.com wrote:

+1 (and also +1 for having a way to vote on issues on GitHub so I don't have to +1-spam issues) :)


Reply to this email directly or view it on GitHub.

@jokeyrhyme

INCLUDE or IMPORT would greatly reduce duplication for me, as I have a project with multiple Dockerfiles, with various common tasks between them.

The only behavioural changes should be at the time of parsing the Dockerfile (by replacing the INCLUDE ... with the appropriate entries from that resource).

It may be useful to allow INCLUDE to work with HTTPS URLs, but I would still be very happy if it initially only supported local resources that are already able to be accessed with an ADD.

I actually just contemplated a build process to concatenate my snippets together into the final Dockerfiles I need, so I'm relieved to see that others have the same issues and that progress is being made. :)

@marcellodesales

I have a runtime tha depends on:

  • Ruby sources: some older scripts that were not ported to Node.js
  • Node.js Sources: new micro-services that makes use of ruby services.

Right now I started a Dockerfile with Ruby, and then installed Node.js...

FROM binaryphile/ruby:2.1.2
MAINTAINER Marcello_deSales@intuit.com

# Install Node.js and npm
RUN apt-get install -y python-software-properties --fix-missing && \
    apt-add-repository ppa:chris-lea/node.js && \
    apt-get update && apt-get install -y nodejs

Would this cover my requirement? I'm looking for something in the lines of:

FROM ubuntu/ruby:2.1.2
FROM ubuntu/nodejs 
...
...

Thanks!

@pypeng
pypeng commented Nov 13, 2014

+1

@jokeyrhyme

So, I'm not making any promises as I'm fairly new to Go, but I thought I'd try to tackle this over a few weekends. My progress will be found at https://github.com/jokeyrhyme/docker/tree/735-include-dockerfile until I'm ready to make a Pull Request.

Proposed specification:

  • new IMPORT keyword for Dockerfiles (inspired by HTML and CSS, rather than PHP's include) with similar syntax compared to ADD, e.g. IMPORT path/to/fragment.to.read.in
  • IMPORT sources must obey all the same rules as ADD sources, although initially I'll focus on individual source files (not archives, directories or remote URLs) unless it's super easy
  • IMPORT lines can be anywhere in the Dockerfile
  • there are no mandatory directives in individual IMPORTed source (e.g. FROM)
  • IMPORTs can be chained when sources include more IMPORTs
  • as this is a feature intended to maximise reuse, it makes sense to allow the same source to be
    IMPORTed multiple times whilst processing the same Dockerfile
  • the final result should not break any Dockerfile rules
    • FROM still needs to be at the top, and there can only be one
  • parser errors tell you which IMPORTed file is involved, if any
  • the directed graph of IMPORTs should be acyclic (no circular references)

Typing this all out makes me realise that this is not a trivial issue. Any comments? Is this a fairly complete specification for how this feature needs to work?

@prologic
Contributor

@jokeyrhyme What does an IMPORt look like in a Dockerfile?

@jokeyrhyme

@therealprologic e.g. IMPORT path/to/fragment.to.read.in

@prologic
Contributor

Ahh gotcha :)

@suntong
suntong commented Nov 30, 2014

+1.

Please review jokeyrhyme's patch and incorporate it soon.

I am using Docker to do my Debian/Ubuntu Package Developing, the only difference between my Debian DockerFile and Ubuntu's is their FROM line. All the rest are the same! So you can see how that INCLUDE is useful to me.

PS. I'd rather it is named as INCLUDE, not IMPORT, following C/C++. IMPORT implies more action, e.g., it's a perfect word for specifying multiple images per context, if it is ever possible.

@cc272309126
Contributor

I want to know if the “import” feature of dockerfile can be used now? I have write a ugly version of this feature。cc272309126@3f7e775

@cc272309126
Contributor

the test case I have not finish it. it is not for import but for include. that means you can use include /dockerfile in your Dockerfile

@jokeyrhyme

That's great, @cc272309126. My own work on this is slow. I'm having trouble running the Docker automated tests on my system (OSX with Boot2Docker), and I haven't had much time to work on this.

@analytically

+1

@cc272309126
Contributor

Hi,I have a question about the import command. like if dockerfile B want import A from a url like http:xxx/xxx/A. and A has ADD and COPY command in the dockerfile. How I get the ADD and COPY command's source files? one way is I should prepare the source files advance。 but for import command I think it should not care for the source files。the another way is that I should write the ADD command in dockerfile A like ADD http:xxx/xxx/files dir。but when I write the dockerfile。I should not care for which url I should put the dockerfile。

@jokeyrhyme

@cc272309126 is it okay if all ADD and COPY directives (even in IMPORTed files) work from the directory of the root Dockerfile as they normally would? The end result is just as though you only had 1 big Dockerfile?

@cc272309126
Contributor

@jokeyrhyme yes, it is okey for all ADD and COPY directives (even in IMPORTed files) work from the directory of the root Dockerfile. the end result is not a big Dockerfile. it build the imported dockerfile when it run into the IMPORT command. And all the resource of imported dockerfile is relative from root Dockerfile's path。

@ahoy-jon

I did a bash script to compensate the missing INCLUDE : https://gist.github.com/ahoy-jon/e8c534883a5a8c87c046, but finally I use it to compose existing Dockerfile (removing FROM directives from sub Dockerfile). It doesn't support proper ADD (eg : an ADD from an included URL should resolve on the network, relatively (an ADD from an included file should resolve relatively too ? )).

@suntong
suntong commented Dec 27, 2014

I did a bash script to compensate the missing INCLUDE ...

A showcase of how badly that people need this, every one is jumping through the hoops in their own way trying to fix the problem. ...

  • cpp can do recursive but won't fetch from the web
  • this one can fetch from the web but can't do recursive

we need a solution, not patches on patches.

@ahoy-jon

This bash solution is recursive, but doesn't solve stuff around ADD.

I changed now the example to make it clear. Here is the recursive part of the script : https://gist.github.com/ahoy-jon/e8c534883a5a8c87c046#file-m5-bash-L31

@suntong
suntong commented Dec 28, 2014

On Sat, Dec 27, 2014 at 5:05 PM, Jonathan Winandy notifications@github.com
wrote:

This bash solution is recursive, but doesn't solve stuff around ADD.

Oh, yes. Sorry.

@neglectedvalue

That's will be great! 👍

@ajvb
ajvb commented Jan 9, 2015

This would be really nice, especially when combined with the docker build -f flag that has recently been merged into master. Allows for multiple dockerfiles with a core one so that they can stay DRY.

@mpareja
mpareja commented Jan 13, 2015

-1 Include would only solve a very narrow use-case. You'll soon find yourselves needing just a slight bit more power.

This is why tools like chef, ansible and puppet have thriving ecosystems of "cookbooks/recipes", "playbooks/tasks" and "resources". They also allow you to tweak how those things are applied. While I don't want to manage everything using these tools, I do appreciate the re-usability and composability you get from a library of tools to help you get stuff on a machine - even if it is just once during a docker build.

Disclaimer, I haven't tried using something like chef inside the image build process, but I've certainly felt the pain of wanting to add the same functionality to multiple images and copying it around in Dockerfiles. I also don't think using parent images to DRY (don't repeat yourself) up your Dockerfiles is a good idea. Parent images remind me a lot of inheritance in object oriented programming - it's either precisely what you need or steer the hell clear of it because the coupling makes changing things very difficult.

@jessfraz jessfraz changed the title from please add INCLUDE <file> to dockerfile build so we can build more complex images to Proposal: Dockerfile add INCLUDE Feb 26, 2015
@aanand aanand referenced this issue in docker/compose Mar 4, 2015
Closed

request: add build priority/order #295

@duglin
Contributor
duglin commented Mar 5, 2015

@tiborvass @jfrazelle @crosbymichael
I'd like to hear your take on this as the topic of an INCLUDE does pop up from time to time.

Personally, I don't see an issue with it since adding it doesn't break any design principle of Docker and if it makes people's lives better (or at least if they think it does :-) ), and there is no nice alternative, then I'm ok with it. I don't really see it being much different than an "include" statement in a Makefile.

I would however suggest that we allow for it to be a bit more powerful by allowing env vars to be used - e.g.:

INCLUDE dockerfiles/os-specific.${os:-default}

But we can always do that as a secondary step.

(edit: added bit about no nice alternative)

@devTristan

If you want to generate Dockerfiles with includes, conditionals, and turing completeness you can use dockerscript. Full disclosure: I made this because I wanted includes.

@duglin duglin added a commit to duglin/docker that referenced this issue Apr 16, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
5da21d0
@duglin duglin added a commit to duglin/docker that referenced this issue Apr 24, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
c6311c4
@duglin duglin added a commit to duglin/docker that referenced this issue May 5, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
323ceff
@duglin duglin added a commit to duglin/docker that referenced this issue May 6, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
5d9fdde
@duglin duglin added a commit to duglin/docker that referenced this issue May 7, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
6b9132b
@duglin duglin added a commit to duglin/docker that referenced this issue Jun 16, 2015
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

Once #10775 is merged, I'd like to add a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo}

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
3618080
@jessfraz
Contributor

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Mainly:

Allowing the Builder to be implemented as a separate utility consuming the Engine's API will open the door for many possibilities, such as offering alternate syntaxes or DSL for existing languages without cluttering the Engine's codebase

Then from there, patches/features like this can be re-thought. Hope you can understand.

@jessfraz jessfraz closed this Jul 10, 2015
@jcoffland

I know you guys are stubborn as hell on this topic, which is commendable, but why doesn't docker just run m4 on Dockerfiles by default. This would give docker many powerful features including include() with out changing the syntax. docker build would need to be modified to accept some m4 command line options such as -I but that's about it.

@caffinatedmonkey

Its because we want an include directive which is smart and merges the images together. Someone correct me if I am wrong. Saying use m4 is like telling a car owner to start their car with a screwdriver. It works but its not elegant and adds more dependencies.

On Wed, Oct 21, 2015 at 1:26 PM -0700, "Joseph Coffland" notifications@github.com wrote:
I know you guys are stubborn as hell on this topic, which is commendable, but why doesn't docker just run m4 on Dockerfiles by default. This would give docker many powerful features including include() with out changing the syntax. docker build would need to be modified to accept some m4 command line options such as -I but that's about it.


Reply to this email directly or view it on GitHub:
#735 (comment)

@thaJeztah
Member

The reason this feature is closed now, is that we are working on moving the builder out of the daemon to the client. This allows other implementations (you'd even be able to create your own Dockerfile alternative), see https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Before that has arrived, we don't want to make changes to the Dockerfile syntax. Apart from that, we want to keep the Dockerfile syntax simple (we will never be able to have a one-size-fits-all solution), hence allowing other implementations that can be tailored more to specific needs.

@jcoffland

@caffinatedmonkey I think you are assuming that m4 is not good because it's old. m4 is more like a swiss army knife than a screwdriver. But what does this have to do with starting cars?

@thaJeztah "we don't want to do it now" is a terrible reason for closing an issue. This is exactly why you want to leave issues open.

@thaJeztah
Member

@jcoffland it was decided to close these issues until it's become more clear what the future of the builder/Dockerfiles will look like, but please "ping" once the builder change is implemented.

I'll also ask among the other maintainers if re-opening these requests before that is something to consider

@jcoffland

I recommend using GitHub milestones. You could reopen these issues and put them in a future milestone. In my experience, this keeps issue reporting users happy because they don't see you dismissing ideas they consider important. Yet you can still easily sort out issues which are not currently important to you. I often create a non-versioned milestone with a name like unscheduled for this purpose.

@caffinatedmonkey

@jcoffland I'm not assuming m4 is not good because it's old. It is an example of design has stood the test of time. I was trying to use the car screwdriver thing as a metaphor.

@tomalakgeretkal

I would have used this so that my 32-bit* and [native] 64-bit CentOS 6 images wouldn't be 184 identical lines with a couple added to the top of just one.

* Same approach as https://hub.docker.com/r/toopher/centos-i386/~/dockerfile/

@jcoffland

@tomalakgeretkal I use m4 for exactly this purpose. You can do so now by preprocessing your Dockerfile with something like this:

m4 -I . base.m4 > Dockerfile

It would be even better if docker either had an option to automatically preprocess with m4 or better yet always did so. Docker would need to add -I, for include, to the docker build command but that's about it. Supper easy to implement and a general solution which solves a lot of problems.

@rncry
rncry commented Nov 12, 2015

+1 for this (whenever it gets reopened)

@assalw
assalw commented Dec 2, 2015

+1 This would really be great! I need this to structure my Docker files in a sensible way.

@kujiy
kujiy commented Dec 3, 2015

+1
WE REALLY NEED!

@avirshup
avirshup commented Dec 8, 2015

Docker Make might be a solution to many of the problems here. It supports both:

  1. multiple dockerfile "inheritance", and
  2. ADDs/COPYs from multiple locations.

Here's @binocarlos 's example, with files added from multiple locations. You specify the build with a YAML file:

imagemagick:
 build_directory: /path/to/data
 build: |
   RUN apt-get install imagemagick
   ADD x.png #adds /path/to/data/x.png

redis-session:
 build_directory: /another/path/
 build: |
   RUN apt-get install redis-server
   ADD y.dat #adds /another/path/y.dat

combined:
 FROM: appserver
 requires:
  - redis-session
  - imagemagick

And create the final image just by running

docker-make.py combined

Full disclosure: I wrote DockerMake because I was sick of dealing with "FROM" dependencies.

@kujiy
kujiy commented Dec 10, 2015

@avirshup Is Docker-Make a tool like docker-compose for making a docker image?

@juanmirocks

+1

@jakirkham

Yeah, the idea of parsing Dockerfiles through m4 is abhorrent to me for exactly the reason mentioned by @caffinatedmonkey. Namely we are shipping off a feature that really belongs in the core spec. This means it will be reimplemented in various different ways and I may not be able to reuse the same strategy that someone else has as there is no official standard (worse code duplication than just setting and enforcing a standard). Not to mention, we have now added a dependency that will be hard to get on some platforms. As far as a path, we are already looking at paths for context anyways and we already have mechanisms for changing them. In short, I prefer the original proposal.

@Vanuan
Vanuan commented Mar 6, 2016

Another solution is Rocker: https://github.com/grammarly/rocker
IMO, it solves all the weak points of Dockerfile, making it suitable for development.

@rgpublic

Rocker doesn't seem to have INCLUDE. At least I can't seem to get it to work. Surprisingly complex issue, indeed! I'm a novice Docker user and was really suprised that sth. so simple and straight-forward like an INCLUDE statement isn't - no pun intended - included. If Rocker doesn't have this as well I think I will need to write some kind of custom script to merge multiple files together, Oh, well...

@Vanuan
Vanuan commented Mar 14, 2016

@rgpublic it's called IMPORT

@rgpublic

@Vanuan: Thank you so much for your response, but I think IMPORT does sth. different:

INFO[0000] IMPORT "includes/init.rocker"
FATA[0000] You have to EXPORT something first in order to IMPORT

I don't want to export sth. first or move stuff between images, I originally only wanted to split a long and confusing .docker file into multiple ones to make it easier to read and more modular. Just like the INCLUDE statement in PHP, CSS, C++ and many other languages. Isn't that what INCLUDE is all about? Strangely, they even added INCLUDE to the syntax highlighting even if it doesnt work:

grammarly/rocker#29

This all simply turns into too much of a mess for me for such a "minor" issue. I think I'll simply write a small preprocessor script to sort this out.

@Vanuan
Vanuan commented Mar 15, 2016

Yeah, it's a different one. Sorry

@phpdude
phpdude commented Mar 23, 2016

+1 to INCLUDE

@adw0rd
adw0rd commented Mar 23, 2016

👍

@boneyao
boneyao commented Apr 16, 2016

+1

@caffinatedmonkey
caffinatedmonkey commented Apr 17, 2016 edited

😭 Does no one realize the +1 spam isn't necessary anymore? https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments

@SJLC SJLC added a commit to VCTLabs/ansible-test that referenced this issue May 3, 2016
@SJLC SJLC Example Dockerfile for working Gentoo base image
Add example Dockerfile mentioned in commit f3a90f9, which is based off
gentoo/stage3-amd64 with commands from gentoo/portage pasted in.

Would have been nice use-case for non-existent INCLUDE command
docker/docker#735 ...
2b14c59
@SJLC SJLC added a commit to VCTLabs/ansible-test that referenced this issue May 26, 2016
@SJLC SJLC Example Dockerfile for working Gentoo base image
Add example Dockerfile mentioned in commit f3a90f9, which is based off
gentoo/stage3-amd64 with commands from gentoo/portage pasted in.

Would have been nice use-case for non-existent INCLUDE command
docker/docker#735 ...
4e86447
@igorpejic

+1

@fengli79

INCLUDE +1

@duglin duglin added a commit to duglin/docker that referenced this issue Sep 24, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
597480b
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 24, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
aa76181
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 24, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
4204920
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 25, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
92a67fb
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 25, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
803dc10
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 25, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
ba5e593
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 26, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
7593f5f
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 27, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
ea4ab0d
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 27, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
6c49624
@bitsofinfo

+1

@nfedyk
nfedyk commented Sep 29, 2016

+1

@peloncano

+1

@duglin duglin added a commit to duglin/docker that referenced this issue Sep 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
e64aeb3
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
269336d
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
a61018c
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
a08c08f
@duglin duglin added a commit to duglin/docker that referenced this issue Sep 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
f40ba81
@duglin duglin added a commit to duglin/docker that referenced this issue Oct 2, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
0500d56
@duglin duglin added a commit to duglin/docker that referenced this issue Oct 2, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
1251fe7
@duglin duglin added a commit to duglin/docker that referenced this issue Oct 2, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
cbfac9a
@TalBeno
TalBeno commented Oct 11, 2016

It is mind boggling that there is no way to reuse building blocks i.e. collection(s) of layers per functional intent, at arbitrary positions within a dockerfile. With INCLUDE or any equivalent solution. What am I missing?

@Vanuan
Vanuan commented Oct 11, 2016

The thing I was missing previously is that each line in Dockerfile creates a new layer. Thus you should never have more than 5-10 lines in Dockerfile to keep image minimal.

@suntong
suntong commented Oct 11, 2016

Thus you should never have more than 5-10 lines in Dockerfile to keep image minimal.

Not necessary. I put in whatever necessary, then remove the intermediate containers/layers to save space & speed things up with the -rm switch

@thaJeztah
Member

@suntong the --rm flag only removes the intermediate container, but does nothing to optimize your image, also there's no need to specify it, as it's actually the default:

 --rm                      Remove intermediate containers after a successful build (default true)

Intermediate layers will still be added, so if (for example) you ADD a 100 megabyte file, and in the next step RUN rm big-file, your final image will still be 100 megabyte in size.

However, that discussion is separate from the discussion at hand here; an INCLUDE statement would only be for convenience and not make any difference other than that.

@bmudda
bmudda commented Oct 12, 2016

+1

@Vanuan
Vanuan commented Oct 12, 2016 edited

Yeah, it would be much more convenient to create thousands of layers and monolithic apps. Personally, I wouldn't want to see such Dockerfiles in the docker ecosystem.

Let's look a the use case.

It sounds like people want multiple inheritance here, to build monolithic images which use multiple languages and technologies.

AFAIK, the philosophy of docker ecosystem is to break down monolithic apps into distinct microservices by technology. So that you wouldn't need to import ruby Dockerfile into nodejs Dockerfile.

That what I believe is the rationale behind the hesitance to include INCLUDE: to enforce microservices architecture to keep Docker ecosystem a lightweight look and feel.

@suntong
suntong commented Oct 13, 2016

On Wed, Oct 12, 2016 at 11:27 AM, John Yani wrote:

It sounds like people want multiple inheritance here, to build a
monolithic image which uses multiple languages and technologies.

I don't agree. But anyway...

after all, "restricting how people use software" is what MS does best,
and what I dislike.

I believe, in essence, docker is meant to give people freedom,
not restriction.

@leethomas
leethomas commented Oct 23, 2016 edited

Since it doesn't look like this will ever be officially supported, I wrote a command line tool called harbor that suffices for my personal use cases.

While it supports recursive INCLUDEs, I've only tested it in moderation for my purposes. Haven't tried loading 1000+ files or anything. Posting it here in case anyone else might find it useful.

Obviously some directives should not be included multiple times in a single Dockerfile. Like FROM or CMD for example, so take caution there.

@duglin duglin added a commit to duglin/docker that referenced this issue Oct 29, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
a4f5e5e
@artheus
artheus commented Nov 28, 2016

+1

@duglin duglin added a commit to duglin/docker that referenced this issue Dec 30, 2016
@duglin duglin Support INCLUDE in Dockerfiles
Note that as of now this is just a syntax feature. Meaning it just pulls
in the included Dockerfile and continues parsing. If the target Dockerfile
has a FROM then it is processed and new image will be created.

After this we could consider adding a flag like:
   INCLUDE --no-from ...
which will tell it to skip the FROM so that the modifications will be done
within the same/current image.

Also, note that you can do:  INLCUDE myfile.${foo} where 'foo' is an
env var or a build arg

Closes: #735

Signed-off-by: Doug Davis <dug@us.ibm.com>
8bbb078
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment