Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Dockerfile add INCLUDE #735

Closed
dysinger opened this issue May 28, 2013 · 145 comments
Closed

Proposal: Dockerfile add INCLUDE #735

dysinger opened this issue May 28, 2013 · 145 comments

Comments

@dysinger
Copy link

@dysinger dysinger commented May 28, 2013

No description provided.

@shykes
Copy link
Contributor

@shykes shykes commented Jun 21, 2013

+1

1 similar comment
@keeb
Copy link
Contributor

@keeb keeb commented Jun 21, 2013

+1

@ptone
Copy link

@ptone ptone commented Jun 21, 2013

Yes this would be great to see +1

@jpfuentes2
Copy link

@jpfuentes2 jpfuentes2 commented Aug 9, 2013

I think this would be a great feature as I want to leverage some of my knowledge/experience with systems like Chef whereby you can compose complex builds using smaller/simpler building blocks.

Does anyone have a way they're implementing this now?

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Aug 11, 2013

Can someone give me a few examples on how they would use this? ping @ptone @keeb @dysinger

@ptone
Copy link

@ptone ptone commented Aug 12, 2013

Lets say I have build file snippets for different components of web architecture.

I may want to include 1 or more of them on a single container image - or in general bundle things that always go together.

Simple examples might be nginx and varnish always go on the same container, or redis and pyredis.

I might have a "Scientific Python" list of python packages and linked libs, that I may want to include in other images.

The problem with FROM and base images is that you can't remix things the same way you can with includes. A base image is all or nothing - if you don't want something, your only hope is that you can go back to a 'lower' base image, and re-add stuff manually that you want, skipping the stuff you don't.

It is essentially a case of inheritance vs composition in many ways.

@binocarlos
Copy link

@binocarlos binocarlos commented Aug 29, 2013

+1 to this - @crosbymichael @ptone @keeb @dysinger @shykes

I am at this exact point where I have an appserver base image (which is just node.js).

I also have a collection of Dockerfiles representing little bits of services that have deps - so:

ImageMagick:

from appserver
run apt-get install imagemagick

RedisSession:

from appserver
run apt-get install redis-server

To have a container that is both ImageMagick and RedisSession:

from appserver
run apt-get install imagemagick
run apt-get install redis-server

Whereas the following syntax means I can build up a folder of modules and include them by name in the application Dockerfile:

from appserver
include ./modules/imagemagick
include ./modules/redis-server

Now, because Docker is so darn brilliantly good : ) this is currently trivial to do (i.e. read module file and inject here) and so I'm not sure if for me this is a required feature.

However - it would make a user app's Dockerfile (i.e. the last thing in the chain for this scenario) much more about composing modules together than about creating a strict inheritance tree where some combinations are not possible - my 2 cents on what is a joy to work with otherwise :)

@Thermionix
Copy link
Contributor

@Thermionix Thermionix commented Sep 11, 2013

+1 would also be good to reference out generic blocks (possibly to a github raw link?)

@frankscholten
Copy link

@frankscholten frankscholten commented Oct 3, 2013

+1

flavio added a commit to flavio/docker that referenced this issue Oct 15, 2013
Added the 'include' command to dockerfile build as suggested by issue moby#735.
Right now the include command works only with files in the same directory of
the main Dockerfile or with remote ones.
@joelreymont
Copy link

@joelreymont joelreymont commented Oct 21, 2013

I'll re-implement this on top of #2266.

@prologic
Copy link
Contributor

@prologic prologic commented Feb 10, 2014

+1 Turns Docker and the Dockerfile into a portable rudimentary configuration management system for the constructions of portable Docker images :)

@ghost
Copy link

@ghost ghost commented Feb 14, 2014

This would help a lot. +1

@deeky666
Copy link

@deeky666 deeky666 commented Feb 26, 2014

+1

@peenuty
Copy link

@peenuty peenuty commented Mar 10, 2014

+1

1 similar comment
@newhoggy
Copy link

@newhoggy newhoggy commented Mar 10, 2014

+1

@jfinkhaeuser
Copy link

@jfinkhaeuser jfinkhaeuser commented Mar 11, 2014

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also get the need for parametrized Dockerfiles, and then for conditionals, etc. Continue down that path, and you'll end up implementing a programming language in Dockerfiles, which may even become turing complete. The cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program your own, you could use m4 (used in autotools for this purpose) or the C preprocessor (used in IMake, which does a similar job as autotools but is pretty much defunct these days).

Makefile:

Dockerfile: Dockerfile.in *.docker
  cpp -o Dockerfile Dockerfile.in

build: Dockerfile
  docker build -rm -t my/image .

Dockerfile:

FROM ubuntu:latest
MAINTAINER me

#include "imagemagick.docker"
#include "redis.docker"

Run make and it'll re-build the Dockerfile if any input file changed. Run make build and it'll re-build the Dockerfile if any input file changed, and continue to build the image.

Look ma, no code!

@prologic
Copy link
Contributor

@prologic prologic commented Mar 11, 2014

On Tue, Mar 11, 2014 at 6:45 PM, Jens Finkhaeuser
notifications@github.comwrote:

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also
get the need for parametrized Dockerfiles, and then for conditionals, etc.
Continue down that path, and you'll end up implementing a programming
language in Dockerfiles, which may even become turing complete. The
cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program
your own, you could use m4 (used in autotools for this purpose) or the C
preprocessor (used in IMake, which does a similar job as autotools but is
pretty much defunct these days).

I'm in agreement with this. Having toyed with design and implemtantions of
a few languages myself over the years
turning Dockerfile(s) into a "scripting" language even if it's not turning
complete sounds like something that Docker should not do.

As Jens clearly points out there are better more appropriate tools for this
job.

cheers
James

James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

@hunterloftis
Copy link

@hunterloftis hunterloftis commented Apr 9, 2014

+1

The slippery-slope argument for a turing-complete scripting language in Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just lets you create a common base image locally so you can reference a file system (for example, in your app's repository) instead of relying on the Docker registry for what should be a self-contained app definition.

@prologic
Copy link
Contributor

@prologic prologic commented Apr 9, 2014

On Wed, Apr 9, 2014 at 12:15 PM, Hunter Loftis notifications@github.comwrote:

The slippery-slope argument for a turing-complete scripting language in
Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just
lets you create a common base image locally so you can reference a file
system (for example, in your app's repository) instead of relying on the
Docker registry for what should be a self-contained app definition.

I don't agree with this. The very notion of referencing and building a base
image is already there
and it only accesses the public registry (or a private onf if you're so
inclined) if you don't have said image.

I'm still -1 on this -- it adds more complexity for little gain. I'd rather
see Docker pick up some YAML-style
configuration format for "configuring one or more containers" ala fig et
all.

cheers
James

James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

@jfinkhaeuser
Copy link

@jfinkhaeuser jfinkhaeuser commented Apr 9, 2014

The slippery-slope argument stems from experience with a bunch of other DSLs. There's a general trend for DSLs to become turing complete over time.

The include statement in itself presents little danger here, but consider that in almost every language, include or import statements are linked to the concept of an include path, quite often set via an environment variable.

There's a reason for that: having include statements means you can collect building blocks into reusable libraries, and having reusable libraries means you'll want to use the same building blocks in multiple Dockerfiles. The best way to do that is to be agnostic to the actual file location in your include statement, but instead have a "canonical" name for a building block, which is usually a file path relative to some externally provided include path.

So what's wrong with that? Nothing, except (and I'm quoting you here):

(...) instead of relying on the Docker registry for what should be a self-contained app definition.

I agree. A Dockerfile should be a self-contained app definition. Once you include stuff from any location that's shared between projects, though, you have anything but that - the needs of one project will lead to modifications of the shared file and those may not reflect the needs of the other project any longer. Worse, any kind of traceability - this version of the Dockerfile should build the same image again - is completely gone.

Besides, once you have re-usable building blocks, parametrizing them is the obvious next step. That's how libraries work.

@hunterloftis
Copy link

@hunterloftis hunterloftis commented Apr 9, 2014

Then follow a common, well-understood example that has certainly not become turing complete:

https://developer.mozilla.org/en-US/docs/Web/CSS/@import

@ryedog
Copy link

@ryedog ryedog commented Apr 14, 2014

+1 on include (or even variables that can be declared on the command line on build would be helpful)

@ChristianKniep
Copy link

@ChristianKniep ChristianKniep commented Apr 23, 2014

+1 on include as well

I have to deal with one site where I have to set http_proxy and one without.

@kkm000
Copy link

@kkm000 kkm000 commented Mar 29, 2019

I am mildly missing this feature for testing multiple platfrorms. This would minimize maintenance, I absolutely do not want to copy/paste common stuff. Currently, bash and cat is my preprocessor. Not super important, but would be nice to have. Nothing Turing-complete here:

FROM opensuse/amd64:13.2
INCLUDE prereq-suse.inc
INCLUDE common.inc

This pattern (if INCLUDE were a thing) would work in my case for opensuse, debian, ubuntu, centos and partly Fedora ("old" Fedora/"new" Fedora single split).

I absolutely do not buy the argument that adding the INCLUDE statement to your declarative language inevitably leads to Turing-completeness, obesity, fleas on your dog and crime in your neighborhood. This is a stretch far and thin.

@borgogelli
Copy link

@borgogelli borgogelli commented Apr 13, 2019

+1

@anton-matosov
Copy link

@anton-matosov anton-matosov commented Apr 28, 2019

Tensorflow team have implemented this functionality using an assembler.py python script
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/dockerfiles which assembles partial dockerfiles and the rules in spec.yml

Edit:
Here is an RFC describing their approach in details https://github.com/tensorflow/community/blob/a8f3baec6c90197d697796004d6352fc942c4bf1/rfcs/20180731-dockerfile-assembler.md

@hholst80
Copy link

@hholst80 hholst80 commented May 18, 2019

It might not be obvious to everyone how to use the multi stage builds to work around the missing INCLUDE. The example below should be fairly self explanatory. Another obvious thing is the inefficiency that the staging build steps are always done even for "production" images.

See also https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

ARG env=production

FROM nginx:alpine AS base

FROM base AS production
COPY nginx.conf /etc/nginx/nginx.conf

FROM base AS staging
COPY nginx.no-ssl.conf /etc/nginx/nginx.conf

FROM $env
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented May 18, 2019

Another obvious thing is the inefficiency that the staging build steps are always done even for "production" images.

Try setting DOCKER_BUILDKIT=1, because that's only the case when using the classic builder; when using buildkit, those stages will be skipped if they're not needed to produce the final stage

@zh794390558
Copy link

@zh794390558 zh794390558 commented May 21, 2019

+1

1 similar comment
@MBeijer
Copy link

@MBeijer MBeijer commented Jun 25, 2019

+1

@vrothberg
Copy link

@vrothberg vrothberg commented Jul 3, 2019

Buildah supports including other Dockerfiles since 2018. To avoid extending the Dockerfile syntax and remain compaitble, it's using the C pre-processor to assemble the Dockerfile (see the Buildah docs). Buildah only runs cpp on a Dockerfile if it's suffix is .in. Other Dockerfiles can be included via the #include $PATH directive.

Certainly, the C pre-processor allows writing macros and we can do horrible things with it, but it's used and understood since the 60s and present on pretty much all systems, giving a certain degree of portability. Note that this method does not require Buildah at all and it can be used by pretty much everybody trying to implement include-semantics for Dockerfiles. Simply run cpp -E -iquote . Dockerfile.in and the output can be parsed by any Dockerfile-compatible tool.

@tanzislam
Copy link

@tanzislam tanzislam commented Dec 24, 2019

If you're using the preprocessor in the latest versions of GCC for this purpose, then you'd need the following switches:

  • -traditional to prevent GCC from removing trailing backslashes in non-directive lines.
  • -undef to prevent replacing any occurrences of the word "linux" in your Dockerfile with "1", etc.

The cpp preprocessor in Xcode Command-line Tools also accepts these switches without any behavior changes.

vrothberg added a commit to vrothberg/buildah that referenced this issue Dec 24, 2019
@tanzislam noted [1] that Buildah should be using the -traditional flag
to prevent GCC from removing trailing backslashes in non-directive lines
and the -undef flag to prevent built-in macros from expansion (e.g.,
"linux" to "1").

[1] moby/moby#735 (comment)

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
vrothberg added a commit to vrothberg/buildah that referenced this issue Dec 24, 2019
@tanzislam noted [1] that Buildah should be using the -traditional flag
to prevent CPP from removing trailing backslashes in non-directive lines
and the -undef flag to prevent built-in macros from expansion (e.g.,
"linux" to "1").

[1] moby/moby#735 (comment)

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
vrothberg added a commit to vrothberg/buildah that referenced this issue Jan 3, 2020
@tanzislam noted [1] that Buildah should be using the -traditional flag
to prevent CPP from removing trailing backslashes in non-directive lines
and the -undef flag to prevent built-in macros from expansion (e.g.,
"linux" to "1").

[1] moby/moby#735 (comment)

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
rh-atomic-bot added a commit to containers/buildah that referenced this issue Jan 6, 2020
@tanzislam noted [1] that Buildah should be using the -traditional flag
to prevent CPP from removing trailing backslashes in non-directive lines
and the -undef flag to prevent built-in macros from expansion (e.g.,
"linux" to "1").

[1] moby/moby#735 (comment)

Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

Closes: #2044
Approved by: TomSweeneyRedHat
@ringerc
Copy link

@ringerc ringerc commented Apr 6, 2020

Spell it as FROM PATH if you're unhappy with calling it "include". But please, please consider adding this, the lack is painful and forces higher levels of a stack to do direct Dockerfile munging when they should be able to consider a single Dockerfile + context dir to be an authoratative source.

@dougmoscrop
Copy link

@dougmoscrop dougmoscrop commented Apr 6, 2020

I still think this could be accomplished in a way that is (hopefully) compatible with the current APIs as long as the containers shared an ancestry, i.e. someone should be able to do

FROM alpine
INCLUDE alpine/xyz
INCLUDE alpine/abc
blah blah

at that point the alpine/xyz just becomes a way of factoring out commons steps. could xyz install something and abc needs a different version? of course, but that's no different than building a container that has two separate package install steps - in fact, it becomes that, it's just about DRYing the canonical build steps for e.g. java, node, etc.

@klakegg
Copy link

@klakegg klakegg commented Sep 13, 2020

This is a crucial feature when moving forward with multiple stages/targets/architectures, and has finally grown into an itch I simply had to scratch.

My humble solution is preprocessing of the Dockerfile so a few lines of code may do some imports and update some references in the imported instructions. It is available here:
https://github.com/klakegg/dockerfile-import

This is not a perfect solution, however I guess it may relieve some of the pain until it is fixed in Docker.

@tantal92
Copy link

@tantal92 tantal92 commented Nov 18, 2020

another scenario is when you have many images, and in every image you need to set up the same set of ENV variables;
(e.g.: INCLUDE file_with_my_envs)

@edrevo
Copy link
Contributor

@edrevo edrevo commented Dec 30, 2020

I have implemented a custom buildlkit frontend that supports a new "INCLUDE+" command. You can use it like this on recent Docker builds:

# syntax = edrevo/dockerfile-plus

FROM alpine

INCLUDE+ Dockerfile.common

ENTRYPOINT [ "mybin" ]

More info at https://github.com/edrevo/dockerfile-plus

@cseufert
Copy link

@cseufert cseufert commented Feb 23, 2021

Even if it was just to be able to reference common build files for multi-stage builds

eg

FROM build-env --dockerfile=./Dockerfile.build-env as build
@eng-dibo
Copy link

@eng-dibo eng-dibo commented Apr 22, 2021

a use case is when you need to create an image from a base image, in this case you need to build the base image first.

but if you want to use a continous deployment service that needs to build your image when pushing a new commit to the head branch of your code base, like 'Google cloud run', it requires you to provide it with the Dockerfile path in your code base and it do the build process.

you don't have the option to build the base image first, before building the app image.

here you need to include the base Dockerfile rathar than building from the base image.

@hexcowboy
Copy link

@hexcowboy hexcowboy commented Jun 3, 2021

Here's a new project I'm working on with a great use case for this:

@endrift
Copy link

@endrift endrift commented Jul 28, 2021

It's really telling the number of different use cases that have been presented here and genuine grievances, which have been brushed off with non-obvious workarounds, forks, and other "are we out of touch? No, it's the user who is wrong" nonsense endemic to the tech industry these days. The people downvoting all the +1s are just being passive aggressive without considering that maybe the reasons are valid. It's really disappointing, even if not surprising, just how little people involved with the project seem to care.

I have a use case. So what did I do? Copy pasted a bunch of stuff and now I have to update 4 different files any time the stuff in the middle changes. Thanks, Docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.