Global .dockerignore

[crepererum]

Problem description

Users and developers use different editors and tool to edit, backup and process code, documentation and assets. It is not feasible to catch them all in a project's .dockerignore. Instead, a user should be able to blacklist the file of their own toolchain.

Proposed solution

Add a global .dockerignore, e.g. at $HOME/config/docker/dockerignore, which gets combined with the project's .dockerignore.

References

Docker is not the only tool with this problem. The problem is well known in the VCS world. GIT for example accepts the global core.excludesfile option which points to a global .gitignore. This file is combined with the projects .gitignore to blacklist more files.

Open questions

• Where should the config file be located on Windows?
• Should the global ignore file virtually appended or prepended to the local file? Does this make a huge different? If so, what are the implications and the pros and cons of the different strategies?

[duglin]

Not sure how I feel about this yet, but if we do have a global .dockerignore then it should probably be in the newly created $HOME/.docker config dir.

[crepererum]

@duglin Nope, docker should not create another directory directly under $HOME. Instead it should follow the XDG directory specification

[ahmetb]

@crepererum I think it already does for quite a while.

[jakirkham]

👍 Definitely, in favor a global .dockerignore. Would be very happy if this was similar to the way the global .gitignore file is specified.

[akash1810]

+1

This might be an osx specific response...

I use docker-osx-dev for folder sharing in dev as vboxsf is slow.

docker-osx-dev obeys the .dockerignore file.

In an ideal world, .dockerignore would be a symlink to .gitignore. However, this results in the .git directory being synced. So instead, I do:

cp .gitignore .dockerignore
echo '.git' >> .dockerignore

Having a global .dockerignore with .git inside would mean a project's .dockerignore can be a symlink to .gitignore, which would be a lot simpler.

-- edit --
I hadn't appreciated the options available with docker-osx-dev as this command achieves what I explained above:

docker-osx-dev -i .gitignore -e .git

[zimbatm]

@akash1810 the semantic of .dockerignore is close to but identical to .gitignore. For example /foo doesn't work in .dockerignore. foo means different things.

[zimbatm]

One use-case for a global .gitignore would be to set os-specific files like the .DS_Store which should not be a concern of the project.

[jakirkham]

^ exactly.

[pawelbaranski]

+1 for that

[joeblau]

I'm not to familiar with what Docker needs to ignore, but someone requested a new gitignore template for https://www.gitignore.io.

Is there a standard .dockerignore file that I can use for the template?

[thaJeztah]

@joeblau the .dockerignore is specifying which local files should be excluded when building an image (e.g, tmp files etc), however, it really depends on the project (if could be a Java, PHP, Python project), so I doubt it's possible to define a sensible default

[joeblau]

@thaJeztah that's what I thought. I feel like the template this person is asking for isn't possible. What they need is a custom template based on the .dockerignore which I could also generate.

[jakirkham]

Though there are some things though like the Readme, License, maybe .git that are often excluded.

[Jhorlin]

+1 I could really use a global .dockerignore.

[FranklinYu]

the .dockerignore is specifying which local files should be excluded when building an image (e.g, tmp files etc), however, it really depends on the project (if could be a Java, PHP, Python project), so I doubt it's possible to define a sensible default

Disagree. At least two things can be safely ignored globally, as mentioned above:

• OS-dependent temporary files, such as .DS_Store in macOS, and *~ in Linux with gedit.
• Version-control folders, such as .git of Git, .hg for Mercurial, and .svn for Subversion.

Whether the project be a Java, PHP, or Python project, you never want them in the image. Currently the only solution is to duplicate all those common entries in every .dockerignore.

[thaJeztah]

@FranklinYu my reply was to "Is there a standard .dockerignore file that I can use for the template?"

Although .git directories are often excluded, there are lots of examples where they are actually needed to build an image. So "safely" is not "safely" in all cases, so cannot be recommended to provide a default template.

[jemc]

I have a custom build system in Sublime text that runs a script in the project directory. This script is where I set up whatever various commands I want to be run when I hit Ctrl+B, and I tweak the script based on whatever I'm working on. It's just for me, and it's highly volatile, so I don't ever want it checked into version control or built into the docker image if I build one.

I have a convention of naming files that are "just for me" with a .jemc suffix so it's obvious to me that they're not meant to be shared, and I have a global .gitignore set up for them. It would be nice if I could do a global .dockerignore for them as well, because we obviously don't want to have to add *.jemc to the .dockerignore for our projects.

[dharmab]

Another use case for this is when using environment managers such as direnv. I have .envrc and .direnv in my global .gitignore because .envrc contains my environment variables (potentially containing API keys and other secrets) and .direnv contains pip packages for my editor (which just add useless files to a container). I have to add these two files to ever .dockerignore manually; a single place to manage them would be nice to have.

[Wirone]

Another use case is that I (and probably not only I) create .Stuff directory inside every project and I store there any project-related non-commitable things ONLY FOR ME. I don't want to add .Stuff to .dockerignore since this is not project's part, this is only my workflow convention to keep useful things easily accessible (from IDE for example).

With global .gitignore i can ignore those directories but for Docker I have to add it to .dockerignore (which is bad) or change convention (which I don't want).

[quinn]

this would be great, for people who use jetbrains, or whatever creates .idea everywhere. maybe ~/.config/docker/ignore or something?

[ktomk]

I miss this also, but I also found that adding .* at the very top of the .dockerignore file works pretty well for local builds. Your mileage may vary, explicit use of ! rules afterwards make certain context requirements more visible, too.

/e: To fully skip context, use stdin for docker build. Maybe building from a local git repo for context works as well, I've never tried it.

[FranklinYu]

@thaJeztah Is there any update? Or at least interest in reviewing PR?

[retnikt]

Any updates? It's been 5 years...

[Pictor13]

I'd also be interested in having updates.

I understand the reasoning about "each project is different".
However, what's globally .dockerignored is a choice up to the host machine owner. Same as with global .gitignore.
Like everything else on their machine, after all.

A team can surely agree on something.
But it's unrealistic for a public repo maintainer to address the needs of every single user, OS or environment.

If Git can do it, I (personally) don't see why Docker can't.

[ktomk]

@Pictor13 Likely because that are two different pair of shoes (and it's not the question of can or can't do - technically this is obviously trivial, which makes me wonder how it imposes any issues...).

So actually wondering about what drives you with your comment beating a dead horse.

Normally I would put the domain of the build more to the project (VCS/SCM) than to the systems side, so I'm wondering why I should let my system administrator configure that?

And have you considered so far to let git do this? As you write, git can do it, was there a blocker doing it with git?

[Pictor13]

So actually wondering about what drives you with your comment beating a dead horse.

What you call "dead" has actually no consensus, no clear decision, is still linked by #40319 as potential improvement and is still marked as open.

I just didn't want to ask for news writing a useless "Any updates? It's been 6 years...".

And have you considered so far to let git do this? As you write, git can do it, was there a blocker doing it with git?

I was talking of feasibility, as you also mentioned. A global .gitignore won't prevent locally added files from being sent to the build context.

[ktomk]

What you call "dead" has actually no consensus, no clear decision, is still linked by #40319 as potential improvement and is still marked as open.

@Pictor13 Sorry, using an idiom and what I meant is that from the three years or so ago when I was sharing with and contributing my comments to moby, somewhere between up to today, I stopped waiting for that feature and instead moved left and never had to return back - and doing so even if today this feature would be available would be a degradation for me. YMMV, this is with all due respect.

Reading your comment - which was not complaining! - and reading about your knowledge about git - I was wondering whether or not you gave it a try already and hit blockers.

A global .gitignore won't prevent locally added files from being sent to the build context.

That depends a lot how you create the directory tree (or tarball or git repository) you let docker build obtain the build context from. Both horizontally and vertically.

But at the end of the day it's the question whether you want to configure the build in the repository or in the system.

So most likely on the one hand I can imagine it appears not possible with git and on the other hand it could be also that it should not be done with git (perhaps the system then is managed with some git, so also git again ).

Choose your weapon first.

And yes,. please continue to demand that feature. It's very important, I at least once needed it myself, see above. So at least my past self still wants this!

And docker-build really needs a driver for git on the local file-system. So that it does not require network to build a revision from a local, non-bare repository.

[zimbatm]

For what it's worth, I think this issue should be closed. If that feature were introduced, it would cause even more confusion for users. Now instead of having reproducibility issues because too many files are added, you have the reverse where some users have .ignored too many files. People on issues now have to ask users about their global .gitignore to make sure it didn't interfere with files in the project.

The only way to be reproducible is to list each file that gets added individually.

[quinn-freshly]

@zimbatm i disagree, dockerignore should only include artifacts relevant to the project, not cruft only relevant to a particular engineer’s local dev env

I’ve never seen anyone asking about a users gitignore, but the only stuff that should go in there are .DS_Store, .idea, etc. Junk produced by your personal env. what are people doing :(

[pxeger]

@quinn-freshly I think zimbatm's concern is that (for example) if a user's global .dockerignore contains .env, but the .env file is needed for the project to build properly, then you will have a whole class of strange bugs. Even if putting .env there is a bad idea, people may still do it and it will be hard to diagnose. This is a good point, but I don't think it means this shouldn't be added at all. (maybe ! .env in a local .dockerignore could override it?)

[zimbatm]

This idea is probably inspired by git. But git is not the same because the list of files is easily introspectable since it's part of the commit. If a user tries to add a file that has been globally ignored they get a notice, and can force it with git add -f.

In docker's case, the docker client compiles a tarball on the fly and sends that to the daemon along with the build instructions. That tarball is not exposed and the developer inspects the resulting image to find out what was added instead.

[thaJeztah]

In docker's case, the docker client compiles a tarball on the fly and sends that to the daemon along with the build instructions.

Somewhat orthogonal to the discussion, but I should add here that "client compiles a tarball" is indeed how the "classic" builder works. When using BuildKit as builder (set DOCKER_BUILDKIT=1), files are sent through a session between the daemon and client. When using BuildKit, the daemon-side builder will parse the Dockerfile and request files from the client when needed. This feature may (depending on your situation) remove (or reduce) the need for using a .dockerignore file if COPY and ADD instructions in your Dockerfile are specific enough.

For example, given the following "project":

mkdir -p my-project/subdir && cd my-project
touch file1.txt file2.txt subdir/file3.txt subdir/file4.txt

And this Dockerfile:

FROM alpine
RUN mkdir -p /app/foo/

# This only sends file1.txt from the client, and ignores file-changes in other
# files. As a result, these steps are only executed if `file1.txt` changes, and
# will be cached otherwise.
COPY /file1.txt /app/
RUN cat /app/file1.txt

# This only sends files inside "subdir", and will be cached if those files did
# not change (and if the steps above were cached).
COPY /subdir/*.txt /app/foo/
RUN cat /app/foo/*.txt

After doing an initial build:

docker build .

[+] Building 1.4s (11/11) FINISHED
...
 => => writing image sha256:7dbaefbed0389e2c52dff7138dc3a28456f55d6741a4baac94fef1d2ad757286  0.0s

Adding new files, or changing files that are not referenced by the Dockerfile will not affect the build:

mkdir subdir2
touch new-file.txt subdir2/another-new-file.txt
echo "changed" >> file2.txt

And all steps will still be cached:

docker build .
...
 => CACHED [2/6] RUN mkdir -p /app/foo/                                                       0.0s
 => CACHED [3/6] COPY /file1.txt /app/                                                        0.0s
 => CACHED [4/6] RUN cat /app/file1.txt                                                       0.0s
 => CACHED [5/6] COPY /subdir/*.txt /app/foo/                                                 0.0s
 => CACHED [6/6] RUN cat /app/foo/*.txt                                                       0.0s
 => exporting to image                                                                        0.0s
 => => exporting layers                                                                       0.0s
 => => writing image sha256:7dbaefbed0389e2c52dff7138dc3a28456f55d6741a4baac94fef1d2ad757286  0.0s

[thaJeztah]

I agree with most of what's said above, and understand use-cases such as excluding common files (like .DS_Store, thumbs.db etc). While implementing a global ignore file would (likely) not be too difficult from a technical perspective, it's primarily the side-effects of such a feature that need to be looked at (some of which mentioned above);

• reproducibility; the presence of a global .dockerignore can easily lead to "it (doesn't) work on my machine" situations; identical source repositories producing different results on different machines because files were excluded.
• discoverability: docker (currently) doesn't have a way to easily discover what files were excluded (and by which "exclude" rule); this can either lead to cryptic errors (file doesn't exist), or (worse) a build that is "seemingly" successful, but where files were excluded that are needed at runtime.
• in general; .dockerignore is a bit of a two-edged sword. While there are some good reasons to use a .dockerignore, it also results in Dockerfiles not being as "declarative" as they should be; while this is already a road taken (a .dockerignore will be shared by all Dockerfiles in a source repository), adding the concept of a global .dockerignore digs in deeper into that situation.

For the second bullet, a command to debug what's excluded (similar to git check-ignore) could assist (I listed this as an option in an epic I created on improving .dockerignore here: #40319). In addition, when using BuildKit as builder, the error messages could be improved to mention that a file was excluded (see moby/buildkit#1647). Note that improving the error messages would not help with situations where the build is successful, but the image fails at runtime due to files missing.

So, from the above, I think the first focus should be on making excluded files more discoverable / debuggable before (deciding to) adding a global ignore.

[rvandegrift]

I agree that discoverability is an issue, and I think your proposal would be a nice solution. The BuildKit example looks promising too, thanks for the pointer! But the other two concerns you raised don't seem like barriers to this feature.

reproducibility; the presence of a global .dockerignore can easily lead to "it (doesn't) work on my machine" situations; identical source repositories producing different results on different machines because files were excluded.

The global dockerignore seems orthogonal to reproducability. There's two cases to consider:

• controlled environments (ci pipelines, clean build chroots, etc): supporting a global dockerignore doesn't harm here, since a controlled environment won't have one. So reproducibility isn't negatively impacted.
• "my machine" environments: builds already use local state, and the only way to troubleshoot differences is to compare logs & images. So a global dockerignore doesn't make the current situation worse - assuming a solution to the discoverability problem, of course.

in general; .dockerignore is a bit of a two-edged sword. While there are some good reasons to use a .dockerignore, it also results in Dockerfiles not being as "declarative" as they should be; while this is already a road taken (a .dockerignore will be shared by all Dockerfiles in a source repository), adding the concept of a global .dockerignore digs in deeper into that situation.

I don't follow this concern. A Dockerfiles is a recipe, executed in order, to produce an artifact. It seems inherently non-declarative - and so not made worse by a global dockerignore.

[bkahlert]

❯ docker run -it --rm --entrypoint=/bin/bash test
root@82a2f7cc5f2d:/# ls / -lisa
total 68
 668250 4 drwxr-xr-x   1 root root 4096 Sep 20 21:53 .
 668250 4 drwxr-xr-x   1 root root 4096 Sep 20 21:53 ..
 668212 8 -rw-r--r--   1 root root 6148 Sep 20 21:49 .DS_Store

Definitely in need of a global .dockerignore.

[klondikemarlen]

Not sure if this is the right place to request docker features? But I have another example.

I'm using typescript, so the most practical way to have a scratch file that actually runs, is to put it inside the /src folder of the project. But, of course, this means that said scratch file will be injected into the built image.

Since I use said /src/scratch.ts file for for all sorts of stuff, it often has credentials and things I really don't want in the build image. It's fine that I can add it to the .dockerignore, but what if I forget on a new project? Or what if someone else removes it and I don't notice? A global .dockerignore would provide me with a lot of comfort and safety.

(I use a global .gitignore to stop said file from being committed, obviously).

[klondikemarlen]

I'm thinking a workaround would be to add a macro around the docker/docker compose command, so that it
a. finds the relevant dockerignore file
b. finds the relevant global dockerignore file
c. creates a temporary dockerfile specific .ignore that injects the global options
d. resets or deletes the temporary dockerfile

[bersace]

Could docker build also read ~/.config/git/ignore ?