Exception patterns in .dockerignore do not support wildcard directories

[taylortrimble]

Description

Exception patterns in .dockerignore do not support wildcard directories. I expect to be able to write exception rules like !*/*.txt.

Steps to reproduce the issue:

1. Assume the following files in the Docker context:

$ find . -type f
./.dockerignore
./Dockerfile
./hi.txt
./ignoreme.txt
./photos/steak.jpeg
./recipes/cake.txt

2. Assume the following .dockerignore (specs):

*
!hi.txt
!photos/*.jpeg
!*/*.txt

3. Assume the following Dockerfile:

FROM debian:8.5
WORKDIR /srv
COPY . ./

4. Build the Docker image and run find . from it:

$ docker build -t test-di .
$ docker run --rm test-di find . -type f

Describe the results you received:

Output of docker run:

./photos/steak.jpeg
./hi.txt

./recipes/cake.txt, matching the exception rule !*/*.txt, is missing.

Describe the results you expected:

./photos/steak.jpeg
./recipes/cake.txt
./hi.txt

./recipes/cake.txt, matching the exception rule !*/*.txt, is present.

Additional information you deem important (e.g. issue happens only occasionally):

This change was likely introduced by #20872. cc/ @duglin @icecrime @vdemeester

Output of docker version:

Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 06:14:34 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:02:53 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 145
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:0-128926-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 7.207 GB
 Data Space Total: 107.4 GB
 Data Space Available: 41.11 GB
 Metadata Space Used: 10.95 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.137 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.8.3-x86_64-linode76
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.856 GiB
Name: ivysaur.tntapp.co
ID: MYEL:ZPVM:FQYY:JATZ:VPK3:L25L:3JVD:LNXO:J4XP:HSFV:6VY4:YJIY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

This issue should affect only the client.

[duglin]

try: ! **/*.txt

[taylortrimble]

The special ** wildcard doesn't work, either. I believe this is because of the logic changed by #20872. Specifically, it only prevents a filepath.SkipDir if the file contains exceptions and the exclusion pattern (sans !, plus / at the end) contains the prefix relFilePath + string(filepath.Separator).

So with that, here's how the bug happens in pkg/archive.TarWithOptions based on my example in the description:

• It filepath.Walks to the recipes directory
• recipes gets excluded by the * rule
  • and none of the exceptions add the recipes directory back explicitly
• There are exceptions, continue
• Construct dirSlash
  • dirSlash := relFilePath + string(filepath.Separator)
  • dirSlash = recipes/ in this example
• Process pat
  • pat = pat[1:] + string(filepath.Separator)
  • pat is converted from !*/*.txt to */*.txt/
• Check the prefix match
  • Does pat contain the prefix dirSlash?
  • a.k.a. does */*.txt/ contain recipes/
  • No, it does not; the bug appears to lie in the reliance upon prefix matching, then

Note that if the pat had been !recipes/*.txt, the processed pat would have been recipes/*.txt/, which would have had the prefix dirSlash (recipes/).

This is the heart of the bug. Directories can't be used as wildcards for exception rules because of this prefix matching business.

[enkoder]

This issue still exists... Any update on this?

.dockerignore

*
!**/*.txt

Dockerfile

FROM alpine

RUN apk update && apk add tree
ADD . /stuff

CMD tree stuff

tree output

.
├── dir1
│   ├── dir2
│   │   └── file2.txt
│   ├── file1.py
│   └── file1.txt
└── Dockerfile

docker run $(docker build --no-cache -q .) output

stuff

0 directories, 0 files

[Mobe91]

This is a pain in the ***.

[ccbrown]

This is a frustration every single time I want to make a Docker build for my projects using a whitelist-style dockerignore. Maybe I've just been using Docker wrong from the start, but I don't understand how dockerignore files, a tremendously simple, yet fundamental feature of Docker, can just be left broken and neglected for literally years.

Are whitelist-style dockerignore files simply an unsupported use-case?

Also, this seems like it might be the same as this issue: #23693

[weisjohn]

Are there any work-arounds to this?

[Frondor]

Worst thing is; there's no workaround for this. I've been manually adding each directory to it.
I have a /dir/Dockerfile which composer uses with ./ context and every time I create a new folder and I forget to add it to the .dockerignore, it gets in the image until someone notices it.

Why is it so hard to make this file to behave just like .gitignore ?

[dinvlad]

Any updates on this issue?

[dimaqq]

Why is it so hard to make this file to behave just like .gitignore ?

A wild guess: original thinking was to include whatever functionality go provided natively.

Perhaps it's time to reconsider, docker build is used together with git a lot, it would be awesome if same format was supported! One may even hope for .gitignore to be picked up automatically fi it exists, though that could break dirty builds (git checkout, build artefacts in-place, docker build)... I'd welcome this though!

[dinvlad]

I'd second that. An ability to import a given .gitignore (as there might be one at each folder level in a repo) would be hugely helpful here, as it would drastically reduce config repetition and the potential to overlook something. It should still be paired with an ability to override or prohibit certain patterns from .gitignore.

[krhubert]

Any Updates?

[thaJeztah]

created an "epic" to collect related issues; #40319

[AceHack]

I'm also having this issue, please fix

[arvenil]

The only thing I wish to COPY is Makefile and *.go files.

*
!Makefile
!**/*.go
!vendor/modules.txt
!go.mod
!go.sum

(well, go.mod as well)
This doesn't seem to be possible right now :(

[chrisdonahue]

I would also love to use such a feature 👍 . Use case is the same as @arvenil 's, i.e., it would be much easier to keep track of what I'm sending to the build context using an allow list rather than a deny list.

[Yuyan-Li]

The only thing I wish to COPY is Makefile and *.go files.

*
!Makefile
!**/*.go
!vendor/modules.txt
!go.mod
!go.sum

(well, go.mod as well)
This doesn't seem to be possible right now :(

I may have a workaround for you:

**/*.*
!**/*.go
!vendor/modules.txt
!go.mod
!go.sum

This will copy all directories and files without extensions. The exclusions work if the directories are not ignored. So you only have to add files without extensions (like Dockerfile) to the ignore list.

[samuelbsoares]

@Yuyan-Li I ended up doing something similar but hit another limitation, any files with more than one dot on its name will be missed:

.
├── dir1
│   ├── dir2
│   │   └── file2.go
│   └── file1.file1.go       <---- Will be missed
...