Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker 18.06 docker build "can't stat" ignored directory that belongs to another user with 700 permission #380

Open
2 of 3 tasks
moznuy opened this issue Jul 31, 2018 · 32 comments

Comments

@moznuy
Copy link

moznuy commented Jul 31, 2018

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Docker ignores directory which is specified in .dockerignore and proceeds with building an image.

Actual behavior

Docker reports an error: error checking context: 'can't stat '.../DB''.

Steps to reproduce the behavior

docker -v
printf "FROM alpine:latest\n" > Dockerfile
printf "DB\n" > .dockerignore
mkdir DB
sudo chown root:root DB
sudo chmod 700 DB
docker build -t test .

Version 18.03 output:

Docker version 18.03.1-ce, build 9ee9f40
Sending build context to Docker daemon   5.12kB
Step 1/1 : FROM alpine:latest
 ---> 11cd0b38bc3c
Successfully built 11cd0b38bc3c
Successfully tagged test:latest

Version 18.06 output:

Docker version 18.06.0-ce, build 0ffa825
error checking context: 'can't stat '/REDACTED/DB''.

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Wed Jun 20 21:43:51 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Wed Jun 20 21:42:00 2018
  OS/Arch:      linux/amd64
  Experimental: false
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:09:54 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:07:56 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 11
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-29-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.709GiB
Name: SCharykovPC
ID: DUGK:34M2:G6EY:CK5R:ARHF:ISVB:XKB3:FWCF:5AYK:GK5P:3IBO:TOMS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 11
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-29-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.709GiB
Name: SCharykovPC
ID: DUGK:34M2:G6EY:CK5R:ARHF:ISVB:XKB3:FWCF:5AYK:GK5P:3IBO:TOMS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

@Noah-Huppert
Copy link

I am also encountering this issue.

I am developing an application which requires a database for development. I am running a local Postgres instance in a Docker container. I mount this Docker container's data directory on my local file system so that the database is persisted between shutdowns. I have added the Postgres container's data directory to my .dockerignore.

When I try to build my application's Docker image I get the error from the original bug report. Even though I have .dockerignored the directory Docker still tries to stat it.

@AngelOnFira
Copy link

AngelOnFira commented Nov 6, 2018

I'm running into this as well, same situation as @Noah-Huppert . I came up with a fix that works for now, which is storing the Postgres data folder inside of another folder, and .dockerignoreing that. Janky solution though.

@Noah-Huppert
Copy link

Thanks @AngelOnFira, that's actually really clever.

@Francesco149
Copy link

i'm experiencing the same thing with docker's root directory. I have a dir structure like

/vault
  Dockerfile
  .dockerignore
  docker

the dockerignore contains

*

and /vault/docker is my docker root dir (runs with -g /vault/docker )

when building i get error checking context: 'can't stat '/vault/docker''

and yeah i can work around this by moving /vault/docker into a nested dir

@alekratz
Copy link

I am experiencing this issue with this version of Docker:

Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

The "put it in a subdirectory and ignore the subdirectory" workaround works for me.

@miend
Copy link

miend commented Feb 5, 2019

I am experiencing this issue on Docker version:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:23 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Again with a postgres DB volume included in the .dockerignore, but still statted by Docker. Nesting my pgdata directory inside another directory (data) and adding data/ to the .dockerignore successfully worked around the issue.

@krizex
Copy link

krizex commented Feb 19, 2019

Encountered the same issue on
Docker version 18.09.1, build 4c52b90

@sweber83
Copy link

sweber83 commented Mar 8, 2019

Stumbled on this issue, too using docker version:
18.09.2, build 624796

@bartfeenstra
Copy link

I believe I am running into the same problem. Not with a database, but with the containerized application's private file storage.

$ docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:44:15 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false
$ cat .dockerignore 
/app/private
$ ls -la ./app
total 400
drwxr-xr-x 13 bart bart   4096 Apr 30 15:40 .
drwxr-xr-x  7 bart bart   4096 Apr 30 16:25 ..
drwx------  3   82   82   4096 Mar 19 13:41 private

Partial docker build output that includes the relevant error:
error checking context: 'can't stat '(...)/app/private''.

@shukriadams
Copy link

shukriadams commented Aug 26, 2019

For me this was caused by running

docker build -t something . 

in a folder with a lot of messy folders that were linked to other containers etc. Fixed by moving Dockerfile to a clean, dedicated build folder of its own.

@cpuguy83
Copy link
Collaborator

cpuguy83 commented Sep 7, 2019

But what does the dockerfile look like? I cannot repro this.

@moznuy
Copy link
Author

moznuy commented Sep 10, 2019

But what does the dockerfile look like? I cannot repro this.

From the second step of "Steps to reproduce the behavior":

printf "FROM alpine:latest\n" > Dockerfile

So the example Dockerfile looks like this:
FROM alpine:latest

@cpuguy83
Copy link
Collaborator

Ok, I think I see why.
The ignore syntax doesn't really do recursive ignores so it tries to stat it to read what's inside.

@rupeshtech
Copy link

in my case following was root cause and soltuions
Cause:

  1. When I delete a project from soltuion (thru visual studio), Folder (which was supposed to be deleted) was locked
  2. Even deleting that folder thru Command promt or powershell was not happening

Solution:
A system restart and the deleting the folder works.
Now not getting error.

@jordiclariana
Copy link

Still happening:

$ docker version
Docker version 19.03.3, build a872fc2f86

@VadimN92
Copy link

VadimN92 commented Nov 7, 2019

Just add file .dockerignore with "node_modules" inside

@ndik11
Copy link

ndik11 commented Nov 20, 2019

The same
Server Version: 19.03.5

@gregfr
Copy link

gregfr commented Nov 22, 2019

I have the same problem but adding the culprit to .dockerignore has no effect

@moznuy
Copy link
Author

moznuy commented Nov 22, 2019

So based on the fact that this problem still exists I'll explain solution that work for me for the last year on different projects:

The problem for me was with database storage volume.
I move directory that needs to be ignored one folder above building context(project directory in my case), because its presence is not required in project structure. (It used to be also git ignored before)

@chrisghill
Copy link

Also running into this. Would be nice if .dockerignore respected this.

@guillaumedsde
Copy link

guillaumedsde commented Jul 6, 2020

I have the same problem but adding the culprit to .dockerignore has no effect

Same issue here

@thaJeztah
Copy link
Member

Slightly related docker/buildx#1781

@bolner
Copy link

bolner commented Jul 15, 2020

For me the workaround was to stop using dockerignore and do a COPY for specific folders and files.

The main reason for this feature would be to avoid including problematic files/folders in the image. But it dies when an ignored file is problematic, like when it belongs to a non-existent user ID.

@unni12345
Copy link

sudo chown -R $USER:$USER . executing this command before building the image worked for me.

@hinorashi
Copy link

Simply just build with sudo, work for me 😄

sudo docker build -t test .

mruffalo added a commit to hubmapconsortium/cross_modality_query that referenced this issue Oct 8, 2020
Even if 'postgres' is included in .dockerignore, Docker still attempts to stat
the directory when constructing the build context:
docker/for-linux#380

Work around this by moving the Postgres directory and ignoring its parent dir.
@alxayeed
Copy link

alxayeed commented Oct 23, 2020

SOLVED

after searching and trying soo many methods , this answer actually solved my issue!

experiencing the same issue
error checking context: 'can't stat '/mnt/WORKSHOP/Practice/docker/python_flask''.

running the command using 'sudo' is not working for me

sudo docker build -t myapp .
error checking context: 'can't stat '/mnt/WORKSHOP/Practice/docker/python_flask''.

also, tried adding the path to .dockerignore
and tried moving the Dockerfile into a dockerfiles directory as well
error checking context: 'can't stat '/mnt/WORKSHOP/Practice/docker/python_flask/dockerfiles''.

docker version 

Client:
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.12
 Git commit:        dd360c7
 Built:             Mon Jun  8 20:23:26 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.12
  Git commit:       77e06fd
  Built:            Mon Jun  8 20:24:59 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

jnm added a commit to kobotoolbox/reports that referenced this issue Nov 3, 2020
* Unreadable Postgres directory must be inside a readable,
        `.dockerignore`d directory
        * See docker/for-linux#380
* Fetch `bballgames03.csv`, used by unit tests, from the Internet
        Archive
* Upgrade `r-base`
* Unpin `pyopenssl`
* Unpin `urllib3` and install with the `[secure]` extra
* Add instructions for building base image to README
@denisvmedia
Copy link

denisvmedia commented Jan 22, 2021

This happens to me in a rootless installation of docker:

Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:11:26 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:23 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I'm able to build for the first time, but repeating the action fails. It is a headache especially in the case when a user is not allowed to run sudo commands.

@Cheaterman
Copy link

Cheaterman commented Jan 26, 2021

I don't get it though, this used to work, right? I'm fairly sure I had a setup where the .dockerignore worked properly, but now I don't...
For reference, it worked here on 19.03.12 and broke in 19.03.13 or 14. Or it's possibly a case of "it works the first time" as some other users reported?

EDIT: My bad, it's not about the docker version. Build succeeds when using docker-compose and fails when using docker directly with what I presumed (probably wrongly) was the same command...

@Fuseteam
Copy link

Fuseteam commented Sep 1, 2021

i have this issue with a sshfs mount too it simply ignores dockerignore :/ i'm not using the snap

this workaround still works tho

rr- added a commit to rr-/TRCustoms that referenced this issue Dec 23, 2021
@bryangeplant
Copy link

Is this fixed? is it stuck? What happened here? The issue is still open.

lmaotrigine added a commit to lmaotrigine/heartbeat that referenced this issue Jan 21, 2023
@Cheaterman
Copy link

Still not fixed, hit this again today ; the behavior changed however - docker-compose is now in line with docker and fails the same way.

SomberNight added a commit to SomberNight/electrum that referenced this issue Mar 20, 2023
- repro builds to use fixed uid=1000 inside the container
  - in case the file permissions leak into the binaries, they are still reproducible
  - chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
  - otherwise the local dev build would still interact with the fresh_clone dir
    - due to e.g. recursive "find -exec touch",
    - and even the "docker build" cmd itself would try to stat/read it
      - see docker/for-linux#380
  - and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
  - this way the local dev build does not need sudo

to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
  - does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
  - if the host userid is 1000, does not need sudo
  - otherwise, needs sudo

closes spesmilo#8261
SomberNight added a commit to SomberNight/electrum that referenced this issue Mar 20, 2023
- repro builds to use fixed uid=1000 inside the container
  - in case the file permissions leak into the binaries, they are still reproducible
  - chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
  - otherwise the local dev build would still interact with the fresh_clone dir
    - due to e.g. recursive "find -exec touch",
    - and even the "docker build" cmd itself would try to stat/read it
      - see docker/for-linux#380
  - and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
  - this way the local dev build does not need sudo

to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
  - does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
  - if the host userid is 1000, does not need sudo
  - otherwise, needs sudo

closes spesmilo#8261
SomberNight added a commit to SomberNight/electrum that referenced this issue Mar 20, 2023
- repro builds to use fixed uid=1000 inside the container
  - in case the file permissions leak into the binaries, they are still reproducible
  - chown 1000:1000 fresh_clone
- repro builds to create fresh_clone dir outside git clone
  - otherwise the local dev build would still interact with the fresh_clone dir
    - due to e.g. recursive "find -exec touch",
    - and even the "docker build" cmd itself would try to stat/read it
      - see docker/for-linux#380
  - and "rm -rf fresh_clone" needs sudo if the host uid is not 1000
  - this way the local dev build does not need sudo

to recap:
- local dev builds use the host userid inside the container, directly operate on the project dir
  - does not need sudo
- repro builds create a fresh git clone, chown it to 1000, and use userid=1000 inside the container
  - if the host userid is 1000, does not need sudo
  - otherwise, needs sudo

closes spesmilo#8261
justinclift added a commit to pgautoupgrade/docker-pgautoupgrade that referenced this issue Jul 8, 2023
@ipepe
Copy link

ipepe commented Jul 26, 2023

Still relevant and blocker for some situations.

p12tic added a commit to p12tic/buildbot-infra that referenced this issue Dec 17, 2023
This allows to put "volumes" subdirectory into .dockerignore and get its
contents ignored. Currently the directory that is being ignored may get
wrong permissions and operating on it will fail.

See also: docker/for-linux#380
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests