New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.9: build caching broken from symlink?? #17290

Closed
ibuildthecloud opened this Issue Oct 23, 2015 · 25 comments

Comments

Projects
None yet
@ibuildthecloud
Contributor

ibuildthecloud commented Oct 23, 2015

This is an odd one but I tried to extract out the offending files out of a real build. In the end second build in the below script will end up using the cached image for CP . ./ when it shouldn't

#!/bin/bash

cat > Dockerfile << "EOF"
FROM busybox
COPY . ./
EOF

mkdir -p Godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/data
ln -s doesnotexist Godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/data/symlink

docker build -f Dockerfile .
echo ${RANDOM} > "$(date)"
docker build  -f Dockerfile .

This will output

Sending build context to Docker daemon  12.8 kB
Step 1 : FROM busybox
 ---> 8c2e06607696
Step 2 : COPY . ./
 ---> Using cache
 ---> 09b76d77bc36
Successfully built 09b76d77bc36
Sending build context to Docker daemon 13.82 kB
Step 1 : FROM busybox
 ---> 8c2e06607696
Step 2 : COPY . ./
 ---> Using cache
 ---> 09b76d77bc36
Successfully built 09b76d77bc36

Notice the offending lines

Step 2 : COPY . ./
 ---> Using cache
 ---> 09b76d77bc36

Version stuff

docker version; docker info
Client:
 Version:      1.9.0-rc2
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   60d36f7
 Built:        Fri Oct 23 00:20:59 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-rc2
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   60d36f7
 Built:        Fri Oct 23 00:20:59 UTC 2015
 OS/Arch:      linux/amd64
Containers: 17
Images: 1249
Server Version: 1.9.0-rc2
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.1-040001-generic
Operating System: Ubuntu 14.10
CPUs: 4
Total Memory: 15.4 GiB
Name: inotmac
ID: FPSO:Z7SO:BZJ3:IXH4:DNKV:UPZQ:KLP2:IRXW:5CBY:PBDM:HAKO:UOC7
Debug mode (server): true
 File Descriptors: 12
 Goroutines: 20
 System Time: 2015-10-22T19:45:15.96379998-07:00
 EventsListeners: 0
 Init SHA1: 
 Init Path: /home/darren/Downloads/docker-1.9.0-rc2
 Docker Root Dir: /var/lib/docker
Username: ibuildthecloud
Registry: https://index.docker.io/v1/

@tiborvass tiborvass added this to the 1.9.0 milestone Oct 23, 2015

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Oct 23, 2015

Contributor

Somebody let me know if they can reproduce.

Contributor

ibuildthecloud commented Oct 23, 2015

Somebody let me know if they can reproduce.

@vdemeester

This comment has been minimized.

Show comment
Hide comment
@vdemeester

vdemeester Oct 23, 2015

Member

@ibuildthecloud hum, using docker-1.9.0-rc2 I cannot reproduce on my laptop.

Sending build context to Docker daemon 6.656 kB
Step 1 : FROM busybox
 ---> d7057cb02084
Step 2 : COPY . ./
 ---> Using cache
 ---> 9f24ac82cfc0
Successfully built 9f24ac82cfc0
Sending build context to Docker daemon  7.68 kB
Step 1 : FROM busybox
 ---> d7057cb02084
Step 2 : COPY . ./
 ---> 2f80eb99c501
Removing intermediate container fac333456dd4
Successfully built 2f80eb99c501
Member

vdemeester commented Oct 23, 2015

@ibuildthecloud hum, using docker-1.9.0-rc2 I cannot reproduce on my laptop.

Sending build context to Docker daemon 6.656 kB
Step 1 : FROM busybox
 ---> d7057cb02084
Step 2 : COPY . ./
 ---> Using cache
 ---> 9f24ac82cfc0
Successfully built 9f24ac82cfc0
Sending build context to Docker daemon  7.68 kB
Step 1 : FROM busybox
 ---> d7057cb02084
Step 2 : COPY . ./
 ---> 2f80eb99c501
Removing intermediate container fac333456dd4
Successfully built 2f80eb99c501
@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Oct 23, 2015

Contributor

@vdemeester I've yet to find anyone who can reproduce and I haven't reproduced except on my laptop.... But it consistently fails on my laptop.

Contributor

ibuildthecloud commented Oct 23, 2015

@vdemeester I've yet to find anyone who can reproduce and I haven't reproduced except on my laptop.... But it consistently fails on my laptop.

@vdemeester

This comment has been minimized.

Show comment
Hide comment
@vdemeester

vdemeester Oct 23, 2015

Member

@ibuildthecloud hum.. more info on your laptop (storage driver & co) ?

Member

vdemeester commented Oct 23, 2015

@ibuildthecloud hum.. more info on your laptop (storage driver & co) ?

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Oct 23, 2015

Member

I can't repo this

Member

runcom commented Oct 23, 2015

I can't repo this

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Oct 24, 2015

Contributor

Okay, I'm closing. I wiped out my /var/lib/docker and now I can't reproduce anymore. So.... fixed that :)

Contributor

ibuildthecloud commented Oct 24, 2015

Okay, I'm closing. I wiped out my /var/lib/docker and now I can't reproduce anymore. So.... fixed that :)

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Nov 3, 2015

Contributor

I'm reopening because I consistently get this issue. I will work on a new test case. Tested against rc5 and still failing.

Contributor

ibuildthecloud commented Nov 3, 2015

I'm reopening because I consistently get this issue. I will work on a new test case. Tested against rc5 and still failing.

@ibuildthecloud ibuildthecloud reopened this Nov 3, 2015

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 3, 2015

Member

All on overlay? or also on other drivers?

Member

thaJeztah commented Nov 3, 2015

All on overlay? or also on other drivers?

@tiborvass tiborvass modified the milestones: 1.9.1, 1.9.0 Nov 3, 2015

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Nov 3, 2015

Contributor

@thaJeztah preparing a new test case, more involved but should be more reproducible...

Contributor

ibuildthecloud commented Nov 3, 2015

@thaJeztah preparing a new test case, more involved but should be more reproducible...

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 3, 2015

Member

thanks @ibuildthecloud we've moved this to the 1.9.1 milestone for now, but thanks so much for this 👍

Member

thaJeztah commented Nov 3, 2015

thanks @ibuildthecloud we've moved this to the 1.9.1 milestone for now, but thanks so much for this 👍

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Nov 3, 2015

Contributor

@thaJeztah I found the root cause and shared it on #docker-maintainer (sorry, on plane, and have spotty internet, i'll try to update issue later)

Contributor

ibuildthecloud commented Nov 3, 2015

@thaJeztah I found the root cause and shared it on #docker-maintainer (sorry, on plane, and have spotty internet, i'll try to update issue later)

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 3, 2015

Member

@ibuildthecloud yup, saw it on IRC, thx!

Member

thaJeztah commented Nov 3, 2015

@ibuildthecloud yup, saw it on IRC, thx!

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Nov 3, 2015

Contributor

If this could be fixed in 1.9.0 it would be greatly appreciated :)

Contributor

ibuildthecloud commented Nov 3, 2015

If this could be fixed in 1.9.0 it would be greatly appreciated :)

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 3, 2015

Member

The whole team is currently looking at the issue to see if there's a way to still get it fixed before the 1.9 release, but (as you probably understand) no promises ❤️

Member

thaJeztah commented Nov 3, 2015

The whole team is currently looking at the issue to see if there's a way to still get it fixed before the 1.9 release, but (as you probably understand) no promises ❤️

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Nov 3, 2015

Contributor

I'm in SF, I'll personally show up to the office for moral support if needed 😄 (I think I might have made this release hell...)

Contributor

ibuildthecloud commented Nov 3, 2015

I'm in SF, I'll personally show up to the office for moral support if needed 😄 (I think I might have made this release hell...)

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 3, 2015

Member

@ibuildthecloud unfortunately, it didn't make it in the release, but I'm sure they welcome you for moral support (and proper coffee 😄) ❤️

Member

thaJeztah commented Nov 3, 2015

@ibuildthecloud unfortunately, it didn't make it in the release, but I'm sure they welcome you for moral support (and proper coffee 😄) ❤️

@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Nov 3, 2015

Contributor

we <3 moral support

Contributor

jessfraz commented Nov 3, 2015

we <3 moral support

tonistiigi added a commit to tonistiigi/docker that referenced this issue Nov 5, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.


Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@akatrevorjay

This comment has been minimized.

Show comment
Hide comment
@akatrevorjay

akatrevorjay Nov 5, 2015

Just got bit by this on the 1.9.0 stable release. It was not easy to track down either.
Can we put a big notice out that 1.9.0 has a bug that breaks many builds that use symlinks?
Maybe in the release notes?
Thanks!

Just got bit by this on the 1.9.0 stable release. It was not easy to track down either.
Can we put a big notice out that 1.9.0 has a bug that breaks many builds that use symlinks?
Maybe in the release notes?
Thanks!

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 5, 2015

Member

@tiborvass perhaps on the releases page on GitHub? ^^

Member

thaJeztah commented Nov 5, 2015

@tiborvass perhaps on the releases page on GitHub? ^^

@icecrime icecrime added the priority/P0 label Nov 6, 2015

tonistiigi added a commit to tonistiigi/docker that referenced this issue Nov 6, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.


Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

@cpuguy83 cpuguy83 closed this in #17710 Nov 7, 2015

tiborvass added a commit to tiborvass/docker that referenced this issue Nov 9, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 47da59f)

tiborvass added a commit to tiborvass/docker that referenced this issue Nov 9, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 47da59f)

tiborvass added a commit to tiborvass/docker that referenced this issue Nov 9, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 47da59f)
@pwaller

This comment has been minimized.

Show comment
Hide comment
@pwaller

pwaller Nov 12, 2015

Contributor

I just lost several hours editing code I wasn't running to this exact situation. The failure mode is pretty gnarly. I also have a vendor'd copy of go-dockerclient. It's amusing what broken symlinks can do to you. I would very much appreciate a release of this ASAP so we can use docker for our builds again!

Contributor

pwaller commented Nov 12, 2015

I just lost several hours editing code I wasn't running to this exact situation. The failure mode is pretty gnarly. I also have a vendor'd copy of go-dockerclient. It's amusing what broken symlinks can do to you. I would very much appreciate a release of this ASAP so we can use docker for our builds again!

@akatrevorjay

This comment has been minimized.

Show comment
Hide comment
@akatrevorjay

akatrevorjay Nov 13, 2015

@tiborvass Is this showcased on the releases page on GitHub? I still don't see it?

@tiborvass Is this showcased on the releases page on GitHub? I still don't see it?

Mic92 added a commit to Mic92/docker that referenced this issue Dec 10, 2015

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.


Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

netbrain added a commit to netbrain/docker that referenced this issue Jan 4, 2016

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.


Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

jheiss added a commit to jheiss/docker that referenced this issue Mar 10, 2016

Fix symlink handling in builder ADD/COPY commands
Fixes #17290

Fixes following issues:

- Cache checksums turning off while walking a broken symlink.

- Cache checksums were taken from symlinks while targets were actually copied.

- Copying a symlink pointing to a file to a directory used the basename of the target as a destination basename, instead of basename of the symlink.


Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@reejosamuel

This comment has been minimized.

Show comment
Hide comment
@reejosamuel

reejosamuel Jul 6, 2016

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 35
Server Version: 1.12.0-rc3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 23
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: ZUAS:CKCG:YAG7:XWIN:TYFK:INOY:TPZE:REI6:DEO7:AQCX:QD4T:UTAU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 16
 Goroutines: 30
 System Time: 2016-07-06T15:57:55.119832527Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: carbonr
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

Happening again on this version

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 35
Server Version: 1.12.0-rc3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 23
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: ZUAS:CKCG:YAG7:XWIN:TYFK:INOY:TPZE:REI6:DEO7:AQCX:QD4T:UTAU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 16
 Goroutines: 30
 System Time: 2016-07-06T15:57:55.119832527Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: carbonr
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

Happening again on this version

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 6, 2016

Member

@reejosamuel do you have a way to reproduce?

Member

thaJeztah commented Jul 6, 2016

@reejosamuel do you have a way to reproduce?

@akatrevorjay

This comment has been minimized.

Show comment
Hide comment
@akatrevorjay

akatrevorjay Jul 6, 2016

Was this ever added to the release notes?

Thanks,
Trevor

On Wed, Jul 6, 2016 at 12:49 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

@reejosamuel https://github.com/reejosamuel do you have a way to
reproduce?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#17290 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABQGIuYkI25DQMRwJUoKuPc5wtrXUm3-ks5qS9yKgaJpZM4GUPoe
.

Was this ever added to the release notes?

Thanks,
Trevor

On Wed, Jul 6, 2016 at 12:49 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

@reejosamuel https://github.com/reejosamuel do you have a way to
reproduce?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#17290 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABQGIuYkI25DQMRwJUoKuPc5wtrXUm3-ks5qS9yKgaJpZM4GUPoe
.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
Member

thaJeztah commented Jul 6, 2016

It's in the release notes for 1.9.1 https://github.com/docker/docker/releases/tag/v1.9.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment