New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tar command fails with OverlayFS storage driver but not with aufs #19647

Open
olalonde opened this Issue Jan 24, 2016 · 45 comments

Comments

Projects
None yet
@olalonde

olalonde commented Jan 24, 2016

Command to reproduce:

docker run -it --rm heroku/cedar:14 /bin/bash -c "curl -sS https://install.meteor.com | /bin/sh"

On Docker v1.8.1, aufs storage driver, the command completes successfully:

Downloading Meteor distribution
######################################################################## 100.0%

Meteor 1.2.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com

On Docker v1.8.3, overfayfs storage driver, the command fails:

Downloading Meteor distribution
######################################################################## 100.0%
tar: .meteor/packages/coffeescript/.1.0.11.148kw9n++os+web.browser+web.cordova/plugin.compileCoffeescript.os/npm/babel-compiler/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/with: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/transformers: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/defs: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules/.bin: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules/meteor-babel: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm/node_modules: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova/npm: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler/.5.8.24_1.127zxv6++os+web.browser+web.cordova: Directory renamed before its status could be extracted
tar: .meteor/packages/babel-compiler: Directory renamed before its status could be extracted
tar: Exiting with failure status due to previous errors
Installation failed.

More logs:

core@deis-03 ~ $ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Sat Dec  5 05:57:26 UTC 2015
 OS/Arch:      linux/amd64
core@deis-03 ~ $ docker info
Containers: 11
Images: 69
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-coreos-r1
Operating System: CoreOS 835.9.0
CPUs: 1
Total Memory: 1.959 GiB
Name: deis-03
ID: SSCW:KJB5:3MI4:WCSV:RXKY:WG2W:HB7U:TRYL:36L4:ZDTC:5KUL:JIMD

Related issues: deis/deis#4867 meteorhacks/meteord#64 meteor/meteor#5762

@olalonde olalonde changed the title from Some tar commands fail on Docker with OverlayFS to tar command fails on Docker with OverlayFS but not with aufs Jan 24, 2016

@olalonde olalonde changed the title from tar command fails on Docker with OverlayFS but not with aufs to tar command fails with OverlayFS storage driver but not with aufs Jan 24, 2016

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Jan 25, 2016

Could not reproduce in my two environments:

  • Docker 1.10.0-rc1, Kernel 4.2.0-25-generic (Ubuntu 15.10)
  • Docker 1.6.2, Kernel 4.4.0-rc8 (vanilla)

@olalonde Could you please try newer CoreOS?

@olalonde

This comment has been minimized.

olalonde commented Jan 25, 2016

@AkihiroSuda doubt that will help, CoreOS 835.9.0 was released last month. Are you sure you tested with OverlayFS and not AUFS storage driver?

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Jan 26, 2016

Yes, I'm using OverlayFS.
I'll try other environments including CoreOS later.

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.10.0-rc1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.2.0-25-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.6 GiB
...
$ docker run -it --rm heroku/cedar:14 /bin/bash -c "curl -sS https://install.meteor.com | /bin/sh"`
Downloading Meteor distribution
######################################################################## 100.0%

Meteor 1.2.1 has been installed in your home directory (~/.meteor).
Writing a launcher script to /usr/local/bin/meteor for your convenience.

To get started fast:

  $ meteor create ~/my_cool_app
  $ cd ~/my_cool_app
  $ meteor

Or see the docs at:

  docs.meteor.com
@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Jan 26, 2016

I confirmed that the bug is reproducible with CoreOS 935.0.0 (kernel 4.4.0-coreos-r2).
But reproduciblity is not 100%

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 26, 2016

@olalonde can you report this issue in the CoreOS issue tracker? They have their own builds, and issues for those builds should be reported in the CoreOS issue tracker first.

I'll close this issue here, but happy to reopen if it turns out to be an issue here

@thaJeztah thaJeztah closed this Jan 26, 2016

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Jan 26, 2016

Oh, it's even reproducible with Boot2Docker v1.10.0-rc1.

@thaJeztah This is not specific to CoreOS. So I think it's worth reopening and investigating here.

It seems that the the bug is highly reproducible with a fresh overlay env, but not so much reproducible with aged one. I guess that's why I couldn't reproduce in some my aged environments (#19647 (comment))

@olalonde

This comment has been minimized.

olalonde commented Jan 26, 2016

@thaJeztah I'm fairly certain the bug is with OverlayFS actually, but couldn't figure out where to open an issue :) (maybe in the Linux Kernel issue tracker?)

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 26, 2016

Sure, I'll reopen :-)

Hm, not sure if reporting upstream will help with overlay; we're actually trying to get the upstream maintainer to apply some patches, but no response so far 😢

@mitar

This comment has been minimized.

mitar commented May 31, 2016

Interesting. I am getting this error sometimes for Docker hub automatic builds for my Meteor apps in exactly the same step, when install Meteor inside, which uses tar to extract files. But what is interesting is that sometimes it happens and sometimes it does not. Does Docker hub build images on different underlying configurations?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented May 31, 2016

@mitar Docker Hub starts a fresh machine for each build to guarantee your builds are executed in a clean environment, and no artifacts are left around (see https://blog.docker.com/2015/11/upgraded-autobuild-docker-hub/). For what I've heard, they updated the configuration to address this issue, but if you're still seeing this, please contact support@docker.com; they should be able to look into that.

@mitar

This comment has been minimized.

mitar commented May 31, 2016

Yes, I have just seen it. And it is crazy, so I compile the same commit to latest and master, and one succeeded while the other one failed. At the same time. 4 hours ago.

I will contact them.

@lpgeiger

This comment has been minimized.

lpgeiger commented Jul 1, 2016

+1 This seems to be a problem on Docker Cloud, as well as Docker Hub. One the competing service Quay.io the automated builds work just fine.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 1, 2016

@keyscores could you send an e-mail to support@docker.com?

@lpgeiger

This comment has been minimized.

lpgeiger commented Jul 9, 2016

@thaJeztah @borjaburgos I emailed 8 days ago. I still don't have a meaningful response from support@docker.com

@bkil-sandbox

This comment has been minimized.

bkil-sandbox commented Jul 15, 2016

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Jul 22, 2016

Progress in coreos/bugs#1095 (comment) 😄

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2016

ping @dmcgowan ^^

@dmcgowan

This comment has been minimized.

Member

dmcgowan commented Jul 26, 2016

@AkihiroSuda thanks for the pointer, looks similar to other inode change issues with OverlayFS. There is a relevant discussion going on here as well http://thread.gmane.org/gmane.linux.file-systems.union/879/focus=879. This likely effects both overlay and overlay2 since it can occur anytime an extraction is done on an OverlayFS mount. Adding the kernel flag.

@dmcgowan

This comment has been minimized.

Member

dmcgowan commented Nov 29, 2016

@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jan 14, 2017

I hit this today :(

@Bregor

This comment has been minimized.

Bregor commented Jan 17, 2017

Same here with relatively fresh kernel and docker and overlay2.

# uname -a
Linux kube06 4.8.17-040817-generic #201701090438 SMP Mon Jan 9 09:40:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux


# docker info
Containers: 39
 Running: 27
 Paused: 0
 Stopped: 12
Images: 28
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.8.17-040817-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: kube06
ID: BXB2:WQP3:GZ5I:JDBH:GWPO:ZSKQ:AHNM:XH77:V2RR:E2CO:TJ5X:6GUD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
@samoht

This comment has been minimized.

Member

samoht commented Jan 31, 2017

Reported as well by @mfeblowitz in Docker for Mac docker/for-mac#1219. Switching back to aufs fixed the issue...

@xinity

This comment has been minimized.

xinity commented Feb 7, 2017

got the issue here using Docker for windows latest release.
switched back to aufs definitely solved it !

what's wrong with overlayfs2 implementation ?

@jasonkeene

This comment has been minimized.

jasonkeene commented Feb 7, 2017

@xinity It plays tricks that tar does not like.

Bregor added a commit to evilmartians/chef-kubernetes that referenced this issue Mar 15, 2017

Bregor added a commit to evilmartians/chef-kubernetes that referenced this issue Mar 15, 2017

mhubig pushed a commit to mhubig/docker-partkeepr that referenced this issue Mar 31, 2017

@acathrow

This comment has been minimized.

acathrow commented Apr 26, 2017

Is there any update on this issue, ideally ETA for a fix.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Apr 28, 2017

@acathrow it's not a Docker bug, but an issue with the way overlay is implemented in the kernel; the thread to the kernel discussion seems to have stalled on december 6 though; http://www.spinics.net/lists/linux-unionfs/msg01321.html

@cameronbrunner

This comment has been minimized.

cameronbrunner commented May 4, 2017

Looks like there is progress being made on this:

https://lwn.net/Articles/721470/

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Aug 7, 2017

This information seems useful: coreos/bugs#1095 (comment)

I think kernel 4.13's OVERLAY_FS_INDEX=y config should fix this. See https://lwn.net/Articles/725276/

@karanhiremath

This comment has been minimized.

karanhiremath commented Sep 12, 2017

Was running into this bug here today and Docker 17.09.0-ce-rc1 from the Edge channel (Docker for Mac download here) ended up fixing the issue for me. Obviously be careful with the Edge channel releases as always but it looks like progress has been made in there.

@antis81

This comment has been minimized.

antis81 commented Sep 17, 2017

Ran into same issue on Xubuntu 16.04 host (inside a VirtualBox). Docker version: 17.06.2-ce. For the moment, the bsdtar workaround should do.

@jouve

This comment has been minimized.

jouve commented Nov 28, 2017

@AkihiroSuda using kernel >= 4.13 fixed this issue for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment