Error in sendTar, no such file or directory

[ghost]

Hi,

I've tried to pull images from s3 and found that sometimes I get an error "no such file or directory":

Importing image(504273d1d86d) TAR file to docker hosts: [tcp://192.168.59.103:2376]
Loading image to: tcp://192.168.59.103:2376
[{"Host":"tcp://192.168.59.103:2376","Status":"failed","Error":"API error (404): open /mnt/sda1/var/lib/docker/tmp/docker-import-632579556/repo/c0ed0404a62c757b25a98dcfd6f7ea5ad5408e56c60277cebfc848adc458ef03/layer.tar: no such file or directory\n"}]
2015/04/04 20:17:05 Error in sendTar

This happens on my mac with boot2docker using TCP connection, as well as on linux with native LXC with unix socket. Is there any additional logging I can find out as to why this is happening.

Cheers,
Julian.

[chrisferry]

👍 I am consistently seeing this as well.

[relistan]

@didip can you take a look?

[toffer]

I've seen this sporadically.

The error message indicates that it is the docker daemon API that is throwing the error, and the "no such file or directory" message is talking about a file or directory internal to docker. (Note the /var/lib/docker in the path.)

If the docker API fails when we go to load an image, well, I'm not sure there's much we can do about this. It may be small comfort, but retrying usually works!

What docker version are you running? Perhaps older versions of docker are more prone to errors when using the API to load an image?

[chrisferry]

I was running 1.0.1 so I upgraded to 1.5.0 and am still receiving the same error.

[relistan]

I think there is a chance that this was caused by a race condition which might now be fixed. Give it a shot.

[reu]

I am getting this too, with Docker v1.5.0 on OSX with boot2docker.

[relistan]

This is really odd because we use this to deploy many applications every day. Can one of you guys get ahold of the Docker logs while this is happening? It might help us debug it. Thanks!

[reu]

Well, I don't know if these logs could help... but that's what I have here:

time="2015-05-08T14:51:55Z" level=debug msg="Calling POST /images/load"
time="2015-05-08T14:51:55Z" level=info msg="POST /images/load"
time="2015-05-08T14:51:55Z" level=info msg="+job load()"
time="2015-05-08T14:51:56Z" level=info msg="+job image_get(1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a)"
could not find image: no such id: 1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a
time="2015-05-08T14:51:56Z" level=info msg="-job image_get(1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a) = ERR (1)"
time="2015-05-08T14:51:56Z" level=debug msg="Loading 1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a"
time="2015-05-08T14:51:56Z" level=debug msg="Error reading embedded tar%!(EXTRA *os.PathError=open /mnt/sda1/var/lib/docker/tmp/docker-import-123152796/repo/1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a/layer.tar: no such file or directory)"
open /mnt/sda1/var/lib/docker/tmp/docker-import-123152796/repo/1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a/layer.tar: no such file or directory
time="2015-05-08T14:51:56Z" level=info msg="-job load() = ERR (1)"
time="2015-05-08T14:51:56Z" level=error msg="Handler for POST /images/load returned error: open /mnt/sda1/var/lib/docker/tmp/docker-import-123152796/repo/1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a/layer.tar: no such file or directory"
time="2015-05-08T14:51:56Z" level=error msg="HTTP Error: statusCode=404 open /mnt/sda1/var/lib/docker/tmp/docker-import-123152796/repo/1facee05a9a3cacdaf2eee4ed1acbd4fc8e2e25b6695f398216d2c770fe4778a/layer.tar: no such file or directory"

[leonidra]

Just started to receive messages like that. Anyone had any chance solving this?

[didip]

Yes, I found one other cause on how this could happen.

When dogestry push is happening, S3 can legitimately timed out/connection error.
The bug is that dogestry did not fail immediately or perform retries.
That behavior causes dogestry pull to be missing 1 or 2 files, causing this issue to happen.

I am fixing it. See #56 and #57 for details.

[didip]

v2.0.0 is just released, and it should fix this issue, but i will leave this ticket open for now.