New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"docker save" does not store image tags. #3877

Closed
ptramsey opened this Issue Jan 31, 2014 · 31 comments

Comments

Projects
None yet
@ptramsey

ptramsey commented Jan 31, 2014

If you call "docker save repo:tag", the resulting tarball will not contain the repo or tag, and there's no way that I can see to figure out what image is inside such a tarball, short of extracting the json and building the revision history yourself. IE, if you're handed a "docker saved" image, the only way to figure out what was loaded when you do "docker load" is to compare "docker images" from before and after.

Docker inspect should work on tarballs, and docker save should store repos and tags.

@tavisrudd

This comment has been minimized.

tavisrudd commented Feb 5, 2014

It's inconsistent: docker save image-repo-name -> docker load restores name, docker save SHA -> docker load no names or tags, docker save name:latest -> docker load no names or tags

@tianon

This comment has been minimized.

Member

tianon commented Feb 6, 2014

Since a single image ID can have multiple names/tags, I'd say the behavior of those first two is exactly what I'd expect (ie, if you've done something like docker tag ubuntu:12.04 my-private-repo/user/ubuntu:12.04, then you don't want docker save <ubuntu-12.04-id> to include "my-private-repo/user/ubuntu:12.04"), but that last one feels wrong. I think the bug there may be in docker load, but I haven't examined the resulting docker save tarballs recently enough to say for sure.

@ptramsey

This comment has been minimized.

ptramsey commented Feb 6, 2014

The problem is really that currently there's no way to know, after calling docker load, what was loaded. If you load an image, you can't then instantiate it, because you don't know its ID.

@tavisrudd

This comment has been minimized.

tavisrudd commented Feb 6, 2014

The docs for docker save should be explicit about the semantics of these three cases. The second case makes sense when you think about it but my initial intuition from reading the docs was that it would find all tags associated with the SHA and save them. docker load should provide some feedback about what was loaded.

@ewindisch

This comment has been minimized.

Contributor

ewindisch commented Feb 7, 2014

👍 The third use-case needs to preserve the tag imho, and I agree with better output from load.

@jamtur01

This comment has been minimized.

Contributor

jamtur01 commented Feb 7, 2014

+1 to Eric.

@techwolf359

This comment has been minimized.

techwolf359 commented Feb 18, 2014

SHA should get the names and tags associated with that too. Also +1 for documentation in docker save that itemizes what gets stored. "Contains all parent layers, and all tags + versions" is misleading if by design, it does not save tags + versions.

@unclejack unclejack added the bug label Feb 26, 2014

@srid

This comment has been minimized.

Contributor

srid commented Mar 8, 2014

i'm attempting to workaround this issue on my end using this script:

# docker-save.sh
#!/bin/bash -ex
# Wrapper for 'docker save' fixing,
# https://github.com/dotcloud/docker/issues/3877
# In addition: this script will always save exactly one image (possibly
# multiple tags).

IMAGE=$1
TARGET=$2
NAME=`echo $IMAGE | awk -F':' '{print $1}'`
ID=`docker inspect $IMAGE | scripts/json 0.id`
TAGS=`docker images --no-trunc | grep $ID | awk '{print $2}'`
DIR=`mktemp -d --suffix=-docker-save`
pushd $DIR

docker save $ID > $TARGET
echo "{\"$NAME\":{" > repositories
for TAG in $TAGS; do
        echo "\"$TAG\":\"$ID\"," >> repositories
done
echo "}}" >> repositories

tar -r -f $TARGET repositories

popd
rm -rf $DIR

however, docker load refuses to load a tar file created by this script (all it does is add a file called "repositories" to it):

$ sudo docker load < x.tar
2014/03/07 18:23:57 Error: load: open /tmp/docker-import-482638663/repo/.wh..wh.orph/json: no such file or directory
$

it seems to me that a tar file created by 'docker save' cannot be manipulated by GNU tar.

@srid

This comment has been minimized.

Contributor

srid commented Mar 8, 2014

and the added 'repositories' file is missing for some reason. i see some strange files appended to it instead:

$ tar -t -f base.tar
./
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/VERSION
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/json
531e5919a09a5107132223644b6108b55429e80bf9bb7e8dbf5b2d324d028acf/layer.tar
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/VERSION
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/json
8dbd9e392a964056420e5d58ca5cc376ef18e2de93b5cc90e868a1bbc8318c1c/layer.tar
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/VERSION
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/json
91b97161ec67a9b7534fe5cd08d3a73c82eb8ea1c5dff6ad727335dff31a55d1/layer.tar
./
.wh..wh.aufs
.wh..wh.orph/
.wh..wh.plnk/
$
@srid

This comment has been minimized.

Contributor

srid commented Mar 8, 2014

it seems to me that a tar file created by 'docker save' cannot be manipulated by GNU tar.

yup, the culprit turns out to be GNU tar (or docker's tar being incompatible with it). using Python's tarfile module works. here is the working wrapper script working around this issue: https://gist.github.com/srid/9424614

@crosbymichael crosbymichael added this to the 1.0 milestone May 15, 2014

@crosbymichael crosbymichael removed this from the 1.0 milestone May 16, 2014

@shin- shin- added the Distribution label Jul 1, 2014

@yukw777

This comment has been minimized.

yukw777 commented Jul 24, 2014

Any development on this issue?

@vieux

This comment has been minimized.

Collaborator

vieux commented Jul 24, 2014

@yukw777 I think there is no issue /cc @vbatts @fkautz

@vbatts

This comment has been minimized.

Contributor

vbatts commented Jul 24, 2014

What versions are you working with? There were inconsistent behavior, but has been fixed recently

@fkautz

This comment has been minimized.

Contributor

fkautz commented Jul 25, 2014

The current (and correct) behavior is as follows:

docker save repo

Saves all tagged images + parents in the repo, and creates a repositories file listing the tags

docker save repo:tag

Saves tagged image + parents in repo, and creates a repositories file listing the tag

docker save imageid

Saves image + parents, does not create repositories file. The save relates to the image only, and tags are left out by design and left as an exercise for the user to populate based on their own naming convention.

@matthughes

This comment has been minimized.

matthughes commented Jul 29, 2014

@fkautz The second one works for you? If I try and save a repo:tag, it fails to create the directory because it's trying to create a directory with a colon in it (well that's my guess):

docker save matt/centos:6.5.0.2014.07.21
2014/07/29 21:05:32 Error: mkdir /tmp/docker-export-083524343/matt/centos:6.5.0.2014.07.21: no such file or directory

@fkautz

This comment has been minimized.

Contributor

fkautz commented Jul 29, 2014

I tried it in trunk and saw it work. What version are you running?
On Jul 29, 2014 5:09 PM, "Matt Hughes" notifications@github.com wrote:

@fkautz https://github.com/fkautz The second one works for you? If I
try and save a repo:tag, it fails to create the directory because it's
trying to create a directory with a colon in it (well that's my guess):

docker save matt/centos:6.5.0.2014.07.21
2014/07/29 21:05:32 Error: mkdir
/tmp/docker-export-083524343/matt/centos:6.5.0.2014.07.21: no such file or
directory


Reply to this email directly or view it on GitHub
#3877 (comment).

@matthughes

This comment has been minimized.

matthughes commented Jul 29, 2014

I'm running 1.1.2. Have you tried with a namespace? Here's a reproduction:

vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker build -t mynamespace/myrepo:1.0.0 .
Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon
Step 0 : FROM busybox
Pulling repository busybox
d200959a3e91: Download complete
37fca75d01ff: Download complete
a9eb17255234: Download complete
fd5373b3d938: Download complete
511136ea3c5a: Download complete
42eed7f1bf2a: Download complete
c120b7cab0b0: Download complete
120e218dd395: Download complete
1f5049b3536e: Download complete
f06b02872d52: Download complete
 ---> a9eb17255234
Step 1 : CMD echo "hello world"
 ---> Running in a1249d508391
 ---> 16a69a2920aa
Removing intermediate container a1249d508391
Successfully built 16a69a2920aa
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker save mynamespace/myrepo:1.0.0 > savedImage.tar
2014/07/29 23:23:52 Error: mkdir /tmp/docker-export-858462892/mynamespace/myrepo:1.0.0: no such file or directory
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker info
Containers: 1
Images: 20
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 22
Execution Driver: native-0.2
Kernel Version: 3.13.0-32-generic
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker --version
Docker version 1.1.2, build d84a070
@matthughes

This comment has been minimized.

matthughes commented Jul 29, 2014

It appears that the namespace is actually the culprit, not the version. If I take the same example but build without a namespace, it works:

agrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker build -t myrepo:1.0.0 .
Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon
Step 0 : FROM busybox
 ---> a9eb17255234
Step 1 : CMD echo "hello world"
 ---> Using cache
 ---> 16a69a2920aa
Successfully built 16a69a2920aa
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ docker save myrepo:1.0.0 > myrepo.tar
vagrant@vagrant-ubuntu-trusty-64:/vagrant/docker/save-test$ ls
Dockerfile  myrepo.tar 
@matthughes

This comment has been minimized.

matthughes commented Jul 30, 2014

Also, saving with namespace/tag (dropping version) does work. So to sum up:

namespace/tag:version -> does not work
namespace/tag              -> does work
tag                                  -> does work
@fkautz

This comment has been minimized.

Contributor

fkautz commented Jul 30, 2014

I did not try it with a namespace. I'll give it a try.
On Jul 29, 2014 8:07 PM, "Matt Hughes" notifications@github.com wrote:

Also, saving with namespace/tag (dropping version) does work. So to sum up:

namespace/tag:version -> does not work
namespace/tag -> does work
tag -> does work


Reply to this email directly or view it on GitHub
#3877 (comment).

@vbatts

This comment has been minimized.

Contributor

vbatts commented Aug 6, 2014

all the behaviour seen in master (042b642) is expected behaviour.

vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev version
Client version: 1.1.2-dev
Client API version: 1.14
Go version (client): go1.2.2
Git commit (client): 042b642
OS/Arch (client): linux/amd64
Server version: 1.1.2-dev
Server API version: 1.14
Go version (server): go1.2.2
Git commit (server): 042b642
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev images busybox
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
busybox             latest              a9eb17255234        8 weeks ago         2.433 MB
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save busybox | tar xO repositories
{"busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save busybox:latest | tar xO repositories
{"busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev tag busybox:latest namespace/busybox
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save namespace/busybox | tar xO repositories
{"namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save namespace/busybox:latest | tar xO repositories
{"namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev tag busybox:latest 0.0.0.0:5000/namespace/busybox
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save 0.0.0.0:5000/namespace/busybox | tar xO repositories
{"0.0.0.0:5000/namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save 0.0.0.0:5000/namespace/busybox:latest | tar xO repositories
{"0.0.0.0:5000/namespace/busybox":{"latest":"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721"}}vbatts@jellyroll ~ (master *) $ 
vbatts@jellyroll ~ (master *) $ /home/vbatts/src/docker/docker/bundles/1.1.2-dev/dynbinary/docker-1.1.2-dev save a9eb17255234 | tar xO repositories
tar: repositories: Not found in archive
tar: Exiting with failure status due to previous errors

@vbatts vbatts closed this Aug 6, 2014

@sroebuck

This comment has been minimized.

sroebuck commented Aug 20, 2014

The comment that closes this issue is unclear as to whether the current released behaviour is considered to be 'expected behaviour' or whether there is a fix to the current release that will appear in the next release (as a result of being in master).

Certainly I'm finding two issues when using the -o option to output to a specified output tar file:

  1. If the image to be saved is identified with a url that contains a / then the command issues a mkdir error trying to produce the /tmp directory to generate the intermediate files.
  2. If the image to be saved is identified with an explicit version tag then the resulting tar (when loaded using docker load) results in an untagged image being loaded.

I would have expected that saving an image with an explicit version would save only that image and its associated layers. When saving an image without an explicit version it is probably beneficial that it saves all of the versions and their associated layers.

@ewindisch

This comment has been minimized.

Contributor

ewindisch commented Aug 20, 2014

@sroebuck re:#2 - is this reproducible on the master branch?

@sroebuck

This comment has been minimized.

sroebuck commented Aug 20, 2014

Your question hits my concern. I'm unfortunately not in a position to say whether this works in master but I know it exhibits these issues in the current latest release 1.1.2. The closing comment is not at all clear on what should or shouldn't work and I don't know if master at the time of the comment corresponds to the current 1.1.2 or a future release with 'fixed' behaviour.

@ewindisch

This comment has been minimized.

Contributor

ewindisch commented Aug 20, 2014

@sroebuck this has been marked closed because it was confirmed to be fixed in 'master'. The presence of this fix will be in the next release (1.2)

@Espace2001

This comment has been minimized.

Espace2001 commented Sep 11, 2014

Very Easy, make a df on command line, and you will see where docker has monted his container, so you can tar it.

@nuest

This comment has been minimized.

nuest commented Jan 3, 2017

I've found that saving a image by id does not preserve the name and tag. Is this intentional?

I understand the digest sha overrides the tag, but I don't see why the loaded image (last command) cannot contain the original image name.

daniel@gin-nuest:~$ docker --version
Docker version 1.12.5, build 7392c3b

daniel@gin-nuest:~$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
fdab12439263: Pull complete 
Digest: sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963
Status: Downloaded newer image for busybox:latest

daniel@gin-nuest:~$ docker save busybox > busybox_name.tar
daniel@gin-nuest:~$ docker load < busybox_name.tar
Loaded image: busybox:latest
Loaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5
Loaded image ID: sha256:e02e811dd08fd49e7f6032625495118e63f597eb150403d02e3238af1df240ba
daniel@gin-nuest:~$ docker save busybox:latest > busybox_name-latest.tar
daniel@gin-nuest:~$ docker load < busybox_name-latest.tar
Loaded image: busybox:latest

daniel@gin-nuest:~$ docker save busybox:latest@sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963 > busybox_name-digest.tar
daniel@gin-nuest:~$ docker load < busybox_name-digest.tarLoaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5

daniel@gin-nuest:~$ docker save busybox@sha256:f102731ae8898217038060081c205aa3a4ae3f910c2aaa7b3adeb6da9841d963 > digest.tar
daniel@gin-nuest:~$ docker load < digest.tarLoaded image ID: sha256:1efc1d465fd6255396d0efd90a899f57aba2b7b294b02d5f55c6f5505ca9f3e5
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 3, 2017

I understand the digest sha overrides the tag, but I don't see why the loaded image (last command) cannot contain the original image name.

A single digest can be tagged under multiple names; when saving by digest, the name:tag is therefore not part of the saved image.

@attila123

This comment has been minimized.

attila123 commented Mar 31, 2018

I wrote a shell script to save all the docker images with repo + tag info as .tar files, see https://gist.github.com/attila123/8f60a0ebce15b6065975ba35e3be7d75

@vipcxj

This comment has been minimized.

vipcxj commented Aug 7, 2018

@thaJeztah IMO, save should save everything including name:tag. And then we can using some option to control what to load from the tar. This should be perfect.

@gixnex

This comment has been minimized.

gixnex commented Nov 23, 2018

I use this one liner, may be it helps:

for i in $(docker images | awk '{if (NR!=1) {print $1":"$2","$3}}');do ID_DOCKER=$(echo $i | cut -d',' -f2); TAG_DOCKER=$(echo $i | cut -d',' -f1); echo "docker tag $ID_DOCKER $TAG_DOCKER"; done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment