docker commit data container with VOLUME

[winggundamth]

I'm not sure anyone has a way or requesting this but this is my use case.

I knew that there's a way to backup data container with --volumes-from, bind mount with host with tar as describe here. But it should be good if Docker have a way to commit data container to image with data in Volumes. So I can use it for snapshot/backup data and push/pull it to transfer data across server.

Use case example

Dockerfile

VOLUME /home/web/app/uploads

On host

docker run -d app
docker run -d --volumes-from app app-data
docker commit --with-volumes=true app-data app-backup:201407131628
docker push app-backup #this will push only diff

On dev or another host

docker pull app-backup:201407131628
docker run --with-volumes=true -d app-data
docker run -d --volumes-from app-data app

Do you have a plan for this feature or any work around to do snapshot on volume?
I know this will be very complicated but it will be the very nice to have feature for sure :)

[thaJeztah]

+1 Volumes are useful in many cases, for example, for running database-storage. However, since volumes are not 'part' of a container, it makes containers no longer portable - which seems in direct conflict with the slogan Build once... Run anywhere...

[crosbymichael]

+1 i need this @cpuguy83

[cpuguy83]

@crosbymichael I thought about this, and potentially enabling it for all docker builds.
Currently a volume specified in a build creates an immutable directory.

[thaJeztah]

I was thinking about this the other day; some things to take into account;

• Should bind-mounted volumes also be considered 'volumes' in this case?
• If so; how to exclude sockets etc.?
• Should there be an option to exclude irrelevant volumes (temp/log volumes?)
• Docker export should support this as well (docker export --with-volumes)
• --snapshot may be a more generic name?

[wasseem-redboxdigital]

+1 for the request by winggundamth
Now, can anyone update us if this request has been granted?

At the time of this writing, I have noticed that $ docker commit does not commit the data within the volume mounted (in the case of a data-only container).
And also the statement $docker commit --with-volumes=true is not there too.

It would be nice to have this request fulfilled in the case of a data-only container, it's much easier+faster to commit & run VS backup & restore.

[Georges0013]

there a proposal about it right here, but it's closed now : #8484

[kojiromike]

+1 I take it the black-belt label means this is going to be harder than breaking boards with one's chin?

[ain]

👍

[soupdiver]

+1

[thaJeztah]

ping @icecrime @calavera wondering what the current position is on this proposal, also in light of the volume-drivers etc.

[Matvey-Kuk]

+1!!!

[crosbymichael]

I think going forward the work on top level volume objects will replace data containers and allow more advanced management of volumes with things like export.

@cpuguy83 should have a PR somewhere or it is already merged in the experimental branch.

[cpuguy83]

#14242 for top-level volumes.

[jessfraz]

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Mainly:

Allowing the Builder to be implemented as a separate utility consuming the Engine's API will open the door for many possibilities, such as offering alternate syntaxes or DSL for existing languages without cluttering the Engine's codebase

Then from there, patches/features like this can be re-thought. Hope you can understand.

[rakesh-james]

Is this Fixed ? Still need to work round? Any one please update...

[sbrennion]

I would also appreciate such a feature. Docker is flexible and there is not only one way to use it.
I want also being able to seperate the data in a volume, but perhaps at some point push the container with volume data in it , and deploy it

[moylop260]

👍

[NZhuravlev]

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here:
Then from there, patches/features like this can be re-thought. Hope you can understand.

Well.. why would you even mention Dockerfile syntax? The issue is about run command.

There is now direct answer to this issue, no merged PR, no opened PR, no info about why you can't do that, nothing.

@thaJeztah reopen this issue, maybe?

[omorillo]

👍

[cvrod]

+1

[linshenqi]

+1

[ropnom]

+1

[paulz1]

+1

[neocortical]

+1

[inlineHamed]

+1

[rikvermeer]

+1

[tabbi89]

+1

[eeezyy]

+1

[blachawk]

+1

[lavvy]

+1. How else can we snapshot volumes

[pokapow]

+1

[qinshulei]

+1

[evanfchan]

+1

[farokojil]

+1

[vguna]

+1

[cdaringe]

woo wee, that took forever to figure out why my commits weren't "working" as expected.

@thaJeztah, or anyone else really, i am not necessarily seeking to commit a volume, but remove the volume declaration, and perhaps recreate it later?

for instance, couchdb ships with a VOLUME declared. my objective is to ship a container with some prebaked data in it, but i can't run any migrations and recommit them because of this volume.

if there is a recipe for the operation, i'd love to hear it.

[7c]

+1

[thaJeztah]

While I commented to be "positive" on this issue in the past, I've since gotten more insight into the complexity and design problems this feature would bring, and implementing this as a feature would be very problematic.

A volume is really separate from a container that's using it; commiting that data into an image will unlikely be implemented. Let me first try to illustrate what's happening when using a volume;

If an image declares a VOLUME, or a --volume is used on docker run;

• docker creates a volume (if a non-existing volume is specified)
• if the volume is empty (and the nocopy option is not set), the data from the path inside the container (where the volume is mounted) is copied from the container to the volume.
• the container is started, and the volume is mounted inside the container (masking the files that are there)

When commiting a container to an image, only changes to the container's filesystem are committed; because the volume is not part of that, those files are not committed. This is the intended behavior for most use-cases; volumes are designed to;

• decouple data from the application (see bullets below)
• persist data beyond the container's lifecyle; this allows you to replace/upgrade a container, reuse the volume from before upgrading, and continue working with the data that was there
• similar; when deleting a container, your important data isn't lost (it's still in the volume)
• volumes prevent the overhead that copy-on-write filesystems cause (volume data is not using a copy-on-write filesystem)

Complexity

Implementing this feature would introduce a lot of additional complexity;

• Before the container is committed, it must be stopped, otherwise data may still be written to the volume. Not only does this stop the container itself, this will likely affect other containers depending on it.
• The volume must be unmounted (this can cause a race condition; at this moment, the volume is no longer marked as "in use", thus can be deleted, for example when running docker volume prune)
• The data from the volume must be copied back into the container (how to deal with the existing files? Should those be merged, deleted?)
• Comitting a container now actually modifies the container; starting the container again after committing is effectively starting a different container than was stopped.
• When starting, the volume has to be attached/mounted again.
• Restarting a container leads to new ports to be assigned (in case of "random" published ports - -p 80 or -P)

What use-case would be resolved by this feature?

The main use-case mentioned in this thread is to be able to ship an image that contains (default) data to use at runtime. There's a number of ways to realize this;

• As an example, the official MySQL image allows you to automatically intialize the database by providing scrips or SQL dumps in the docker-entrypoint-initdb.d directory (see the readme at https://hub.docker.com/_/mysql/). Other images offer this feature as well
• Change the configuration of the software running inside the container to use a data-path outside of a path that's using a volume. At runtime, you can still use a volume through the -v or --mount flag.
• Consider using a separate image containing just the data to put in the volume; you can use that image to populate the volume. The advantage of this approach is that the data is decoupled from the image (so it's easier to upgrade the software running, without having to update the data)

There are improvements to be made, for example: allow a VOLUME that was defined in a parent image to be unset; see #3465

With the information above, I think it'll be very unlikely this feature will get enough support among the maintainers to be implemented.

[gudge25]

so is there a way to commit docker image with files from volumes?

[thaJeztah]

No there isn't, other than copying the files out of the volume to another location in the container before committing

[eaglet3d]

These manual overview steps might work:
Let's say you have an image (i.e. source image), and volume that mounts to a specific directory for the containers of that image.

1. Run a container 'WithVol' from the source image with the volume mounted.
2. Exec to create tar file of the volume directory on the container 'WithVol'
3. Copy the tar file locally
4. Run a container 'NoVol' from the same source image without the volume mounted.
5. Copy the tar file to the container 'NoVol'
6. Exec to extract the tar file to the target directory on the container 'NoVol'
7. Commit the container 'NoVol' to a target image
8. Create a new container from the target image
9. Exec to ls the files in the target directory on the new container
10. Confirm that all of the files are there.

Let me know how you make out.

[hemangjoshi37a]

Is this already solved? Can we commit now with persistent data to a image?
Please help.

[lamba]

Not solved. See the workaround described in the Jan 4, 2019 comment.

[sungaok]

+1， could provide this feature, and leave users to decide how to use it.

[sungaok]

could you provide me any idea, how to modify this feature in my own git code, I will try my own environment.
thks
@eaglet3d

[b4shtirk1n]

+1

[CatIIIIIIII]

+1