Please provide documention on restoring a docker volume from a backup to avoid "volume [volume name] already exists but was not created by Docker Compose. Use external: true to use an existing volume"

[Magnum5234]

Description

Since #9969 was closed and is apparently as designed please provide documentation on how to restore a volume from a backup without setting external to true and avoid getting a warning/error forever.

[ndeloof]

why would you not use external: true if you want to use an existing volume ?
About "how to restore a volume from a backup" as a general question, docker command line does not provide backup capabilities, this could either be provider by a custom volume driver, or by mounting volume in a temporary container running a backup command. But then this is up to you to define the mechanism in use

[Magnum5234]

why would you not use external: true if you want to use an existing volume ?

Lets say you have 3 projects running with VCS and stuff and the docker-compose.yml is also included with volume settings.
Now lets assume that one project has a hiccup and you are forced to restore an older version of the backuped volume.
While before I had 3 exact same systems running with identical docker-compose configuration I now have to change external:true for a single one because I restored the volume from a backup. Does not make sense to me.

If for whatever reason you have not moved or restored docker-compose environments, good for you - but it happens. So it would be kind to provide an easy fix for users that have to do it - without changing the previous docker-compose configuration.

[ndeloof]

Your use case is obviously valid, but since Docker Compose does not manage volume backups and restore, you can't expect it magically discovers one of the existing volume is a good match to be used for your app. This is the reason external: true exists.
If you've been able to restore volume for a previous run of your compose application, including required name and labels, compose will just use it "as if" it was created by a previous docker compose ... command. If such labels are not set, then .. you backup is incomplete.

[Magnum5234]

Your use case is obviously valid, but since Docker Compose does not manage volume backups and restore, you can't expect it magically discovers one of the existing volume is a good match to be used for your app.

Hence why I recommended to remove the warning altogether first in #9969.

If you've been able to restore volume for a previous run of your compose application, including required name and labels, compose will just use it "as if" it was created by a previous

I'm not sure why I would backup anything from the volume but the files itself. But to humor you, how would one create such a backup in an easy way, @ndeloof ? If you care to look at the official docker documentation you may find that they only tar the the files: https://docs.docker.com/storage/volumes/#back-up-restore-or-migrate-data-volumes

[ndeloof]

why I would backup anything from the volume but the files itself

No specific reason, just saying: if you want Compose to consider your restored backup as a "native" volume, you also need to restore metadata.

how would one create such a backup in an easy way

There's indeed no portable volume backup solution. The documented one is the "lowest denominator". I would expect some volume driver do offer some advanced features, typically backup/restore, but this isn't part of the API. So you'll hav no better choice but bring your own solution here

[Magnum5234]

So you'll hav no better choice but bring your own solution here

Yep, I recommend as stated #9969 to remove the warning. This is still the easiest solution for all around. It's even the same behavior as in the old v1.x version - and you can add it back later when there is such a time that this feat (of restoring a backup with labels and stuff) is possible with as easy build-in solution
or in case you want to keep the warning, provide documentation that it is only possible if you backup more than the files (like you suggested) without a working easy solution. "Everybody has to implement their own version of a more complicated backup method" does not appeal to me though.

[ndeloof]

The warning is here by intent, to help users who actually want to use a volume they created on their own, but don't know how to "force compose to use it". We can't just "adopt" a volume, with risk to eventually delete user's data.

So you have two options:

• fully rebuild a volume with required compose metadata. There's no magic tool for this purpose, but achievable with adequate shell script to create volume with required labels + content
• set external: true. Can use a compose override file if you don't want to update the main compose file.

[Magnum5234]

In case someone needs to create a volume with corresponding labels, its working like that:
(but if you were not able to figure this out yourself, maybe don't do it ;))

docker volume create --name "foo_bar" --label "com.docker.compose.project=foo" --label "com.docker.compose.version=2.17.3" --label "com.docker.compose.volume=bar"

[ndeloof]

I would suggest to avoid "hacking" into creating a compose equivalent volume. Better let compose create it.
Typical usage (to adjust to your needs):

services:
   restore-backup:
      profile: ["restore"]
      image: alpine
      workdir: /data
      command: tar xzf -
      volumes:
        - data:/data

volumes:
  data:

such a compose file let you run cat backup.tar.gz | docker compose run restore-backup
this will let compose create the volume with everything needed so it is well identified as "project volume" and have it populated from a backup

[douph1]

Merci @ndeloof pour l'exemple et vive le breizcamp :)

Just to fix a little your sample
profile > profiles
workdir > working_dir

Mine :

version: "3.8"

services:
  restore-backup:
    profiles: ["restore"]
    image: alpine
    working_dir: /
    command: tar xvzf -
    volumes:
      - mariadb:/var/lib/mysql

volumes:
  mariadb:

and use "-T" to avoid "the input device is not a TTY"

cat /tmp/backup.tgz | docker compose run -T restore-backup

[intentionally-left-nil]

The solution I came across is:

1. When restoring your docker compose containers, first run docker compose create This will pull the containers, create the networks and volumes, but not start anything.
2. Now you can mount & restore the files within the volume (using vackup or whichever tool desired)
3. Now you can run docker compose up to start the container (with the restored data)

[robkorv]

I'm getting this warning because one of the volumes in my project was made with the older docker-compose pip package.
Is there an official way to migrate the volumes to the newer Compose plugin or should I just hack the labels so I don't get the warning anymore?

Use external: true to use an existing volume

In my situation the volume was created by compose..... just by an older version or should external: true also be used in this case?

[ndeloof]

@robkorv please use docker volume inspect xx to get labels from your existing docker volume
Then check it has com.docker.compose.project label set to the same project name you're running here
This should not have changed since Docker Compose v1 ("pip package")

[robkorv]

@robkorv please use docker volume inspect xx to get labels from your existing docker volume
Then check it has com.docker.compose.project label set to the same project name you're running here
This should not have changed since Docker Compose v1 ("pip package")

Thanks. Yeah... for some reason 3 of the 5 volumes did not have any labels. They were really old though, created around "2020-12-18T10:38:02+01:00".... while the two that are not getting the warning are much newer.
I also checked the rights on the _data dir. The volumes that did not get the warning are owned by 999:docker while the one with warnings were owned by the current user.

Thankfully the volumes did not contain anything important in my case. So I deleted them and on the next up they were recreated with the correct labels and same owner as the working volumes.
My issue is resolved.