Container Manager Move Failed

[holocronology]

Hello, I set up M.2 volume per the other helper scripts. I then began the app mover script to move Container Manger to volume2.

It appeared to work OK, but I initially left all my docker data folders on volume1. Somewhere in the process of moving them to volume2 something got mucked up. Could have been me, could have been Syno starting a scrub at the same time, I do not know.

I started to get errors on all my containers that looked like this:

If I tried to delete a container it changed to this:

and

Steps to correct:

1. I used app mover script back to volume1. Did not fix.
2. I stopped CM, restarted NAS, started CM. Did not fix.
3. I uninstalled CM, which should have deleted everything, restarted NAS, reinstalled CM. Did not fix. In fact all my old containers showed up with the same errors and results when trying to delete.

No sure where to go from here.

[ctrlaltdelete007]

You need to map your path inside the containers to volume 2, this can't be done by the script.

[holocronology]

Yes, I get that internal to the container/compose the mapping needs to be updated.

[holocronology]

So this is the error I'm seeing when I try to delete/remove a container now:

sudo docker rm -f wud Password: Error response from daemon: container 0ae2b262a503d1fd009bc0123a874420602e4d7e61f375db282da170ce9604ac: driver "btrfs" failed to remove root filesystem: Failed to destroy btrfs snapshot /volume1/@docker/btrfs/subvolumes for d4f8102d5931ed71f83b3622859d3a902c64a9c288aed70ebbdb3b2c2e85294b: invalid argument

[007revad]

In the pre-release v3 of the script it no longer skips moving folders if the target folder is not empty because it caused issues like this. Instead v3 will copy the folder's contents if the target folder is not empty.

Did you answer yes to backup @docker folder when the script asked if you wanted to back it up?

[holocronology]

Did you answer yes to backup @docker folder when the script asked if you wanted to back it up?

Yes, I did. But it was unclear to me how to restore from it.

Currently running the v2.0.7 version. Right now, I'm just trying to recover back to some semblance of normalcy, even if that is back onto volume1. I was attempting another move to volume2, in the hopes that it would fix things, but it is unclear to me if it is stuck on moving @appsotre/ContainerManager to /volume2 or just being very slow.

[007revad]

Which volume was docker originally installed on before you tried moving it?

[holocronology]

volume1

This is where it is right now. Been at this point a long time:

[holocronology]

I downloaded your pre-release 3.0.38. Evidently I did not backup....

[007revad]

It should only takes seconds to move @appstore/ContainerManager. The next step would have been moving @docker (which can take hours if you have a lot of container images).

What does this command return?

ls /volume1 | grep '@docker'

[holocronology]

It should only takes seconds to move @appstore/ContainerManager. The next step would have been moving @docker (which can take hours if you have a lot of container images).

What does this command return?

ls /volume1 | grep '@docker'

@docker
@docker_backup

[007revad]

Did you backup @docker with v3 just now, or before you first moved container manager with v2?

What do these commands return?

du -s /volume1/@docker

du -s /volume1/@docker_backup

[007revad]

And which volume is container manager currently installed on?

[holocronology]

Okay....

Did you backup @docker with v3 just now, or before you first moved container manager with v2?

Before my first move with v2.

What do these commands return?

du -s /volume1/@docker
du: cannot read directory '/volume1/@docker': Permission denied
0 /volume1/@docker

and
du -s /volume1/@docker_backup
du: cannot read directory '/volume1/@docker_backup': Permission denied
0 /volume1/@docker_backup

I tried to sudo both of those commands and it just hangs after the password.

And which volume is container manager currently installed on?
Package Center shows it as currently installed on volume2

[007revad]

Try:

sudo -i

du -s /volume1/@docker

du -s /volume1/@docker_backup

[holocronology]

I'm getting nothing from the above. Just hangs.

I think I'm at the point of nuking CM, reformatting the vol2 ssd's and starting fresh. Not great. Not terrible.

[007revad]

Not great. Not terrible.

But your experience will help others. I'm in the process of changing the script to advise people export their docker container configs just in case.

I may also change the script to copy docker's @ folders instead of moving them. Then if something went wrong users would just need edit the 6 symlinks in /var/packages/ContainerManager to get back to where they started.

[holocronology]

Glad I could contribute. For me, it was a pretty easy restore, as I copied my shared docker folder to a "docker_old" folder, so once I did the clean install of CM on the SSD volume2 I could just copy them into the new shared docker folder. Then it was just a matter of updating the compose volume mappings in various CM Projects and Portainer Stacks.

But I had to do things like remake the docker networks I had set up, etc. So I still see value in this.

I'm assuming that v3.x can be used to backup Container Manager in general? It would be nifty to be able to trigger that via options on the script so that it could be set up as a scheduled task.

[007revad]

I'm assuming that v3.x can be used to backup Container Manager in general?

Yes. You can backup (and restore) just the selected app or all apps.

It would be nifty to be able to trigger that via options on the script so that it could be set up as a scheduled task.

I've added this to my to-do list.

I'm thinking the script could be scheduled with something like an --auto option:

syno_app_mover.sh --auto=ContainerManager

and even support a | separated list of packages to backup:

syno_app_mover.sh --auto=ContainerManager|Calendar|WebStation

[holocronology]

the script could be scheduled with something like and --auto option

Yes, something like both of these. I guess the question is does this provide some sort of better functionality than internal backup or some other docker based backup (such as nautical-backup)? I don't believe that any of those would backup the @docker data itself.

[holocronology]

advise people export their docker container configs just in case

Could you include an option in the CM move queue to copy the "docker shared folder" to "docker-move" (or some other shared folder name)? This would give someone the opportunity to do it during the move process. In theory all the configs should be in that folder.

[007revad]

For the docker shared folder the v3 version of the script shows the steps to move the docker share, which is best done via the DSM ui. It also warns that if you move the docker shared folder (or other shared folders that your containers access) you need to edit those container's docker compose, .env files, or the container's settings, to point to the new volume number.

[007revad]

I guess the question is does this provide some sort of better functionality than internal backup or some other docker based backup (such as nautical-backup)? I don't believe that any of those would backup the @docker data itself.

I don't know if other backup methods also backup @docker or not. They may just backup the data that each container needs instead of backing up the whole folder.

[eudaldgr]

One possible solution will be remove all this btrfs subvolumes snapshots before doing anything:

for subvolume in /volume2/\@docker/btrfs/subvolumes/*; do
    btrfs subvolume delete "$subvolume"
done

and add this to /etc/docker/daemon.json

[...]
  "storage-driver": "overlay2"
[...]

and no more btrfs subvolumes

link

[007revad]

@eudaldgr

In DSM 6.2.4 and 7.2.1 there is no /etc/docker/daemon.json but there is a dockerd.json in a different location:

• In DSM 7 it's /usr/syno/etc/packages/ContainerManager/dockerd.json
• In DSM 6 it's /usr/syno/etc/packages/Docker/dockerd.json
• DSM 7's dockerd.json contains "storage-driver":"btrfs"
• DSM 6's dockerd.json contains "storage-driver" : "btrfs"

I'm looking into this now but the link you provided says overlay does not support btrfs.

Storage driver	Supported backing filesystems
overlay2	xfs with ftype=1, ext4
fuse-overlayfs	any filesystem
btrfs	btrfs
zfs	zfs
vfs	any filesystem

So it seems that overlay2 is not an option when using btrfs. I'd have to use btrfs, fuse-overlayfs or vfs

overlay2 seems to also require kernel 5.19+ according to this page. But Synology NAS have either kernel 5.10, 4.4 or 3.10

So I decided to try overlay2 and see what happens. When I start Container Manager it replace my "storage-driver":"overlay2" with "storage-driver":"btrfs" !?!?!

[007revad]

Fixed in https://github.com/007revad/Synology_app_mover/releases/tag/v3.0.50