Error restarting containers after backup

[andrewmooreio]

What are you trying to do?

I've been using this container for months to back up my other containers to S3. However, in the past 2 days, the backup succeeds but then the containers which are stopped during the backup (docker-volume-backup.stop-during-backup=true) fail to start again.

The logs indicate a log of storage space for the /var/lib/docker directory, however there is more than enough, and I can start the containers manually without any issue at all.

Filesystem                             Size  Used Avail Use% Mounted on
udev                                   7.8G     0  7.8G   0% /dev
tmpfs                                  1.6G  2.3M  1.6G   1% /run
/dev/mapper/docker01--vg-root           14G  3.2G  9.9G  25% /
tmpfs                                  7.9G     0  7.9G   0% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/sda2                              456M  106M  325M  25% /boot
/dev/mapper/docker--vg-docker--pool    126G   70G   50G  59% /var/lib/docker
/dev/sda1                              511M  5.9M  506M   2% /boot/efi
tmpfs                                  1.6G     0  1.6G   0% /run/user/1000

What is your current configuration?

backup:
    container_name: backup
    environment:
      - AWS_ENDPOINT=${S3_ENDPOINT}
      - AWS_S3_BUCKET_NAME=${S3_BUCKET_NAME}
      - AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
      - AWS_S3_PATH=${HOSTNAME_OVERRIDE}
      - BACKUP_CRON_EXPRESSION=0 5 * * *
      - BACKUP_RETENTION_DAYS=14
      - NOTIFICATION_URLS=${BACKUP_NOTIFICATION_URLS}
    image: offen/docker-volume-backup:latest
    network_mode: bridge
    restart: always
    volumes:
      - etc...

Log output

Running docker-volume-backup failed with error: stopContainers: 12 error(s) restarting containers and services: Error response from daemon: mkdir /var/lib/docker/overlay2/387d1fc4adc47bea32c4cbe6b8d1801f934095effdc043c193578413c0a0af7f/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/d9c62ed8d24cd03314635c884f233fd22ea35e56e98d25b8c4a619cf3270b52d/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/c2647ae33303bf6d907365d27f267aaaff67df5e1e66c16f41fb33925a1d08c6/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/e5f273e5a23d2127dee0cca12d2e17e9416a4c660c1b3106e61654842a1e300a/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/ecc9bfcccc23d8a7b919cc6a5573e9d893d4fccad8fe2f86f7d092e276bb17aa/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/097c683d6ff5f374e8a5c94068ea9fba3d8b9ff9dcbe87638079134a8807e0a3/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/f98c0782b746246ecc0d8146b909582373a7a1d676e65862609f5975d7d8ac68/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/97e6990e4ed4b42c6430b284b0bf02f32b3716d0e34a261430a725aead0fed51/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/3484c02aff3d8e5891dde90d25eaa8505ae35730b7fd9ac2002f715e64e0120d/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/90b7482baa0932f7442576d946a8e7a75521d3ae71dd2a002bc255f4e106ac9e/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/d78788b70e9a1eb0fedc315ad2f5e847ba086cef3940f3a4d0c3c29c98400238/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/c3f89baa3a47a65baf6d44da33842a945caad3440040910079c6b03856c00a20/merged: no space left on device

Additional context

Thanks in advance!

[pixxon]

I also ran into this issue in the past. In my case it was not running out of disk space, but running out of available inodes on the ext4 filesystem. You can check this by running df -i.

The solution for me was to configure log rotation on the json-file ( default ) logger. Here is the related article from Docker.

[andrewmooreio]

Thanks, @pixxon. I hadn't thought of inodes. However it seems I have plenty free.

Filesystem                              Inodes  IUsed    IFree IUse% Mounted on
udev                                   2040899    415  2040484    1% /dev
tmpfs                                  2046874   1213  2045661    1% /run
/dev/mapper/docker01--vg-root           920272  49117   871155    6% /
tmpfs                                  2046874      1  2046873    1% /dev/shm
tmpfs                                  2046874      3  2046871    1% /run/lock
/dev/sda2                               124928    347   124581    1% /boot
/dev/mapper/docker--vg-docker--pool    8388608 598435  7790173    8% /var/lib/docker
/dev/sda1                                    0      0        0     - /boot/efi
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/9e551cecdca009b0f6e1bc893496eec414d09d790385d28cc3fdc9ebeb4a0d94/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/c3f89baa3a47a65baf6d44da33842a945caad3440040910079c6b03856c00a20/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/8263f2a866454156ae772f1ce3c83655e1328330356c35e5bad6cacc2a1cdaef/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/c7974e6c3347f36b336cebe3b385d518fa801ac70aec5a18c2ed0f50e2b2115f/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/90b7482baa0932f7442576d946a8e7a75521d3ae71dd2a002bc255f4e106ac9e/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/a12410dedd580814d67c5884812e4db3fb09f2366733db2968572ddbd0815761/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/d78788b70e9a1eb0fedc315ad2f5e847ba086cef3940f3a4d0c3c29c98400238/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/097c683d6ff5f374e8a5c94068ea9fba3d8b9ff9dcbe87638079134a8807e0a3/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/387d1fc4adc47bea32c4cbe6b8d1801f934095effdc043c193578413c0a0af7f/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/3484c02aff3d8e5891dde90d25eaa8505ae35730b7fd9ac2002f715e64e0120d/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/d9c62ed8d24cd03314635c884f233fd22ea35e56e98d25b8c4a619cf3270b52d/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/c2647ae33303bf6d907365d27f267aaaff67df5e1e66c16f41fb33925a1d08c6/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/e5f273e5a23d2127dee0cca12d2e17e9416a4c660c1b3106e61654842a1e300a/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/ecc9bfcccc23d8a7b919cc6a5573e9d893d4fccad8fe2f86f7d092e276bb17aa/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/e75cb9857c417d4d1500aa5fee7ec9203e981b0d0b155625ca7757255244c2bb/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/1b48d4efb2267947703a61e125d8de590a5d8edb28b8818328f0473a31176c0d/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/f98c0782b746246ecc0d8146b909582373a7a1d676e65862609f5975d7d8ac68/merged
overlay                                8388608 598435  7790173    8% /var/lib/docker/overlay2/97e6990e4ed4b42c6430b284b0bf02f32b3716d0e34a261430a725aead0fed51/merged
tmpfs                                   409374     13   409361    1% /run/user/1000

[m90]

Unless this started happening after you did an image upgrade (which I think you did not?), I'm not sure if there is anything actionable regarding this tool here. The docker daemon on your setup fails to do something and the error is being forwarded.

It is indeed surprising seeing that your disk seems to have enough space and that you can restart containers manually, but I fail to see how the stop/restart mechanism in this image could be related.

One thing I am wondering: are the archives you are creating rather large? I.e. could storing them temporarily while uploading them to S3 exhaust (or almost exhaust) your available disk space (i.e. #95 and #316)?

[andrewmooreio]

Unless this started happening after you did an image upgrade (which I think you did not?), I'm not sure if there is anything actionable regarding this tool here. The docker daemon on your setup fails to do something and the error is being forwarded.

It is indeed surprising seeing that your disk seems to have enough space and that you can restart containers manually, but I fail to see how the stop/restart mechanism in this image could be related.

One thing I am wondering: are the archives you are creating rather large? I.e. could storing them temporarily while uploading them to S3 exhaust (or almost exhaust) your available disk space (i.e. #95 and #316)?

That was it, everything looks fine storage wise the day after, but of course by then the backup archive has been deleted.

Thanks for the help!