bug: rsync backup of Immich data stalls, leads to zombie process, and memory pressure #815
Description
Describe the bug
- Attempting to back up the Immich data directory with rsync results in the process stalling indefinitely.
- Rsync freezes at 0% (xfr#0, ir-chk=1003/1402) for hours.
- Killing the process leaves behind a zombie [rsync] child.
- During long stalls, Portainer and other containers stopped unexpectedly, and system memory dropped to under 1 GiB free (likely due to page cache exhaustion or OOM events).
- Only a fraction of the data (e.g., backups/ and part of encoded-video/) was copied before stalling; large directories like library/ (~325 GB) and thumbs/ (~5 GB) were untouched.
To Reproduce
Steps to reproduce:
-
Stop Immich container(s).
-
Run backup on low ram:
rsync -avh --progress /media/files/photos/immich/ /media/files/photos-bk/
- Observe that rsync starts transferring a few files, then halts.
Expected behavior
- Rsync should either complete the backup or error out cleanly.
- Processes should terminate fully without leaving zombies.
- System services (like Portainer) should not be disrupted by a userland file copy.
Actual behavior
- Rsync stalls during directory traversal.
- Zombie processes remain even after termination attempts.
- System shows reduced available memory, and containers like Portainer crash/stop.
Some Logs & Output
Resume command I tried
rsync -aH --info=progress2 --partial --append-verify \
/media/files/photos/immich/ /media/files/photos-bk/
32,768 0% 0.00kB/s 0:00:00
which I let sit over night so about 8 hours and nothing
checking if rsync is working and force killing it
mike@markvm2:~$ ps -o pid,ppid,stat,etime,cmd -p 2778605,2778606
PID PPID STAT ELAPSED CMD
2778605 7461 R+ 13:42:13 rsync -aH --info=progress2 --partial --append-verify /media/files/photos/immich/ /media/files/photos-bk/
2778606 2778605 Z+ 13:42:13 [rsync] <defunct>
mike@markvm2:~$ sudo kill -KILL 2778605
Checking space of drive
df -h /media/files/photos/immich /media/files/photos-bk
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.1T 2.8T 6.4T 31% /media/files
/dev/sda1 9.1T 2.8T 6.4T 31% /media/files
Compare sizes
mike@markvm2:~$ du -sh /media/files/photos/immich/* | sort -h
136K /media/files/photos/immich/profile
300K /media/files/photos/immich/upload
1.3G /media/files/photos/immich/backups
4.4G /media/files/photos/immich/encoded-video
5.0G /media/files/photos/immich/thumbs
325G /media/files/photos/immich/library
mike@markvm2:~$ du -sh /media/files/photos-bk/* | sort -h
0 /media/files/photos-bk/library
0 /media/files/photos-bk/profile
0 /media/files/photos-bk/thumbs
0 /media/files/photos-bk/upload
1.3G /media/files/photos-bk/backups
4.3G /media/files/photos-bk/encoded-video
Environment
OS: Debian 12 (Bookworm)
Rsync version: 3.2.7
Immich data directory size: ~335 GB
Filesystem: btrfs