machine learning fills disk with hundreds of gigabytes

[oskardotglobal]

Sometimes, immich machine learning's container filesystem (in /var/lib/docker/overlay2) baloons to hundreds of gigabytes in size, filling my disk completely. I am running latest (v2.7.5) with an almost unchanged compose configuration (I had to relax apparmor restrictions using security_opt: [apparmor=unconfined]) using rocm on a Ryzen 5 2400G iGPU.

[yudin-s]

If the space is growing under /var/lib/docker/overlay2, I would first confirm which path inside the ML container maps to that growth. The important distinction is container writable layer vs the intended ML cache volume.

The stock Immich compose normally has a persistent ML cache mount similar to:

machine-learning:
  volumes:
    - model-cache:/cache

I would check the actual large directory before deleting anything:

docker system df -v
docker exec immich_machine_learning sh -c 'du -xh -d1 / /cache /root/.cache /tmp 2>/dev/null | sort -h'

If /cache is the large path, make sure that path is backed by a named volume or bind mount, not just the container writable layer. If /root/.cache or /tmp is growing instead, that may point to runtime/model-cache behavior outside the expected /cache mount; the du output, Immich version, image tag, and compose snippet would make the next step much clearer.

I would avoid deleting files directly from overlay2; identify the in-container path first, then recreate/prune only the affected container or volume once you know what is filling it.

If this narrows down the disk growth, please mark it answered so other ML-cache cases find the same diagnostic path.

[oskardotglobal]

The volume is present. When the issue occurs, a simple docker compose down immich-machine-learning does the trick. I am running immich v2.7.5 with the default compose configuration downloaded directly from Github, which is also up-to-date.

[mion00]

I think I am in a similar situation as OP. By analyzing the output of docker system df and filtering for the immich-machine-learning container, I obtain the following:

docker system df -v --format "{{json .Containers}}" | jq '.[] | select(.Image | contains("immich-machine-learning")) | del(.Labels)'

{
  "Command": "\"tini -- python -m immich_ml\"",
  "CreatedAt": "2026-06-01 21:30:13 +0200 CEST",
  "ID": "709e6637a9c0818a47b5d7ee656479b6d650c570e548f7ff542f3c2727c1181f",
  "Image": "ghcr.io/immich-app/immich-machine-learning:v2.7.5",
  "LocalVolumes": "0",
  "Mounts": "/mnt/media/immich/model-cache,/etc/localtime",
  "Names": "immich-machine-learning",
  "Networks": "db",
  "Ports": "",
  "RunningFor": "4 days ago",
  "Size": "4.58GB",
  "State": "running",
  "Status": "Up 4 days (healthy)"
}

As visible, the container is using ~4.5GB of space in its internal FS.

docker exec immich-machine-learning sh -c 'du -xh -d1 / /cache /root/.cache /tmp 2>/dev/null | sort -h'
4.0K    /boot
4.0K    /home
4.0K    /media
4.0K    /mnt
4.0K    /srv
4.0K    /tmp
8.0K    /run
1.4M    /etc
8.3M    /var
21M     /cache/ocr
340M    /usr
544M    /cache/facial-recognition
1.1G    /opt
4.3G    /cache/clip
4.3G    /root
4.9G    /cache
5.7G    /

Excluding from the analysis /cache, that is correctly mounted from the host:

docker inspect immich-machine-learning | jq .[].HostConfig.Binds
[
  "/etc/localtime:/etc/localtime:ro",
  "/mnt/media/immich/model-cache:/cache:rw"
]

The next culprint lies in the /root/.cache dir:

docker exec immich-machine-learning sh -c 'du -xh -d3 /root/.cache 2>/dev/null | sort -h'
28K     /root/.cache/matplotlib
4.3G    /root/.cache
4.3G    /root/.cache/huggingface
4.3G    /root/.cache/huggingface/xet
4.3G    /root/.cache/huggingface/xet/https___cas_serv-tGqkUaZf_CBPHQ6h

From a quick glance at the Hugging face doc it seems that these folders are created automatically to optimize the download of the ML models.

Proposed improvement

Maybe it should be possible to configure this cache to also use the already existing mount point under /cache?

[oskardotglobal]

Seems like a different issue? I've had the fs go as large as 150GB. I am just waiting for it to happen again so I can get more useful output on what's using the space

[yudin-s]

mion00's output helps narrow down one concrete case: /cache is mounted correctly, but Hugging Face/Xet is still writing chunk cache data into /root/.cache/huggingface/xet, which stays in the container writable layer.

For that case, I would try putting the Hugging Face cache under the existing ML cache mount too:

immich-machine-learning:
  environment:
    HF_HOME: /cache/huggingface
    HF_XET_CACHE: /cache/huggingface/xet
    MACHINE_LEARNING_CACHE_FOLDER: /cache/immich_ml

Then recreate just the ML container and compare docker system df -v after the models are downloaded again.

For the original 150 GB report, I would still want the same du output while the growth is happening. If it is also /root/.cache/huggingface/xet, this is probably the same cache-placement issue; if the growth is elsewhere, it should be treated separately.