External Library scan incorrectly marks thousands of existing assets as offline and deleted

[daciodf]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Technical Incident Report — Immich
Bug Report / Incident Report — Version 3
Suggested issue title: External Library scan incorrectly marks thousands of existing assets as offline and deleted

1. Executive Summary
   Conclusion: the investigation confirms that on August 14, 2026, an automatic External Library scan performed by
   LibraryService found zero files in all three libraries. Thousands of existing assets were then marked offlined and
   received a deletedAt timestamp in PostgreSQL. There is no evidence of manual user deletion.
   The incident affected 68,811 assets between 00:00:00.573 and 00:00:07.211 UTC, in approximately 6.638
   seconds. The physical files remained accessible in storage. After restoring the logical asset state, a validation
   scan on August 17 showed normal behavior: assets were brought online and 0 assets were offlined in the library
   that had completed processing.
2. Environment
   Component Information
   Operating system ZimaOS v1.7.0
   Hardware ZimaCube
   Architecture amd64
   Immich image ghcr.io/immich-app/immich-server:release
   External Libraries 3
   Storage External libraries backed by remote storage mounts
3. Evidence from the August 14 Incident
   Evidence Result
   Affected assets 68,811
   First deletedAt 2026-08-14 00:00:00.573 UTC
   Last deletedAt 2026-08-14 00:00:07.211 UTC
   Duration 6.638 seconds
   Libraries affected 3
   Disk crawl 0 files found in each library
   The logs recorded the start of the External Library scan and, for each library, “Finished disk crawl, 0 file(s) found on disk”.
   The LibraryService then checked the existing assets and marked them offlined in large batches.
4. Recovery
   The Immich UI restoration initially logged restoration operations, but PostgreSQL still showed the original deletedAt
   timestamps for the incident assets. Because the physical files were accessible and a database backup was available, a
   controlled transactional recovery was performed directly in PostgreSQL.
   Exactly 68,811 assets were restored, limited to the incident's August 14 time window. Items that had already been in the
   trash before the incident were left untouched.
   Post-recovery state Count
   Total assets 69,169
   Post-recovery state Count
   Active 68,982
   In trash 187
   Incident assets still with deletedAt 0
5. Validation Test on August 17, 2026
   After recovery, a new External Library scan was performed. The LibraryService logs showed disk crawls for all three
   libraries and reported 0 assets offlined due to import paths/exclusion patterns.
   The first library to complete reported: 2,532 assets checked, 0 offlined, 2,468 onlined, 0 updated and 64
   unchanged, with 100% progress. This demonstrates that Immich could again find the physical files and bring the
   assets online.
   At the time of this report, the provided logs show completion for one of the three libraries; the other two were still being
   processed. Final results for all three libraries should be recorded once their scans complete.
6. Incident vs. Validation Comparison
   Aug. 14 — incident Aug. 17 — validation
   Automatic scan Manual/validation scan
   0 files found during disk crawl Files found
   Assets marked offlined 0 assets offlined
   68,811 assets received deletedAt Assets restored/onlined
   Libraries appeared empty to the scan Completed library: 2,468 onlined
7. Confirmed Findings
   • LibraryService performed the scan associated with the incident.
   • The disk crawl returned zero files in all three External Libraries during the incident.
   • 68,811 assets received deletedAt in less than seven seconds.
   • The physical files remained in storage and were later accessible from the container.
   • All 68,811 incident assets were recovered in the database without modifying the 187 pre-existing trash items.
   • A subsequent validation scan found the files and, for the completed library, brought 2,468 assets online without offlining
   any assets.
8. Root Cause Not Yet Determined
   The August 17 validation scan shows that the issue is not reproduced when the mounts are accessible, but it does not
   establish why the August 14 disk crawl returned zero files. Further investigation is needed into remote mount availability,
   mount timing, connectivity, permissions, storage behavior, and possible race conditions during the scan.
9. Request to the Immich Team
   Please investigate whether LibraryService has sufficient safeguards for External Library scans when a remote mount is
   temporarily unavailable, empty, or not mounted. In this incident, a zero-file crawl was followed by mass marking of valid
   assets as offline/deleted. Ideally, a scan that finds zero files in a previously populated library should be treated as an
   anomalous condition before applying a large-scale destructive state change.
   Please also determine whether this behavior corresponds to a known issue or has already been fixed in a newer Immich
   release.

The OS that Immich Server is running on

ZimaOS v1.7.0

Version of Immich Server

3.1.0

Version of Immich Mobile App

3.1.0

Platform with the issue

• Server
• Web
• Mobile

Device make and model

Samsung Galaxy S23 Ultra

Your docker-compose.yml content

# Immich docker-compose.yml — REDACTED FOR BUG REPORT
# Sensitive values removed:
# - Database passwords
# - Private IP addresses
# - Host filesystem paths
# - User-specific storage names

name: immich-server

services:
    database:
        cpu_shares: 90
        command: []
        deploy:
            resources:
                limits:
                    memory: "16339599360"
                reservations: {}
        environment:
            POSTGRES_DB: immich
            POSTGRES_INITDB_ARGS: --data-checksums
            POSTGRES_PASSWORD: <REDACTED>
            POSTGRES_USER: postgres
        image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
        networks:
            immich-server: null
        restart: unless-stopped
        volumes:
            - type: bind
              source: <REDACTED>/immich/postgres
              target: /var/lib/postgresql/data

    immich-machine-learning:
        cpu_shares: 90
        command: []
        deploy:
            resources:
                limits:
                    memory: "16339599360"
                reservations: {}
        image: ghcr.io/immich-app/immich-machine-learning:release
        networks:
            immich-server: null
        restart: unless-stopped
        volumes:
            - type: bind
              source: <REDACTED>/immich/model-cache
              target: /cache

    immich-server:
        cpu_shares: 90
        command: []
        depends_on:
            database:
                condition: service_started
                required: true
            redis:
                condition: service_started
                required: true
        deploy:
            resources:
                limits:
                    memory: "16339599360"
                reservations: {}
        environment:
            DB_DATABASE_NAME: immich
            DB_HOST: database
            DB_PASSWORD: <REDACTED>
            DB_PORT: "5432"
            DB_USERNAME: postgres
            REDIS_HOST: redis
        image: ghcr.io/immich-app/immich-server:release
        networks:
            immich-server: null
        ports:
            - target: 2283
              published: "2283"
              protocol: tcp
        restart: unless-stopped
        volumes:
            - type: bind
              source: <REDACTED>/immich/library
              target: /data

            # External Library 1
            - type: bind
              source: <REDACTED>/external-library-1
              target: /external-library-1

            # External Library 2
            - type: bind
              source: <REDACTED>/external-library-2
              target: /external-library-2

            # External Library 3
            - type: bind
              source: <REDACTED>/external-library-3
              target: /external-library-3

    redis:
        cpu_shares: 90
        command: []
        deploy:
            resources:
                limits:
                    memory: "16339599360"
                reservations: {}
        image: docker.io/valkey/valkey:9@sha256:3acc0687f2a2e1091fae6450d7842dd658c941338cf0a873ddd9e14b9e4ea4dd
        networks:
            immich-server: null
        restart: unless-stopped

networks:
    default:
        name: immich-server_default
    immich-server:
        name: immich-server

x-casaos:
    author: self
    category: self
    hostname: ""
    icon: https://raw.githubusercontent.com/immich-app/immich/384d3a0984eac72722438a3ec0774f94c2eed39e/design/immich-logo-stacked-light.svg
    index: /
    is_uncontrolled: false
    port_map: "2283"
    scheme: http
    store_app_id: immich-server
    title:
        custom: ""
        en_us: immich-server

Your .env content

I don't use it

Reproduction steps

1. Set up Immich with External Libraries pointing to directories located on remote/network-mounted storage.
2. Bind-mount these host directories into the Immich Server container.
3. Configure the corresponding directories as External Libraries in Immich.
4. Populate the External Libraries with a large number of existing photos and videos and allow Immich to index them normally.
5. Leave the External Library storage mounted and accessible to the host and container.
6. Run an automatic or manual External Library scan.
7. During the scan, if the disk crawl temporarily returns zero files for the mounted library (for example, due to a transient mount, network, or storage availability issue), allow the LibraryService scan to continue.
8. Observe that Immich may treat the existing assets as missing/offline.
9. In the affected instance, 68,811 existing assets were subsequently assigned a deletedAt timestamp within approximately 6.6 seconds, even though the physical files were still present on the storage.
10. Verify the database and observe that the affected assets remain registered in the asset table but are marked as deleted.
11. Restore the physical storage availability and run another External Library scan.
12. In our validation test, the same libraries were scanned successfully, the files were found, and 0 assets were offlined, demonstrating that the physical files were still present and accessible.

Important note:

We have not been able to reproduce the original failure deterministically under controlled conditions. The steps above describe the observed failure scenario and the environment in which it occurred.

The critical observation is that, during the incident, LibraryService reported 0 files found during the disk crawl for all three External Libraries, followed by a mass offline/deletion state affecting 68,811 existing assets. The physical files were never deleted and remained available on the underlying storage.

A subsequent scan, after the storage was accessible again, found the files normally and reported 0 assets offlined.

Relevant log output

Additional information

==========================================

ORIGINAL INCIDENT — 2026-08-14

==========================================

[Nest] LOG [Microservices:LibraryService] Initiating scan of all external libraries...

[Nest] LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library ...
[Nest] LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library ...
[Nest] LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library ...

[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 0 file(s) found on disk and queued 0 file(s) for import into library
[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 0 file(s) found on disk and queued 0 file(s) for import into library
[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 0 file(s) found on disk and queued 0 file(s) for import into library

Database evidence:

68,811 assets received deletedAt timestamps between:

2026-08-14 00:00:00.573 UTC

and

2026-08-14 00:00:07.211 UTC

==========================================

RECOVERY — 2026-08-17

==========================================

[Nest] LOG [Api:TrashService] Restored 1 asset(s) from trash
[Nest] LOG [Api:TrashService] Restored 27433 asset(s) from trash

==========================================

VALIDATION SCAN — 2026-08-17

==========================================

[Nest] LOG [Microservices:LibraryService] Initiating scan of all external libraries...

[Nest] LOG [Microservices:LibraryService] 0 asset(s) out of 2532 were offlined due to import paths and/or exclusion pattern(s) in library
[Nest] LOG [Microservices:LibraryService] 0 asset(s) out of 27451 were offlined due to import paths and/or exclusion pattern(s) in library
[Nest] LOG [Microservices:LibraryService] 0 asset(s) out of 39186 were offlined due to import paths and/or exclusion pattern(s) in library

[Nest] LOG [Microservices:LibraryService] Checked existing asset(s): 0 offlined, 2468 onlined, 0 updated, 64 unchanged of current batch of 2532 (Total progress: 2532 of 2532, 100.0 %) in library

[Nest] LOG [Microservices:LibraryService] Checked existing asset(s): 0 offlined, 7447 onlined, 0 updated, 4 unchanged of current batch of 7451 (Total progress: 27451 of 27451, 100.0 %) in library

[Nest] LOG [Microservices:LibraryService] Checked existing asset(s): 0 offlined, 8910 onlined, 0 updated, 276 unchanged of current batch of 9186 (Total progress: 39186 of 39186, 100.0 %) in library

[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 2468 file(s) found on disk and queued 0 file(s) for import into

[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 27436 file(s) found on disk and queued 2 file(s) for import into

[Nest] LOG [Microservices:LibraryService] Finished disk crawl, 38910 file(s) found on disk and queued 0 file(s) for import into

==========================================

SECONDARY ISSUE — MACHINE LEARNING

==========================================

[Nest] ERROR [Microservices] Unable to run job handler (AssetDetectFaces):
Error: Machine learning request
'{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}'
failed for all URLs

[Nest] ERROR [Microservices] Unable to run job handler (AssetDetectFaces):
Error: Machine learning request
'{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}'
failed for all URLs

[danieldietzler]

Please don't send us slop. Your docker mount point(s) likely broke (or at least the mount point flaked for a brief moment), causing docker to lose the mount.

[Germs2004]

I just came across this issue as I'm researching how to safely point Immich to a SMB share for storing its assets. It's unclear how Immich handles situations where the SMB may go offline temporarily during one of its scans. Does it lose data and require a manual restore from backup, or does it just temporarily mark files as missing and later automatically corrects itself when the SMB share comes back online? Does it matter whether the assets are in an External Library or Internal Library?

This documentation makes it sound like it triggers a 30-day countdown to data loss. If the user fixes the mount problem within 30 days, it's unclear whether some metadata would still be lost since it says the rediscovered file would be considered a freshly added new file.

If the import paths are edited in a way that an external file is no longer in any import path, it will be removed from the library in the same way a deleted file would. If the file is moved back to an import path, it will be added again as if it was a new file.

I think it would be safest to just label them as a "missing asset", but not move them to trash or anything harmful. The "missing asset" tag could be automatically removed if the asset is detected during a future scan, so zero data loss and zero user effort. Or, the user could log in, select the set of "missing assets," and confirm that they do want to delete those assets.

Edit: Looks like other people are reporting related issues with temporarily unavailable libraries. It really seems like the automatic-trash idea is overly risky for a system users trust to store photos. Another.

[danieldietzler]

People who use an external library also want Immich to sync deletes, if they delete a file on disk. It's virtually impossible to distinguish such a delete from a library becoming unmounted within a docker container. As such, after 30 days, Immich-only metadata (faces and album membership) will be gone, since the Immich-internal asset will be deleted then. This of course does not affect the file on disk.
Generally though, having flaky mounts is very bad and really you should mount it in a way that if the mount goes down, the container will be stopped (often called a "hard mount").

All of this applies to external libraries. The internal library expects a stable mount to work properly (and you may have all sorts of weirdness while using Immich), but will not react to changes on disk (e.g., delete the asset from Immich after x days)

[Germs2004]

It's virtually impossible to distinguish such a delete from a library becoming unmounted within a docker container.

Would it work to just check whether 100% of the asset files in an external library are suddenly missing? Or let Immich check for a specific flag file on the local mount folder, which is only visible when the SMB mount breaks?

Generally though, having flaky mounts is very bad and really you should mount it in a way that if the mount goes down, the container will be stopped (often called a "hard mount").

I looked up "hard mount" and it looks like an NFS feature, not a SMB feature. With a hard mount, if the remote server goes down for some reason, queries against that hard mount will get an error rather than return an empty directory. If I understand right, you're saying that Immich won't move my assets to the trash if its library scanner gets a hard mount error, but it will move them to trash if it sees an empty directory.

Flag files, hard mounts, or Docker down/up scripts all seem like a more complicated and error-prone solution than just configuring Immich to pretty please don't deliberately delete metadata automatically. Apologies for being so stern about this - it's a truly scary behavior if using remote storage.

Thank you.

[daciodf]

In my case, since I don't know anything about databases, I used an AI to help me troubleshoot the issue.
Yes, the files had been moved to the Trash; had I not noticed within the 30-day window—or 15 days, in my case—I would have lost everything. After attempting to restore everything from the Trash, the system failed to move the files back to the "Photos" folder. However, with the help of ChatGPT, I was able to access the database directly and flag the files; this changed their status from "deleted" and restored them to my gallery.
I’m not sure exactly when the SMB mapping went offline, but I don't think the system should delete files simply because it lost its connection.

[danieldietzler]

I was able to access the database directly and flag the files; this changed their status from "deleted" and restored them to my gallery.

This is very bad. You shouldn't run some random queries you got from ChatGPT, that's extremely dangerous for your files. If the mount is back up, Immich would've picked that up the next time library scanning ran

[daciodf]

The SMB share was fine, the Immich server was running, and I had already run a library scan; I even ran the library scan manually, yet the file status didn't change, nor did it accept the restore from the trash. Anyway, thanks—fortunately, it worked out. Thanks.