External libraries - Externally removed assets are moved to Trash by IMMICH and can't be emptied

[sigulete]

The bug

I have an external library that is read-only for IMICH and the adding and removing assests is handled externally.
I have set up the cron job to scan the library every hour.

When assets are removed and the library is re-scanned, the removed assets disapear from Home page as expected, but they show in Trash, which is not expected. Because what I see in Trash is simply a thumbnail, the actual asset is already gone.

I believe that externally managed asset shouldn't show in Trash.

Furthermore, when I click on "Empty Trash" in the Web UI, it doesn't work and Trash can't be cleaned. It is now forever building up...

The OS that Immich Server is running on

Fedora

Version of Immich Server

v1.118.2

Version of Immich Mobile App

v1.118.2

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - <mylibrary>:/usr/src/app/external:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Remove an asset from the external library (not using IMMICH)
2. Scan external library (using Web UI)
3. Open Trash and the asset removed will show there (using Web UI and Android App)
4. Click on Empty Trash to try to clean trash (Using Web UI, doesn't work)
5. Click on Empty Trash to try to clean trash (Using Android App, does work. But the Web UI still show it)

Relevant log output

0d50f4619b4e [Nest] 12  - 10/28/2024, 12:44:34 PM   ERROR [Api:GlobalExceptionFilter~wn5sz5xf] Unknown error: Error: ENOENT: no such file or directory, access '/usr/src/app/external/video [dDPAANARsHM].mp4'

Additional information

No response

[bo0tzz]

This is working as expected. Displaying them in trash is currently a placeholder, eventually there'll be a dedicated page for them.

[sigulete]

Ok, so for now, my Trash will keep growing.
Is the page you mentioned a replacement of the Trash?

Is there a way to clean the Trash?

[edif30]

Has this been addressed? Is there a way to delete these from the container manually?

[ripcityhandyman]

@bo0tzz, is there an issue we can track for the implementation of this dedicated page? Thanks!

[Gert-P]

I have the same issue, is there a way to empty my trash to get rid of the assets I deleted from the external library outside of Immich?

[Gert-P]

I have written a script to empty the trash for immich in docker. Just run this on the host. It will remove all assets from the trash including deleted files from read only external libraries.

#!/bin/sh

# Postgres credentials 
DB_USER="postgres"
DB_NAME="immich"

# Immich container names
SERVER_CONTAINER="immich_server"
DB_CONTAINER="immich_postgres"

# Directories inside `immich_server`
UPLOAD_DIR="/usr/src/app/upload"
THUMBNAIL_DIR="$UPLOAD_DIR/thumbs"
ENCODED_VIDEO_DIR="$UPLOAD_DIR/encoded-video"

QUERY="DELETE FROM public.assets WHERE \"deletedAt\" IS NOT NULL;"
docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -q -c "$QUERY"

# Check the exit status
if [ $? -eq 0 ]; then
    echo "Deleted all assets from trash, doing cleanup..."
else
    echo "Error occurred while deleting assets."
fi

# Temp files for valid paths
VALID_THUMBNAILS_FILE="/tmp/valid_thumbnails.txt"
VALID_ENCODED_VIDEOS_FILE="/tmp/valid_encoded_videos.txt"
ALL_FILES_TMP="/tmp/all_files.txt"
ORPHANED_FILES_TMP="/tmp/orphaned_files.txt"

# Counters for summary
TOTAL_ORPHANS_THUMBNAIL=0
TOTAL_ORPHANS_ENCODED=0
TOTAL_DELETED_THUMBNAIL=0
TOTAL_DELETED_ENCODED=0

# Function to get valid file paths from database and store them in a file
get_valid_files() {
    local query="$1"
    local output_file="$2"
    docker exec -i $DB_CONTAINER psql -U $DB_USER -d $DB_NAME -t -c "$query" | \
        sed -E 's/^ +//;s/ +$//' | awk 'NF' | sort > "$output_file"
}

# Function to identify (and optionally delete) orphaned files
process_orphans() {
    local dir="$1"
    local valid_files_file="$2"
    local file_type="$3"
    local total_orphans_var="TOTAL_ORPHANS_${file_type}"
    local total_deleted_var="TOTAL_DELETED_${file_type}"

    echo "Checking for orphaned files in: $dir"

    # List all actual files in the directory inside the container (relative paths) excluding `.immich` files
    docker exec -i $SERVER_CONTAINER find "$dir" -type f ! -name "*.immich" | sed "s|$UPLOAD_DIR/||" | sort > "$ALL_FILES_TMP"

    # Find orphaned files using `comm`
    comm -23 "$ALL_FILES_TMP" "$valid_files_file" > "$ORPHANED_FILES_TMP"

    # Count orphaned files
    ORPHAN_COUNT=$(wc -l < "$ORPHANED_FILES_TMP")
    eval "$total_orphans_var=$ORPHAN_COUNT"
    if [ "$ORPHAN_COUNT" -gt 0 ]; then 
        docker exec -i $SERVER_CONTAINER sh -c "
            while IFS= read -r file; do
                rm \"$UPLOAD_DIR/\$file\"
            done < /dev/stdin
        " < "$ORPHANED_FILES_TMP"
        eval "$total_deleted_var=$ORPHAN_COUNT"
    fi
}

# Get valid file lists from database
echo "Fetching valid file paths from database..."
get_valid_files "
    SELECT REPLACE(\"thumbnailPath\", 'upload/', '') FROM public.person WHERE \"thumbnailPath\" IS NOT NULL
    UNION
    SELECT REPLACE(\"path\", 'upload/', '') FROM public.asset_files WHERE \"path\" IS NOT NULL;
" "$VALID_THUMBNAILS_FILE"

get_valid_files "
    SELECT REPLACE(\"encodedVideoPath\", 'upload/', '') FROM public.assets WHERE \"encodedVideoPath\" IS NOT NULL AND LENGTH(\"encodedVideoPath\") > 0;
" "$VALID_ENCODED_VIDEOS_FILE"

# Process orphaned files
process_orphans "$THUMBNAIL_DIR" "$VALID_THUMBNAILS_FILE" "THUMBNAIL"
process_orphans "$ENCODED_VIDEO_DIR" "$VALID_ENCODED_VIDEOS_FILE" "ENCODED"

# Print summary
echo "Deleted $TOTAL_DELETED_THUMBNAIL orphaned thumbnail files."
echo "Deleted $TOTAL_DELETED_ENCODED orphaned encoded video files."
echo "Cleanup Completed!"

# Clean up temp files
rm -f "$VALID_THUMBNAILS_FILE" "$VALID_ENCODED_VIDEOS_FILE" "$ALL_FILES_TMP" "$ORPHANED_FILES_TMP"

[deluxestyle]

thanks. this was helping a lot

[naworm]

You're the goat, clean code ;)

root@srvdocker:~# ./empty-immich-orphans-and-files-from-trash.sh
Deleted all assets from trash, doing cleanup...
Fetching valid file paths from database...
Checking for orphaned files in: /usr/src/app/upload/thumbs
Checking for orphaned files in: /usr/src/app/upload/encoded-video
Deleted 82028 orphaned thumbnail files.
Deleted 1 orphaned encoded video files.
Cleanup Completed!

[Gert-P]

Happy you found it useful :)

[pieshop]

This looked perfect to me. Im on Immich v1.139.3. Unfortunately it errors:

ash-4.4# sh empty-immich-orphans-and-files-from-trash.sh
ERROR: relation "public.assets" does not exist
LINE 1: DELETE FROM public.assets WHERE "deletedAt" IS NOT NULL;
^
Error occurred while deleting assets.
Fetching valid file paths from database...
ERROR: relation "public.asset_files" does not exist
LINE 4: SELECT REPLACE("path", 'upload/', '') FROM public.asset_...
^
ERROR: relation "public.assets" does not exist
LINE 2: ...T REPLACE("encodedVideoPath", 'upload/', '') FROM public.ass...
^
Checking for orphaned files in: /volume1/docker/immich-app/library/thumbs
find: '/volume1/docker/immich-app/library/thumbs': No such file or directory
Checking for orphaned files in: /volume1/docker/immich-app/library/encoded-video
find: '/volume1/docker/immich-app/library/encoded-video': No such file or directory
Deleted 0 orphaned thumbnail files.
Deleted 0 orphaned encoded video files.
Cleanup Completed!

I assume public.assets is no longer valid.

[Gert-P]

This should be an easy fix, I am still on v1.134.0. I will upgrade to the latest version and update the script accordingly.

[pieshop]

I have fixed it and run it - has worked on my machine. Here is the version I fixed for ref:

[pieshop]

#!/bin/sh

#   Testing (default):
#  ./empty-immich-orphans-and-files-from-trash.sh
#  or
#  ./empty-immich-orphans-and-files-from-trash.sh 1
#
#  Actual deletion:
#  ./empty-immich-orphans-and-files-from-trash.sh 0


# Set to 1 for dry-run (test mode), 0 for actual deletion
DRY_RUN=${1:-1}

# Postgres credentials 
DB_USER="postgres"
DB_NAME="immich"

# Immich container names
SERVER_CONTAINER="immich_server"
DB_CONTAINER="immich_postgres"

# Directories inside `immich_server`
UPLOAD_DIR="/data"
THUMBNAIL_DIR="$UPLOAD_DIR/thumbs"
ENCODED_VIDEO_DIR="$UPLOAD_DIR/encoded-video"

if [ "$DRY_RUN" -eq 1 ]; then
    echo "[DRY-RUN] Would delete all assets from trash"
    QUERY="SELECT COUNT(*) FROM public.asset WHERE \"deletedAt\" IS NOT NULL;"
    COUNT=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -t -c "$QUERY" | tr -d ' ')
    echo "[DRY-RUN] Found $COUNT assets in trash that would be deleted"
else
    QUERY="DELETE FROM public.asset WHERE \"deletedAt\" IS NOT NULL;"
    docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -q -c "$QUERY"
    
    # Check the exit status
    if [ $? -eq 0 ]; then
        echo "Deleted all assets from trash, doing cleanup..."
    else
        echo "Error occurred while deleting assets."
    fi
fi

# Temp files for valid paths
VALID_THUMBNAILS_FILE="/tmp/valid_thumbnails.txt"
VALID_ENCODED_VIDEOS_FILE="/tmp/valid_encoded_videos.txt"
ALL_FILES_TMP="/tmp/all_files.txt"
ORPHANED_FILES_TMP="/tmp/orphaned_files.txt"

# Counters for summary
TOTAL_ORPHANS_THUMBNAIL=0
TOTAL_ORPHANS_ENCODED=0
TOTAL_DELETED_THUMBNAIL=0
TOTAL_DELETED_ENCODED=0

# Function to get valid file paths from database and store them in a file
get_valid_files() {
    local query="$1"
    local output_file="$2"
    docker exec -i $DB_CONTAINER psql -U $DB_USER -d $DB_NAME -t -c "$query" | \
        sed -E 's/^ +//;s/ +$//' | awk 'NF' | sort > "$output_file"
}

# Function to identify (and optionally delete) orphaned files
process_orphans() {
    local dir="$1"
    local valid_files_file="$2"
    local file_type="$3"
    local total_orphans_var="TOTAL_ORPHANS_${file_type}"
    local total_deleted_var="TOTAL_DELETED_${file_type}"

    echo "Checking for orphaned files in: $dir"

    # List all actual files in the directory inside the container (relative paths) excluding `.immich` files
    docker exec -i $SERVER_CONTAINER find "$dir" -type f ! -name "*.immich" | sed "s|$UPLOAD_DIR/||" | sort > "$ALL_FILES_TMP"

    # Find orphaned files using `comm`
    comm -23 "$ALL_FILES_TMP" "$valid_files_file" > "$ORPHANED_FILES_TMP"

    # Count orphaned files
    ORPHAN_COUNT=$(wc -l < "$ORPHANED_FILES_TMP")
    eval "$total_orphans_var=$ORPHAN_COUNT"
    if [ "$ORPHAN_COUNT" -gt 0 ]; then
        if [ "$DRY_RUN" -eq 1 ]; then
            echo "[DRY-RUN] Would delete $ORPHAN_COUNT orphaned $file_type files:"
            head -10 "$ORPHANED_FILES_TMP" | while read -r file; do
                echo "  [DRY-RUN] Would delete: $UPLOAD_DIR/$file"
            done
            if [ "$ORPHAN_COUNT" -gt 10 ]; then
                echo "  [DRY-RUN] ... and $((ORPHAN_COUNT - 10)) more files"
            fi
            eval "$total_deleted_var=0"
        else
            docker exec -i $SERVER_CONTAINER sh -c "
                while IFS= read -r file; do
                    rm \"$UPLOAD_DIR/\$file\"
                done < /dev/stdin
            " < "$ORPHANED_FILES_TMP"
            eval "$total_deleted_var=$ORPHAN_COUNT"
        fi
    fi
}

# Get valid file lists from database
echo "Fetching valid file paths from database..."
get_valid_files "
    SELECT REPLACE(path, '/data/', '') FROM public.asset_file WHERE type IN ('thumbnail', 'preview')
    UNION
    SELECT REPLACE(\"thumbnailPath\", '/data/', '') FROM public.person WHERE \"thumbnailPath\" != '';
" "$VALID_THUMBNAILS_FILE"

get_valid_files "
    SELECT REPLACE(\"encodedVideoPath\", '/data/', '') FROM public.asset WHERE \"encodedVideoPath\" IS NOT NULL AND LENGTH(\"encodedVideoPath\") > 0;
" "$VALID_ENCODED_VIDEOS_FILE"

# Process orphaned files
process_orphans "$THUMBNAIL_DIR" "$VALID_THUMBNAILS_FILE" "THUMBNAIL"
process_orphans "$ENCODED_VIDEO_DIR" "$VALID_ENCODED_VIDEOS_FILE" "ENCODED"

# Print summary
if [ "$DRY_RUN" -eq 1 ]; then
    echo "\n[DRY-RUN SUMMARY]"
    echo "Found $TOTAL_ORPHANS_THUMBNAIL orphaned thumbnail files (would delete: $TOTAL_DELETED_THUMBNAIL)"
    echo "Found $TOTAL_ORPHANS_ENCODED orphaned encoded video files (would delete: $TOTAL_DELETED_ENCODED)"
    echo "\nTo actually delete files, run: $0 0"
else
    echo "\n[CLEANUP SUMMARY]"
    echo "Deleted $TOTAL_DELETED_THUMBNAIL orphaned thumbnail files."
    echo "Deleted $TOTAL_DELETED_ENCODED orphaned encoded video files."
    echo "Cleanup Completed!"
fi

# Clean up temp files
rm -f "$VALID_THUMBNAILS_FILE" "$VALID_ENCODED_VIDEOS_FILE" "$ALL_FILES_TMP" "$ORPHANED_FILES_TMP"

[pieshop]

I added a dry run so I could test first :)

[Gert-P]

Great, you did the work for me.

public.assets → public.asset

public.asset_files → public.asset_file

Easy fix.

[bobcate]

Hi,
In your previous post you said to run this on the host. What does that mean? How do I use this code?

Also, I'd like note that v1.137.0 changed the default UPLOAD_LOCATION.
In the code above, on line 12, UPLOAD_DIR is set to the old default value, which I guess will not work for first time users.

[Gert-P]

You can adjust this line according to your upload location.

UPLOAD_DIR="/usr/src/app/upload"

running on the host means you need to run it outside docker. In other words the host of Docker.

[bobcate]

I guess this code is meant for Linux users.
I'm on Windows and I managed to run the code which emptied the trash but failed finding the orphans.
The errors:
find: 'C:/Program Files/Git/usr/src/app/upload/thumbs': No such file or directory
find: 'C:/Program Files/Git/usr/src/app/upload/encoded-video': No such file or directory
Still, it cleaned the trash, thank you for that.

For posterity, below is what I did:
immich server must be running
I installed GIT > ran normal CMD > typed sh to switch to GIT Bash > copy pasted and ran the code
the first 35 lines of the code is enough to just clear the trash

[Gert-P]

You are correct. This is indeed a script for linux

[IMNdi]

Could it be adapted to run from inside docker?

It would solve many issues, for example, on Windows docker runs WSL so there is another layer entirely that I would need to bypass, but there is a terminal inside the container that I can use. I can't access Postgres from the outside.

ETA: DEL key works, empty trash is the issue, see below.

[nielstron]

I had the same issue. What worked for me is to manually select all images (select one, then upper right corner select all) and then trash them.

Note: this works even though "trash all" does not work.

[ghost]

Yup - after messing with external (changed the path so all files now in trash) did manually trigger all as:

1. select one
2. Ctrl + A
3. wait a minute if have some large numbers (if the change is close to the root, it can be all assets and easily to be checked on folders itself in OS that the count is 1:1)
4. confirm popup 'delete permanently' from the trash icon in upper bar that is raised while selecting more

[stevenre3d]

Thanks, this worked! So glad I scrolled to the bottom before bothering with the older script posted above.

[timonrieger]

i assume a timeout error server-side on a big deletion request. Could somebody check the "Empty trash" response in the browser network and share server logs (if anything relevant there)?

[IMNdi]

My browser is silent. When clicking "empty trash" and 3 files are uploaded and 100 external it says "deleted 3 files". On subsequent calls, it says "deleted 0".

Selecting the files and hitting Del or the delete button does remove them after the popup.

The network tab claims the response is 11 bytes "{"count":0}" to POST https://,server>/api/trash/empty . There are no console errors or other hints.

[timonrieger]

for everyone looking for automating the deletion (i had 1000s of photos in trash so selection was not feasible) I used this script using the immichpy library. Set library id to filter by a specific library. comment out the delete_assets call for a dry-run.

See script

import asyncio
from datetime import datetime, timezone

from immichpy import AsyncClient
from immichpy.client.generated.models.asset_bulk_delete_dto import AssetBulkDeleteDto
from immichpy.client.generated.models.metadata_search_dto import MetadataSearchDto

API_KEY = ""
BASE_URL = ""
LIBRARY_ID = None
EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
PAGE_SIZE = 1000


async def main() -> None:
    async with AsyncClient(api_key=API_KEY, base_url=BASE_URL) as client:
        page = 1
        sum = 0
        while True:
            result = await client.search.search_assets(
                MetadataSearchDto(
                    trashedAfter=EPOCH,
                    size=PAGE_SIZE,
                    page=page,
                    isOffline=True,
                    libraryId=LIBRARY_ID
                )
            )
            ids = [item.id for item in result.assets.items]
            if not ids:
                break
            await client.assets.delete_assets(AssetBulkDeleteDto(ids=ids, force=True))
            print(f"page {page}: force-deleted {len(ids)} trashed assets")
            page += 1
            sum += len(ids)
        print(f"Deleted in total: {sum} assets")


asyncio.run(main())

[timonrieger]

this is a bug indeed, thanks for the debug info @IMNdi. fix in #29030

[timonrieger]

The patch was rejected with:

IIRC the behavior here is intentional. Emptying the trash is supposed to delete explicitly trashed assets, not offline assets that can come back online. The use of the trash for offline assets is basically a way to hide them from normal views and is more of a stopgap solution.

In the meantime you can use this suggested workaround by @nielstron

[IMNdi]

Can't you just ask the user or something?

"Deleted 0 assets. There are 2100 assets that are offline. Delete those too?"

At least it would keep people from asking and "working around" by using the delete key. At least the solution is prepackaged, I feel like a comment on a bug is too obscure for a behavior that is intended.

Or just add a second button that is visible when offline assets are listed in the trash.

I suspect, admittedly based on nothing, that the stopgap solution is likely to stick around for a while.