Error when trying to backup

[brunoescalona]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Every night or when I create a custom job to do a database dump the job is failing.

The OS that Immich Server is running on

Ubuntu 25.10

Version of Immich Server

v2.5.6

Version of Immich Mobile App

v2.5.6

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    user: "1000:1000"
    volumes:
      - immich-library:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: unless-stopped
    healthcheck:
      disable: false
    networks:
      - immich

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - immich-cache:/cache
    env_file:
      - .env
    restart: unless-stopped
    healthcheck:
      disable: false
    networks:
      - immich

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:fec42f399876eb6faf9e008570597741c87ff7662a54185593e74b09ce83d177
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped
    networks:
      - immich

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - immich-database:/var/lib/postgresql/data
    user: "1000:1000"
    restart: unless-stopped
    networks:
      - immich

networks:
  immich:
    external: true

volumes:
  model-cache:
  immich-library:
    external: true
  immich-cache:
    external: true
  immich-database:
    external: true

Your .env content

TZ=Europe/Madrid
IMMICH_VERSION=release
DB_PASSWORD=test
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1.Just create a job for a database dump

Relevant log output

Logs from the postgres container 

2026-03-19 11:06:10.914 UTC [1] LOG:  database system is ready to accept connections
2026-03-19 11:50:34.750 UTC [214] ERROR:  missing chunk number 0 for toast value 36946 in pg_toast_17280
2026-03-19 11:50:34.750 UTC [214] STATEMENT:  COPY public.smart_search ("assetId", embedding) TO stdout;
2026-03-19 11:51:49.624 UTC [231] ERROR:  missing chunk number 0 for toast value 36946 in pg_toast_17280
2026-03-19 11:51:49.624 UTC [231] STATEMENT:  COPY public.smart_search ("assetId", embedding) TO stdout;
2026-03-19 15:34:15.730 UTC [923] ERROR:  duplicate key value violates unique constraint "UQ_assets_owner_checksum"
2026-03-19 15:34:15.730 UTC [923] DETAIL:  Key ("ownerId", checksum)=(56857443-873e-454b-9a66-5aadff06eb82, \x9c5423e2dc5383a5e864469e3706f95998f7f9f2) already exists.

2026-03-19 15:34:15.730 UTC [923] STATEMENT:  insert into "asset" ("ownerId", "libraryId", "checksum", "originalPath", "deviceAssetId", "deviceId", "fileCreatedAt", "fileModifiedAt", "localDateTime", "type", "isFavorite", "duration", "visibility", "originalFileName") values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) returning *
2026-03-19 15:34:15.946 UTC [1083] ERROR:  missing chunk number 0 for toast value 36947 in pg_toast_17280
2026-03-19 15:34:15.946 UTC [1083] STATEMENT:  with "cte" as (select "asset"."id" as "assetId", "asset"."duplicateId", smart_search.embedding <=> $1 as "distance" from "asset" inner join "smart_search" on "asset"."id" = "smart_search"."assetId" where "asset"."visibility" in ('archive', 'timeline') and "asset"."ownerId" = any($2::uuid[]) and "asset"."deletedAt" is null and "asset"."type" = $3 and "asset"."id" != $4::uuid and "asset"."stackId" is null order by "distance" limit $5) select * from "cte" where "cte"."distance" <= $6
2026-03-19 15:34:51.110 UTC [980] ERROR:  missing chunk number 0 for toast value 36950 in pg_toast_17280
2026-03-19 15:34:51.110 UTC [980] STATEMENT:  with "cte" as (select "asset"."id" as "assetId", "asset"."duplicateId", smart_search.embedding <=> $1 as "distance" from "asset" inner join "smart_search" on "asset"."id" = "smart_search"."assetId" where "asset"."visibility" in ('archive', 'timeline') and "asset"."ownerId" = any($2::uuid[]) and "asset"."deletedAt" is null and "asset"."type" = $3 and "asset"."id" != $4::uuid and "asset"."stackId" is null order by "distance" limit $5) select * from "cte" where "cte"."distance" <= $6
2026-03-19 16:14:02.430 UTC [1] LOG:  received fast shutdown request
2026-03-19 16:14:02.445 UTC [1] LOG:  aborting any active transactions
2026-03-19 16:14:02.447 UTC [1080] FATAL:  terminating connection due to administrator command
2026-03-19 16:14:02.447 UTC [1082] FATAL:  terminating connection due to administrator command
2026-03-19 16:14:02.449 UTC [1] LOG:  background worker "logical replication launcher" (PID 29) exited with exit code 1
2026-03-19 16:14:02.456 UTC [1085] FATAL:  terminating connection due to administrator command
2026-03-19 16:14:02.462 UTC [1081] FATAL:  terminating connection due to administrator command
2026-03-19 16:14:02.467 UTC [1084] FATAL:  terminating connection due to administrator command
2026-03-19 16:14:02.496 UTC [24] LOG:  shutting down
2026-03-19 16:14:02.597 UTC [1] LOG:  database system is shut down


Logs from the server

[Nest] 7  - 03/20/2026, 2:00:00 AM     LOG [Microservices:DatabaseBackupService] Database Backup Starting. Database Version: 14
/usr/lib/postgresql/14/bin/pg_dump exited (1)
[Nest] 7  - 03/20/2026, 2:00:05 AM   ERROR [Microservices:DatabaseBackupService] Database Backup Failure: Error: /usr/lib/postgresql/14/bin/pg_dump non-zero exit code (1)
pg_dump: error: Dumping the contents of table "smart_search" failed: PQgetResult() failed.
pg_dump: error: Error message from server: ERROR:  missing chunk number 0 for toast value 36946 in pg_toast_17280
pg_dump: error: The command was: COPY public.smart_search ("assetId", embedding) TO stdout;
[Nest] 7  - 03/20/2026, 2:00:05 AM   ERROR [Microservices:{}] Unable to run job handler (DatabaseBackup): Error: /usr/lib/postgresql/14/bin/pg_dump non-zero exit code (1)
pg_dump: error: Dumping the contents of table "smart_search" failed: PQgetResult() failed.
pg_dump: error: Error message from server: ERROR:  missing chunk number 0 for toast value 36946 in pg_toast_17280
pg_dump: error: The command was: COPY public.smart_search ("assetId", embedding) TO stdout;
Error: /usr/lib/postgresql/14/bin/pg_dump non-zero exit code (1)
pg_dump: error: Dumping the contents of table "smart_search" failed: PQgetResult() failed.
pg_dump: error: Error message from server: ERROR:  missing chunk number 0 for toast value 36946 in pg_toast_17280
pg_dump: error: The command was: COPY public.smart_search ("assetId", embedding) TO stdout;
    at ChildProcess.<anonymous> (/usr/src/app/server/dist/repositories/process.repository.js:91:23)
    at ChildProcess.emit (node:events:508:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)

Additional information

No response

[bondeabhijeet]

This looks like a classic case of a Postgres database corruption, specifically the part where Immich stores Smart Search embeddings. It does not seem to be a Immich bug as the smart_search table data seems to have been lost or corrupted.
Steps you can take to resolve this issue:

1. remove the broken smart_search data
2. then let Immich rebuild Smart Search again

[brunoescalona]

Is there any guide in the documentation to do that?

Or should I just remove the table related to smart search and that's it?

[bondeabhijeet]

1. I can suggest you to stop the containers using:
   docker compose stop immich-server immich-machine-learning redis

2. Then Backup using this

3. Start the container:
   docker compose up -d database

4. Clear the corrupted ML table
   docker exec -it immich_postgres sh -lc ' psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "TRUNCATE TABLE smart_search;" '

5. Then bring the other containers live
   docker compose up -d immich-server immich-machine-learning redis

6. Then Go to Administration -> Job Queues -> Smart Search and run it for all the items (just click the infinity symbol) so that it rebuilds the Smart Search again.