Attempting to finish incomplete move Failed, some files were missing

[piscesvivian]

The bug

1. I don't know how, may be server restart / upload failure / task failure ? some files were missing.
2. when I run storage template convert task, there server got a lots of ERROR

The OS that Immich Server is running on

Fedora 39

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.4

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    extends:
      file: hwaccel.transcoding.yml
      service: nvenc

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    volumes:
      - ${ML_MODEL_CACHE}:/cache
    env_file:
      - .env
    restart: always
    extends:
      file: hwaccel.ml.yml
      service: cuda

  redis:
    container_name: immich_redis
    image: m.daocloud.io/docker.io/library/redis:6.2-alpine
    restart: always

  database:
    container_name: immich_postgres
    image: m.daocloud.io/docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    ports:
      - 5432:5432
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:z
      # - pgdata:/var/lib/postgresql/data
    restart: always

# volumes:
  # pgdata:
  # model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/f/immich-app/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=./postgresql/data

REDIS_HOSTNAME=immich_redis
HF_ENDPOINT=https://hf-mirror.com

ML_MODEL_CACHE=./models

Reproduction steps

1. You need some file not exists on disk, but I don't know how it happends.
2. Login as Admin
3. Go to PAGE http://127.0.0.1:2283/admin/jobs-status
4. RUN 存储模板转换 [Apply the current 存储模板 to previously uploaded assets]
5. Monitor the server log, and get the error

Relevant log output

[Nest] 14  - 06/20/2024, 3:39:07 PM     LOG [Microservices:MediaService] Attempting to finish incomplete move: upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC => upload/library/c6e140e1-badd-4b5b-b93f-5fceab63da13/2023/01/20/IMG_2577.HEIC
�[Nest] 14  - 06/20/2024, 3:39:07 PM     LOG [Microservices:MediaService] Found file at new location
�[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Problem applying storage template
[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Error: ENOENT: no such file or directory, stat 'upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC'
    at async Object.stat (node:internal/fs/promises:1029:18)
    at async StorageCore.verifyNewPathContentsMatchesExpected (/usr/src/app/dist/cores/storage.core.js:176:25)
    at async StorageCore.moveFile (/usr/src/app/dist/cores/storage.core.js:132:19)
    at async /usr/src/app/dist/services/storage-template.service.js:154:17
    at async /usr/src/app/dist/repositories/database.repository.js:177:23
�[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Object:
{
  "id": "35111153-6a9f-460f-9e76-06c2da588c1d",
  "oldPath": "upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC",
  "newPath": "upload/library/c6e140e1-badd-4b5b-b93f-5fceab63da13/2023/01/20/IMG_2577+1.HEIC"
}

Additional information

Is there a function to check file on disk and if it not exists, then clear the related data.
Also, there is a lots of invalid file(MOV / HEIC / JPEG), in upload/upload folders, it couldn't open by windows photo app or FFMpeg.
Can we supply a way to sync disk with db status, so that they can be consistent

[mmomjian]

The invalid files in the upload/upload folder are due to interrupted mobile sync, in general they can be deleted and we will probably add this feature down the line.

For the missing file, you will want to check on your filesystem and see if there is a cause of why a file was lost.