Attempting to finish incomplete move Failed, some files were missing

[piscesvivian]

The bug

1. I don't know how, may be server restart / upload failure / task failure ? some files were missing.
2. when I run storage template convert task, there server got a lots of ERROR

The OS that Immich Server is running on

Fedora 39

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.4

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    extends:
      file: hwaccel.transcoding.yml
      service: nvenc

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    volumes:
      - ${ML_MODEL_CACHE}:/cache
    env_file:
      - .env
    restart: always
    extends:
      file: hwaccel.ml.yml
      service: cuda

  redis:
    container_name: immich_redis
    image: m.daocloud.io/docker.io/library/redis:6.2-alpine
    restart: always

  database:
    container_name: immich_postgres
    image: m.daocloud.io/docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    ports:
      - 5432:5432
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:z
      # - pgdata:/var/lib/postgresql/data
    restart: always

# volumes:
  # pgdata:
  # model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/f/immich-app/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=./postgresql/data

REDIS_HOSTNAME=immich_redis
HF_ENDPOINT=https://hf-mirror.com

ML_MODEL_CACHE=./models

Reproduction steps

1. You need some file not exists on disk, but I don't know how it happends.
2. Login as Admin
3. Go to PAGE http://127.0.0.1:2283/admin/jobs-status
4. RUN 存储模板转换 [Apply the current 存储模板 to previously uploaded assets]
5. Monitor the server log, and get the error

Relevant log output

[Nest] 14  - 06/20/2024, 3:39:07 PM     LOG [Microservices:MediaService] Attempting to finish incomplete move: upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC => upload/library/c6e140e1-badd-4b5b-b93f-5fceab63da13/2023/01/20/IMG_2577.HEIC
�[Nest] 14  - 06/20/2024, 3:39:07 PM     LOG [Microservices:MediaService] Found file at new location
�[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Problem applying storage template
[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Error: ENOENT: no such file or directory, stat 'upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC'
    at async Object.stat (node:internal/fs/promises:1029:18)
    at async StorageCore.verifyNewPathContentsMatchesExpected (/usr/src/app/dist/cores/storage.core.js:176:25)
    at async StorageCore.moveFile (/usr/src/app/dist/cores/storage.core.js:132:19)
    at async /usr/src/app/dist/services/storage-template.service.js:154:17
    at async /usr/src/app/dist/repositories/database.repository.js:177:23
�[Nest] 14  - 06/20/2024, 3:39:07 PM   ERROR [Microservices:StorageTemplateService] Object:
{
  "id": "35111153-6a9f-460f-9e76-06c2da588c1d",
  "oldPath": "upload/upload/c6e140e1-badd-4b5b-b93f-5fceab63da13/bf/22/bf22018f-ef6e-4ead-a005-6962a398e501.HEIC",
  "newPath": "upload/library/c6e140e1-badd-4b5b-b93f-5fceab63da13/2023/01/20/IMG_2577+1.HEIC"
}

Additional information

Is there a function to check file on disk and if it not exists, then clear the related data.
Also, there is a lots of invalid file(MOV / HEIC / JPEG), in upload/upload folders, it couldn't open by windows photo app or FFMpeg.
Can we supply a way to sync disk with db status, so that they can be consistent