Encoded video files can be left corrupt when transcode process is killed mid-write (no atomic rename)

[yunluyl]

The bug

handleVideoConversion in server/src/services/media.service.ts points ffmpeg directly at the final encoded video path (StorageCore.getEncodedVideoPath(asset)). fluent-ffmpeg opens that file, streams mdat, and only rewrites the header (-movflags faststart) at the end.

If the server process is killed mid-write — OOM, pod/container termination, or a BullMQ stalled-retry spawning a concurrent writer — the final encoded file is left truncated with no moov atom. The DB row from a previous successful encode is not updated by the failed run, so it still points at the now-corrupt file. The UI serves the broken video: red "!" badge, no auto-preview, unplayable.

I hit this in my own deployment running v2.7.5. 43 files on disk were unplayable; ffprobe on each one reported [mov,mp4,m4a,3gp,3g2,mj2 @ ...] moov atom not found. The only way to recover was manually deleting the corrupt files (disk + asset_file row) and letting the queue regenerate them.

Relevant source:

// server/src/services/media.service.ts  (around lines 587–668)
const output = StorageCore.getEncodedVideoPath(asset);
this.storageCore.ensureFolders(output);

// ...

try {
  await this.mediaRepository.transcode(input, output, command);   // writes directly to final path
} catch (error) {
  // HW-decode -> SW-decode fallback, also writes directly to `output`
  // fully-disabled fallback, also writes directly to `output`
}

this.logger.log(`Successfully encoded ${asset.id}`);
await this.assetRepository.upsertFile({ assetId, type: EncodedVideo, path: output, isEdited: false });

Ordering (ffmpeg → DB upsert) is already correct. The gap is that ffmpeg writes to the live file, so any kill mid-write leaves <output> truncated while the DB still references it.

Proposed fix (no code review being requested — high-level only):

• Write ffmpeg output to a temp sibling <output>.tmp.
• On ffmpeg success, rename(<output>.tmp, <output>) — atomic within the same filesystem.
• On failure, unlink <output>.tmp and rethrow (or return JobStatus.Failed for the disabled-accel early-return path).
• On entry, if a stale <output>.tmp exists from a prior SIGKILL, unlink it before starting. StorageRepository.existsSync + .unlink are already in place.

Existing helpers that can be reused:

• StorageRepository.rename — server/src/repositories/storage.repository.ts:77-79
• StorageRepository.unlink — server/src/repositories/storage.repository.ts:142-152 (ENOENT-safe)
• StorageRepository.existsSync — server/src/repositories/storage.repository.ts:188-189

No verification step (e.g., post-ffprobe) is needed — ffmpeg's exit code is already the correctness signal. No new config or dependencies.

Out of scope for this bug: the same pattern exists in thumbnail/preview generation but with a smaller blast radius. BullMQ job config (attempts, lockDuration, maxStalledCount) is unrelated; the atomic-rename fix makes the writer crash-safe regardless of concurrent writers racing on the temp file.

I had initially opened PR #27972 with the above fix + tests, then closed it per CONTRIBUTING.md ("Use of generative AI") since the code was drafted with significant LLM assistance. Filing as an issue instead so a maintainer (or any human contributor) can own the implementation. If a reference diff is useful, the branch is at https://github.com/yunluyl/immich/tree/fix/atomic-rename-encoded-video — but please treat it as one possible shape of the fix, not something to land directly.

The OS that Immich Server is running on

Linux (Kubernetes / Helm chart deployment, NVIDIA GPU nodes)

Version of Immich Server

v2.7.5

Version of Immich Mobile App

N/A — I don't use the mobile app; this is a server-side bug.

Platform with the issue

• Server
• Web
• Mobile

Device make and model

N/A

Your docker-compose.yml content

# N/A — deployed via the Immich Helm chart on Kubernetes, not docker-compose.
# The bug is in the server transcode flow and is independent of the deployment
# topology; any installation that can be OOM-killed or pod-terminated while
# a video conversion job is running is affected.

Your .env content

# N/A — environment injected via Kubernetes Secret/ConfigMap. No deployment-
# specific values are relevant to this bug.

Reproduction steps

1. On an Immich server running any recent version (confirmed on v2.7.5; the same code pattern is present on main as of today), queue a Video Conversion job for a non-trivial video (a few hundred MB+ is easiest).
2. While the ffmpeg child process is actively writing the encoded file, kill -9 the server process (or trigger an OOM, or delete the pod).
3. Restart the server and observe: the encoded video file at <library>/encoded-video/<user>/<asset>/<id>.mp4 exists on disk but is truncated. ffprobe reports moov atom not found.
4. In the UI, the asset shows a red "!" badge and fails to play. The DB asset_file row for the encoded video still points at the corrupt file; the successful-encode log message from the prior run is misleading because a later killed run silently overwrote the good file.

Relevant log output

# From `ffprobe` on a corrupt encoded file produced by this bug:
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x...] moov atom not found
/usr/src/app/upload/encoded-video/<user-id>/<asset-dir>/<asset-id>.mp4: Invalid data found when processing input

# Prior successful-encode log line from the run that produced the now-overwritten good file is still in history:
[Microservices:MediaService] Successfully encoded <asset-id>

Additional information

Summary of the 43 files I found in my deployment: all were videos that had been transcoded at least once successfully, then picked up again for re-transcode (either by a policy change or a manual Missing re-queue), where the second run was interrupted and corrupted the file in place.

Happy to answer follow-ups here or join the Discord if a maintainer wants to discuss.

[github-actions]

This issue has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.

[meesfrensel]

Sorry, the LLM root cause and proposed fix are way too verbose to be helpful. What is the short version of how this can happen? I got the following, is that correct?

1. Video is uploaded, transcode is successful and encoded video exists on disk and in the DB
2. The video gets re-transcoded, but ffmpeg dies midway through for whatever reason
3. Now, the DB entry is still there (from the first successful transcode), but the file is corrupt