[Feature] Configurable retry policy for queue jobs — transient storage failures are permanently fatal

[conghuang]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

Hi team, thanks for the great work on multi-instance support (shared Postgres + Redis).

Setup
Two immich-server v3.3.3 instances sharing one Postgres + Redis, per the multi-instance docs. Each site has local NAS storage with async replication between them. (The same failure also occurs on single-server NAS-backed storage: NFS attribute caching can delay file visibility to the worker. Related: #3901 — users reporting permanent failures after NFS outages.)

Problem
The upload pipeline is a strict chain, and queue jobs are hardcoded to attempts: 1 with no backoff (config.repository.ts, not configurable). If the worker picks up metadataExtraction before the file is visible on its filesystem, the job fails once with ENOENT: no such file or directory — permanently. The asset is then stuck in upload/ forever with no thumbnails/metadata/search vectors, and nothing ever retries it. Recovery today is fully manual per-queue "missing" sweeps (which don't chain downstream, so several queues must be swept in order).

Proposal (scoped)
Make the retry policy configurable instead of hardcoded — e.g. env vars defaulting to attempts=1 / no backoff, fully backward compatible. Alternative tighter variant: classify ENOENT at job start as transient and retry with backoff, while other errors still fail fast; a missing file at job start is almost always a storage-visibility timing issue, not a data problem.

This doesn't ask the project to support async-replication topologies — just makes the pipeline robust to transient storage visibility delays, a fact of life on NAS storage.

Happy to help test or refine. Thanks!

Platform

• Server
• Web
• Mobile

[bo0tzz]

This is an issue with your setup. Immich expects a consistent view of its backing storage.