Job queues crawl at ~0.3 jobs/min on 3.2.0 — single worker, active pinned at 1, workers idle on BZPOPMIN

[itskevinb]

Summary

On 3.2.0, all job queues crawl at roughly 0.3 jobs/min. thumbnailGeneration has ~3,000 jobs in :wait, active is pinned at exactly 1 indefinitely, and the workers are connected but idle. Nothing is resource constrained. A docker restart immich_server briefly produces real concurrency (4 jobs at once, queues visibly draining), which decays back to 1 within about 10-15 minutes and then effectively stops.

This looks similar to #24972, but that turned out to be a Docker network name collision where the workers and the jobs were on two different Redis instances. That is not the case here (ruled out below).

Environment

• Immich 3.2.0, Docker Compose, in an unprivileged LXC on Proxmox
• BullMQ 5.81.3
• Redis 6.2-alpine (not Valkey)
• Postgres image postgres:14-vectorchord0.4.3-pgvectors0.2.0
• ML container is the release-openvino image
• ~132,000 assets. Originals and the external library live on a CIFS/SMB mount; thumbnails are on local disk.

Symptoms

Measured over 18 minutes with no config writes and no new uploads:

Metric	Value
Thumbnails completed	5 in 18 min
thumbnailGeneration:wait	~3,006
thumbnailGeneration:active	1, constantly
immich_server CPU	25-35% steady

The oldest queued job is ~44 hours old. The backlog does drain, roughly 100x slower than it should.

Redis evidence

Sampled three times, 30 seconds apart, all identical:

• 79 connected clients, 78 of them with age > 300s — no churn, so workers are not being recreated in a loop
• 13 clients blocked in BZPOPMIN — workers are parked waiting for work
• llen immich_bull:thumbnailGeneration:wait = ~3,005
• zcard immich_bull:thumbnailGeneration:marker = 1
• Exactly one worker client for this queue: immich_bull:dGh1bWJuYWlsR2VuZXJhdGlvbg==:w:microservices

The single active job is genuine, not an orphan: ttl immich_bull:thumbnailGeneration:<id>:lock returns ~21s and is being renewed, and processedOn advances as jobs rotate.

At IMMICH_LOG_LEVEL=verbose, [Microservices:JobRepository] Waiting for thumbnailGeneration queue to stop... logs once per second, continuously, starting 3 seconds after boot, with no config change having been made. Worth noting for anyone else debugging this: that line is verbose-level, so at default logging it is invisible and its absence means nothing.

Ruled out

• Wrong Redis / network collision (the cause in Thumbnail generation not working - BullMQ queue stuck #24972). One Redis container, one user-defined bridge network, four containers total. REDIS_HOSTNAME resolves to the same instance that holds the :wait list the workers are attached to.
• Disk. 126 GB free, 7% inodes used.
• Storage latency. The exact file a "stuck" job was processing (670 KB JPEG) stats in 58 ms and reads in 11 ms.
• Host pressure. /proc/pressure/{cpu,io,memory} all ~0 on the Proxmox host, 55-76% idle, no D-state processes.
• Postgres. 19 connections, 13 idle, 1 active. Pool is not exhausted.
• Network to the ML container. A fetch loop run inside immich_server with Node's own runtime: 15/15 success at 5-12 ms, DNS returns a single A record. Despite that, real jobs still intermittently throw UND_ERR_CONNECT_TIMEOUT against both the ML container and Postgres, which is what makes me suspect the microservices event loop is being blocked rather than any network fault.

Tried, no effect

• Raising job.thumbnailGeneration.concurrency from 1 to 3 via PUT /api/system-config. It does apply after a container restart (verbose logs show three thumbnails generated within the same second), but active settles back to 1 permanently and the rate is unchanged. Only one worker connection ever registers for the queue.
• Clearing ~4,400 stale failed jobs across all queues with clear-failed. Those failures were all write CONNECT_TIMEOUT immich_postgres:5432 from an unrelated database outage, so they were wreckage rather than cause.
• Changing the CIFS mounts from soft to hard with actimeo=30. No change. The host kernel log contains zero SMB messages either way.
• Multiple container restarts. Each gives a short burst that decays.

Question

Is there a known condition where the microservices worker registers a single connection for a queue and never scales to the configured concurrency, while the Waiting for ... queue to stop poll never completes? I am happy to run further diagnostics or provide a heap/CPU profile from the microservices process.

[itskevinb]

Follow-up: I think I've found at least one concrete bug here, and it's reproducible from the code.

Unbounded wait in waitForQueueCompletion

handleQueueRecognizeFaces in server/src/services/person.service.ts does:

await this.jobRepository.waitForQueueCompletion(QueueName.ThumbnailGeneration, QueueName.FaceDetection);

and waitForQueueCompletion in server/src/repositories/job.repository.ts is:

let pending = await getPending();

while (pending.length > 0) {
  this.logger.verbose(`Waiting for ${pending[0]} queue to stop...`);
  await setTimeout(1000);
  pending = await getPending();
}

where isActive() is just getActiveCount() > 0.

There is no timeout and no upper bound. If thumbnailGeneration has a backlog large enough that it always has at least one job in flight, getActiveCount() never returns 0 and this loop can never exit.

What that looks like on a live system

facialRecognition:    wait=1  active=1   job=11101 name=FacialRecognitionQueueAll  running for 515s and climbing
thumbnailGeneration:  wait=3047 active=1 job=18075 name=AssetGenerateThumbnails

The FacialRecognitionQueueAll job never completes. Because facialRecognition effectively runs at concurrency 1, that slot is occupied permanently, so facial recognition can never run again for as long as a thumbnail backlog exists. The job also logs Waiting for thumbnailGeneration queue to stop... once per second indefinitely, which is the symptom I opened this thread with.

This also looks like a plausible source of the job stalled more than allowable limit failures I found across several queues, including backupDatabase: a job that never returns eventually gets treated as stalled and re-queued, and on the next run it deadlocks the same way.

It is self-reinforcing on any instance with a persistent backlog, which is exactly when you most want face detection to make progress.

Suggested fix

Bound the wait. Something like a maximum duration or iteration count, after which the job logs a warning and either returns JobStatus.Skipped or proceeds anyway:

async waitForQueueCompletion(...queues: QueueName[]): Promise<boolean> {
  const deadline = Date.now() + MAX_QUEUE_WAIT_MS;
  let pending = await getPending();

  while (pending.length > 0) {
    if (Date.now() > deadline) {
      this.logger.warn(`Timed out waiting for ${pending.join(', ')} to drain; continuing`);
      return false;
    }
    this.logger.verbose(`Waiting for ${pending[0]} queue to stop...`);
    await setTimeout(1000);
    pending = await getPending();
  }
  return true;
}

An alternative, if the intent is specifically "don't start face recognition while thumbnails are still being produced", would be to re-queue the job with a delay rather than holding a worker slot while busy-waiting. That frees the slot and avoids the permanent occupancy.

Happy to open a PR for whichever shape you prefer.

Caveat, to be accurate: this bug clearly explains the endless log loop, the permanently occupied facialRecognition slot, and plausibly the stalled-job failures. I do not think it explains the thumbnail throughput itself, which is the ~0.3 jobs/min in the original post. thumbnailGeneration jobs do rotate, just extremely slowly, and that remains unexplained.

[itskevinb]

Second follow-up: I benchmarked the underlying work to rule out hardware, and the gap is stark.

The actual work is fast

All measured inside the running immich_server container, against files that are currently sitting unprocessed in the thumbnailGeneration queue.

Image path, via the bundled sharp:

metadata:        410 ms   (1320x2868 png)
preview 1080px:  273 ms   (quality 70, 30,835 bytes out)

Video path, via the bundled ffmpeg, single frame extraction on four queued videos:

IMG_0392.MP4    3,850,254 bytes   rc=0   707 ms
893cbca9....MOV 3,419,319 bytes   rc=0   643 ms
IMG_0628.MP4    4,316,207 bytes   rc=0  1301 ms
388ba8c7....MOV 3,580,114 bytes   rc=0   844 ms

Storage and host:

sequential write:          198 MB/s
200 x 16K writes w/ fsync: 6 ms per file
read of a queued source:   11 ms (670 KB JPEG, stat 58 ms)
host pressure (PSI):       cpu/io/memory all ~0, 55-76% idle
CPU:                       Intel i7-6770HQ, 8 cores

The gap

Each unit of real work costs roughly 0.3 to 1.3 seconds. The queue completes roughly one job per minute. That is a 50-100x discrepancy that does not live in the disk, the CPU, the network share, or the iGPU — all of them are idle and fast.

For completeness on the queue composition, since large videos were my first suspicion and they are not the problem:

Source	Type	Count awaiting thumbnails
upload	IMAGE	12,090
upload	VIDEO	4,961
external library	VIDEO	1,523
external library	IMAGE	9

Also worth noting

thumbnailGeneration never shows more than 1 active job in steady state, even with job.thumbnailGeneration.concurrency set to 3 and the server restarted so the value is picked up. Immediately after a restart it briefly reaches 4 concurrent and queues visibly drain, then within about 10-15 minutes it settles back to 1 and effectively stops. In startWorkers() every worker is constructed with a hardcoded concurrency: 1 and only raised later via setConcurrency, which may be relevant to why it does not stay raised.

Happy to run anything else that would help, including a CPU profile or heap snapshot of the microservices process.

[itskevinb]

Resolved on my end, and it was a configuration hazard rather than an Immich bug. Posting the root cause in case it helps others.

Root cause: library watching on a network share

I have an external library of 89,735 assets whose import path is a CIFS/SMB mount, with library.watch.enabled = true.

File watching cannot use native change notifications on a network mount, so it degrades to polling the tree. Against ~90k files over SMB, that produced a metadata storm:

SMBs:             13,674,359   (~1,700 ops/sec)
Creates:           4,499,435
QueryInfos:        4,329,612
QueryDirectories:    345,348
Bytes read:          227 MB     <-- almost no actual data

Measured SMB operation latency on that mount is 186 ms serially (though ~20 parallel reads complete in 159 ms wall, so it parallelises fine — latency, not bandwidth, is the problem).

Node does filesystem work on a small libuv thread pool. The watcher kept those threads permanently blocked: I found 5 threads in uninterruptible D state with wchan = wait_for_response (the kernel SMB client waiting on the server). Every job that needs to touch a file then queues behind the watcher, which is why throughput collapsed while CPU, disk and network all looked idle.

Result of disabling library.watch.enabled

Metric	Before	After
Thumbnails/min	0.25	50
SMB ops/sec	~1,700	43
thumbnailGeneration active	1	3 (the configured concurrency)
Backlog	3,053	draining

A 200x improvement, and the configured concurrency of 3 immediately started being honoured — it never was while the pool was starved.

Why it was hard to see

Everything I benchmarked in isolation was fast, which sent me down several wrong paths:

• sharp 1080px preview: 273 ms; ffmpeg single-frame extract on queued videos: 643-1301 ms
• Local disk: 198 MB/s sequential, 6 ms per fsync'd small file
• Reading one queued source file: 11 ms
• Postgres: 19 connections, idle
• Host /proc/pressure/{cpu,io,memory}: all ~0 — this is the trap. PSI tracks block-device I/O and does not account for network-filesystem waits, so a machine whose thread pool is entirely blocked on SMB still reports zero I/O pressure and looks idle.

The single-operation benchmarks were fast precisely because they bypassed the saturated thread pool.

This also retroactively explains the UND_ERR_CONNECT_TIMEOUT errors I mentioned in the original post, against both the ML container and Postgres, on network paths that tested clean on demand. A starved thread pool starves everything, not just image processing.

Suggestions

1. A warning in the admin UI when library.watch.enabled is on and an import path is a network mount would have saved a lot of time here. The docs mention watching may not work on network shares, but the failure mode is not "watching doesn't work" — it is "the entire job system silently grinds to a halt", which is much harder to attribute.
2. The unbounded waitForQueueCompletion I reported in the comment above is still a genuine bug worth fixing independently. It was not the cause of the throughput problem, but it does permanently block facialRecognition on any instance with a persistent thumbnail backlog, and it busy-waits while holding a worker slot.

Thanks, and apologies for the noise on the throughput half of this.