[Feature] Global Job Concurrency

[auanasgheps]

The feature

On poor systems like mine (i3-8300, 4 cores) Immich overwhelms the system when I ingest a lot of media.

I have observed that if I pause ALL services and then have only one running at the time, it is much better: sure it may take more time, but at least won't hog down the CPU to 99.99%, and the system remains usable to a certain degree.

So please, introduce a "Global Job Concurrency", and not only for a specific job, so I can set it to 1 and not have my server die under Immich processing!

Platform

• Server
• Web
• Mobile

[AngelaDMerkel]

I agree with this:

Assuming I have 8 cores, I might limit the microservice to four so transcoding doesn't take forever but then when I add images the CLIP or facial recognition might bring the system to full utilisation. It would be helpful to assign a total system utilisation for the entire stack and then allow concurrency to provide full resources to each process sequentially, rather than hope a particular upload doesn't allow various containers in the stack to stress the system

[jamess60]

+1

Running Immich with docker in a 12 core VM (2x Xeon E5 2697 v2 system). I have 20 thumbnail jobs running at once with no other tasks (manually paused them). CPU spikes 50-100. System becomes unusable if I have any more running.

[jrasm91]

There is no easy way to set a global limit unfortunately

[hirenshah]

Rather than setting an overall limit, would it be possible to run each job sequentially rather than in parallel?

[mertalev]

The main issue with setting a global limit (or making jobs run sequentially) is that the job scheduler we use, BullMQ, handles concurrency at the queue level. Since the scheduler is in charge of handling concurrency, we don't have a non-hacky way of adding a global limit. This is a feature in BullMQ Pro, so it's also unlikely it will be added to BullMQ in the future.

We intend to switch to a different job scheduler at some point (not for this, but to drop Redis and handle jobs in Postgres). It may be more feasible at that point, but it again depends on what the scheduler supports.

[yuhuan417]

bullmq global concurrency: taskforcesh/bullmq#2496

[patrontheo]

@mertalev

The main issue with setting a global limit (or making jobs run sequentially) is that the job scheduler we use, BullMQ, handles concurrency at the queue level. Since the scheduler is in charge of handling concurrency, we don't have a non-hacky way of adding a global limit. This is a feature in BullMQ Pro, so it's also unlikely it will be added to BullMQ in the future.

We intend to switch to a different job scheduler at some point (not for this, but to drop Redis and handle jobs in Postgres). It may be more feasible at that point, but it again depends on what the scheduler supports.

Is there any change to that part since then ?
My server keeps crashing now (maybe since OCR was added, not sure).
I think it would be really useful to be able to set some jobs to be sequential (face detection THEN ocr THEN etc) instead of in parallel. It would lower the load on RAM.
Also global concurrency would help.
Thanks !

[Marco-Z]

Assuming you are running with docker compose, you can limit the resources on a container level. See https://docs.docker.com/compose/compose-file/deploy/#resources
For example on my Raspberry Pi 4 (4GB), I run the immich-microservices with:

deploy:
  resources:
    limits:
      cpus: 2
      memory: 2500mb

You can do the same on the immich-machine-learning (or even offload that to a remote machine when ingesting many files).

[auanasgheps]

Thank you! This is the closest we can get to a solution! I will limit the microservices container to 2 cores as well, since I have 4.

[jamess60]

This is a big help in the sense that it stops the other containers suffering / docker host becoming unstable, but then it does mean immich cant really stretch its legs and utilize the full hardware potential either.

Going to implement this in my setup for now as a fail safe

[yydcool]

+1
I'm using a low-memory machine and I already set concurrency to 1 for every job. But I still see logs showing "Worker (pid:104) was sent SIGKILL! Perhaps out of memory?" from the immich_machine_learning container.

It turns out 3 machine learning jobs [smart search] [face detect] [face recognition] will be running at the same time.

I have to pause these jobs, and manually un-pause them one at a time:

It seems BullMQ made this complicated. Would it make sense to add a CLI feature to get the each job queue status and pause/un-pause each job queue?

With that, I would be able to write our own script to control the global job concurrency.

[yydcool]

wow that's great, thanks for the information!

[yydcool]

I've created a script to set only one job queue to be active, now effectively global concurrency = 1
https://gist.github.com/yydcool/5df3ce4a86e7b1931e794f4e6095d322

[ppskj178]

하나의 작업 대기열만 활성화하도록 설정하는 스크립트를 만들었고, 이제 효과적으로 전역 동시성 = 1이 되었습니다. https://gist.github.com/yydcool/5df3ce4a86e7b1931e794f4e6095d322

How do I apply it?

Apart from this, this feature should be supported by immich itself. For most people, creating a preview is important and AI learning can be done slowly.

[WINVIST]

I've created a script to set only one job queue to be active, now effectively global concurrency = 1 https://gist.github.com/yydcool/5df3ce4a86e7b1931e794f4e6095d322

Hi, saw your script. I want to know if you have made any changes to it? I've had it for more than 24 hours without completing a single task. I can't figure out what's wrong

[DidouJazz]

Hey, here is the one I am using (with good success so far) !

# /usr/local/bin/python3.9 /volume2/docker/Immich/API/main.py

import requests
import json
import time

# Configuration for the API endpoint and headers
url = "http://127.0.0.1:2283/api/jobs"
headers = {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'x-api-key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
}

def fetch_jobs():
    """
    Fetch the current job statuses from the API.

    Returns:
        dict: A dictionary of job information, or an empty dict on error.
    """
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        return response.json()
    except (requests.RequestException, json.JSONDecodeError) as e:
        print(f"Error fetching jobs: {e}")
        return {}

def set_job(job, command):
    """
    Send a command to pause or resume a specific job via the API.

    Args:
        job (str): The job name.
        command (str): The command to send ("pause" or "resume").
    """
    try:
        payload = json.dumps({"command": command, "force": False})
        response = requests.put(f"{url}/{job}", headers=headers, data=payload)
        response.raise_for_status()
        # print(f"Set job {job} to {command}")
    except requests.RequestException as e:
        print(f"Error setting job {job} to {command}: {e}")

def manage_jobs():
    """
    Main logic to manage jobs:
    - Fetches all jobs from the API.
    - Finds the first job in the managed list that has any activity.
    - Unpauses that job and pauses all others in the managed list.
    """
    jobs = fetch_jobs()
    if not jobs:
        return

    # List of jobs to manage in priority order
    managed_job_list = [
        "metadataExtraction",
        "storageTemplateMigration",
        "thumbnailGeneration",
        "smartSearch",
        "duplicateDetection",
        "faceDetection",
        "facialRecognition",
        "videoConversion"
    ]

    # Find the first job with any jobCounts > 0 (active, waiting, paused, or delayed)
    job_to_unpause = None
    for job in managed_job_list:
        job_info = jobs.get(job)
        if job_info and (
            job_info['jobCounts']['active'] > 0 or
            job_info['jobCounts']['waiting'] > 0 or
            job_info['jobCounts']['paused'] > 0 or
            job_info['jobCounts']['delayed'] > 0
        ):
            job_to_unpause = job
            break

    # Unpause the first found job, pause all others in managed_job_list
    for job in managed_job_list:
        if job == job_to_unpause:
            # print(f"Unpausing job: {job}")
            set_job(job, "resume")
        else:
            # print(f"Pausing job: {job}")
            set_job(job, "pause")

if __name__ == "__main__":
    # Run the job manager loop every 10 seconds
    while True:
        manage_jobs()
        time.sleep(10)

[toniglandy1]

here's my contribution to the discussion, hoping that this gets traction ! I'm going a bit beyond global job concurrency and hoping for a job priority queue. :D

Current situation :

• in server settings, you can give each job a number which corresponds to the threads that is allocated to said job to enable parallel execution and better speed at the cost of more resource use
• in the jobs view, you can see which jobs are active, pause some, create some etc.

Issue :
On low powered systems, the concurrency settings is not enough to guarantee proper resource use since importing photos will start (for example) generating thumbnails, smart search, face detection, and facial recognition. Even if each job is marked to have no concurrency, each of these will run, impacting whole system performance.

Proposal :
I'm proposing a priority queue where in the job settings we define :

• whole system job concurrency number
• order jobs from high priority to lower priority
• give a weight to each job, which represents how intensive a single job is relative to others.

for example :
whole system threads : 4
generate thumbnails : priority 0 | 2 concurrent | weight 1
extract metadata : priority 1 | 3 concurrent | weight 1
transcode videos : priority 8 | 1 concurrent | weight 4

with these settings, the server will either :

• generate thumbnails up to 2 at a time & extract metadata at 2 at a time
• if only 1 or no thumbnails are generating, then 3 threads are extracting metadata
• once all jobs are finished, then the video transcoding starts

[alternativniy]

Hello, everyone. I created a Docker container based on a script from @DidouJazz and converted it to shell. You can simply add it to your compose file by specifying all the necessary variables. Instructions are available in the repository.

[bjoe10]

I’ve built a Home Assistant add-on based on immich-job-daemon for fellow Home Assistant OS users like me who run on Raspberry Pi and have limited resources.

https://github.com/[bjoe10/HA_Addon](https://github.com/bjoe10/HA_Addon)

[KureOnGit]

Hey, this is awesome, seems to be saving my system from locking itself entirely while running jobs even when the container is resource limited.
I was wondering if it could support Remote ML, I'm sending those jobs to a different system and would be terrific to set a global concurrency just for those since the other system can handle high throughput.

[KureOnGit]

Adding to my own reply, if it helps others, the limit for a Pi 4 seems to be 3 concurrent Thumbnail jobs (everything else suspended) or 1 Thumbnail job and all others between 4 and 6 concurrent jobs. I am offloading ML to a remote machine though, which is not always on. For these I've set to 20 but resume only when the remote device is on.
I have also given 3 cores and 2500mb RAM to immich_server container.

In other words for day to day I use the container above to set a global limit of 3, for the first intake it's fine to turn it off and have the Pi run everything but the Thumbnails all at once, and vice-versa, pause everything except the thumbnails when you need to get those worked on.

[harmpter]

@bjoe10 wow! this is awesome because I'm looking to run Immich on my RPi 4 2gb.
So, do I understand correctly that your add-on runs alongside of an Immich add-on (I assume either fabio-garavini's or alexbelgium's) and handles only its job queue? And do you mind telling me which Immich add-on you're running on your Pi?
Many thanks all of you people! ❤️

[bjoe10]

I have the same problem — now that OCR is included, it adds about +3 GB of RAM usage and kills my system.
I'm running Immich on a Raspberry Pi 5 with 8 GB RAM.
There should be an option to process these heavy tasks one after another instead of loading everything at once.

[Prolonz]

Agreed, I run on a 2GB Raspberry Pi 4. It crashes nearly every time I upload photos. I know my system is low resources but I'm not convinced it's not sufficient with the right optimization. Once all the processing is done Immich runs perfectly.

Native controls to limit the number of jobs running concurrently would expand the devices capable of running Immich without issue.

Better job customization would help further... I'd like thumbnails and metadata to run only when an upload is finished. The rest of the jobs can run overnight.