Parallel Running

[oldhand7]

AFAIK, This works only for one user at the same time. Would be great if it supports parallel running at the seveal web tabs or several users. Thanks.

[mashb1t]

this is the continued discussion which has started in #292

@oldhand7 you'll have to change function ldm_patched/modules/model_management.py:87 to return not current_device, but map the devices available in torch.cuda.device_count() / torch.cuda.device(i) to the processes of workers spawned in the last line of modules/async_worker.py (threading.Thread(target=worker, daemon=True).start()) and make it GPU count agnostic as well as test all occurrences of get_torch_device() + check loading to GPU and offloading to memory/swap:

[mashb1t]

plus keep VRAM and RAM limits in mind, prevent offloading on overflow (while remembering to also refactor the whole memory optimisations to not offload when the one GPU hasn't finished)
and a few more things.
Do you see yourself in the position to judge the necessary changes and implement a PR for them?

[oldhand7]

Thanks for kind response. I have to take a look at it. I have no idea of the codebase. Some documentation could be my help. Anyway, I ill look into it. I have VPS server that is powerful so no worries about the GPU and CUDA. Btw, could you suggest the BEST plaftform for the best performance? I will test your advice soon. Thanks. Feel free to let me know your any suggestion. Thanks.

[mashb1t]

Regarding performance, i mean, the sky is the limit and H100 GPUs offer the max performance i assume, but they're almost nowhere available and very expensive. https://learn.microsoft.com/en-us/azure/virtual-machines/nd-h100-v5-series (if you need 1900GB memory)

An A100 cluster of 8 GPUs could also be very competitive, but also pretty expensive and too high scaled for Fooocus. https://learn.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series

[mashb1t]

if you mean OS as platform i'd go for Windows or Linux using Nvidia (not AMD)

[oldhand7]

this is the continued discussion which has started in #292

@oldhand7 you'll have to change function ldm_patched/modules/model_management.py:87 to return not current_device, but map the devices available in torch.cuda.device_count() / torch.cuda.device(i) to the processes of workers spawned in the last line of modules/async_worker.py (threading.Thread(target=worker, daemon=True).start()) and make it GPU count agnostic as well as test all occurrences of get_torch_device() + check loading to GPU and offloading to memory/swap:

So only one function and its references should be changed? I guess async_worker.py also needs structural changes. Thanks.

[mashb1t]

Frankly speaking: if you didn't get the changes mentioned in my last message, then this might be too big of a feature to start contributing in this project.
You'll have to change much more than only one line, especially workers and isolation that the currently loaded model on a GPU isn't unloaded from the GPU when the another one has finished text encoding / rendering or loading / etc.
You can find the memory optimisation here:

Fooocus/ldm_patched/modules/model_management.py

Line 354 in 1c999be

def free_memory(memory_required, device, keep_loaded=[]):

One general thing to clarify: this is probably not about more parallelized Gradio tasks with parallelism, this is about parallelized workers available for the job queue in async_workers working on tasks, as you can't lock GPUs to Gradio tasks. There most likely has to be a hard limit of one GPU per job or Fooocus has to spawn a worker process for each GPU available.

[oldhand7]

I would like to know the eta of this functionality to you kind of expert , how long it would take me to do it? And most important, The general step of improvement so that I can follow for safe development in detail, including test. Looking forward to hearing back from you. Thanks.

[mashb1t]

The estimate is just a gut feeling, not planned yet.

#292 (comment)

• There is no eta, the feature isn't planned.
• I don't know your level of python familiarity or in general how fast you can learn the codebase structure. Did you already have a look yourself?
• There are no established steps for implementation and/or tests, just the above ideas. In general, i'd propose to start with spawning a worker for each GPU, then increase parallelism, adjust thr code to stick to the selected GPU during all model loading/caching/unloading events as well as test generation and resource allocation, especially offloading and different GPU types.

[mashb1t]

@oldhand7 i assume that you'll not be working on this feature anytime soon as #2302 now exists, right?

[mistaox]

I know this isn't exactly on topic but here is a PS script I use to launch fooocus and use multiple GPUs

I have server with 6 old Tesla p-100s.
This script launches multiple instances of focus and assigns each one to a dedicated port and GPU. It also creates a corresponding output folder for each instance.

This allows me to assign each instance to a separate task or to other users.
Just drop the Run.ps1 file into the root directory with the .bat files then Right-click>Run With Powershell.

Feel free to examine the script before running it.

Run.ps1.txt

[ChineseNorris]

Any update on the matter? This would improve a lot efficiency with multiple GPU

[mashb1t]

@ChineseNorris No. It would, but would also require quite a bit of effort.
There still is no eta, the feature is in the backlog but isn't planned. It's currently easier to boot up another instance of Foocus setting the --gpu-device-id <id> flag.

[ChineseNorris]

Maybe if you provide a TO-DO list would be easier to implement? I would be happy to help on this matter

[mashb1t]

The whole issue is that currently one worker processes multiple images and yields results for one thread to the gradio UI process (queued). I assume that we would need a multi layer approach on rendering:

• 1 gradio queue for UI thread
• 1 job for async worker to correctly yield results
• X sub-processes spawned one for each GPU, listening on image queue level

This would make the whole generation process most flexible and highly performant. The thing is: Fooocus is by design optimized on resource efficiency, not performance (otherwise LoRAs would already need to be baked into the model for max. efficiency, saves 400-600ms, see #2311).
To keep efficiency high, each GPU would need to have its own model loaded for fast access times, as well as optimally also the LoRAs needed for the render as they also have to be in memory. If even one process uses swap, the whole benefit would be mostly lost.
Fooocus currently offloads all non-required models which is has globally loaded if it detects lack of VRAM/RAM and when a job is finished, but offloading is prevented by arguments. This whole global memory management system has to be refactored and rewritten in order to guarantee not accidentally offloading a model for jobs. This would lead to less compatibilty with third party dependencies such as ldm_patched, and considering SD3 is around the corner this is key. BTW i've already tried to update external dependencies, but am struggling with sigma calculation, see #3084 and #2773 (reply in thread)).
EDIT we also have to think about how to handle multiple GPUs with different specs, such as a 1080 and a 4070 in one system or similar. A1111 uses the most powerful card as reference afaik and if there are more than one with the same specs of this in the system only then parallelizes rendering.

TL;DR: tech concept has to be made how to do this first, incl. checklist. Priority is on SD3 and/or other features for now.

I'm open to suggestions of any kind, code references see first few comments in this issue.
@ChineseNorris It would be great if you could make up your mind how to potentially overcome the mentioned issues. Happy to hear your feedback.

[mashb1t]

also see #2302 for more performance insights

[ChineseNorris]

Thanks for the feedback, looks more complicated than it looks, will dig further into this when I have more time