gunicorn, torch, and cuda

[keighrim]

Bug Description

As found in clamsproject/app-whisper-wrapper#24 (comment), when a CLAMS app runs in HTTP + production mode (app.py --production) with CUDA device support, it runs over gunicorn wsgi with multiple workers. It seems that some torch-based CLAMS apps running under the scenario spawns multiple python processes and load multiple copies of the torch model to memory, resulting in OOM errors at some points.

Reproduction steps

1. pick a computer with a CUDA device (nvidia gpu).
2. run whisper wrapper v10 (https://apps.clams.ai/whisper-wrapper/v10/) in the production mode.
3. send multiple POST (annotate) requests simultaneously or with short gaps between.
4. watch VRAM saturation via e.g. nvidia-smi or a similar monitoring util.

Expected behavior

The app should reuse the already-loaded checkpoint/model in the memory. Instead, the app loads the model for each request and then doesn't release the model after the process is completed.

Log output

No response

Screenshots

No response

Additional context

Also, it's very likely that this issue shares the same root cause with clamsproject/app-doctr-wrapper#6.

[keighrim]

this comment includes outdated/incorrect information. See updated comment in the below.

---

PR #265 implements a better handling for this issue.

1. For developers of GPU apps

New metadata fields to declare VRAM requirements:

• gpu_mem_min - minimum VRAM needed (MB)
• gpu_mem_typ - typical usage with default parameters (MB)

The SDK now

1. uses gpu_mem_min to automatically calculate optimal gunicorn worker count based on available VRAM, preventing the app from over-spawning workers, then hitting a OOM and die.
2. Theoretically, all annotate() calls should now check for vram availability if 1) cuda device is there 2) torch is installed 3) a positive gpu_mem_min value is declared, before calling _annotate() internally
3. Instead of OOM-then-die, the app should now returns a MMIF with error view (runtime error "insufficient vram, try later") (and code 503 if under HTTP mode) See below on how vram availability is calculated

2. At runtime, automatic VRAM estimation and rejection

When PyTorch and CUDA are available and gpu_mem_min > 0:

• First request uses conservative 80% of total VRAM estimate
• Subsequent requests use historical measurements × 1.2 buffer
• Profiles cached in ~/.cache/clams/memory_profiles/

Also enhanced hwFetch to show available VRAM alongside peak usage.

Documentation and limitations

New documentation/gpu-apps.md covers all GPU memory management features.

Main limitation: profiling and runtime checking are PyTorch-only. TensorFlow apps still get worker calculation from gpu_mem_min, but no automatic profiling due to TF's fundamentally different memory model (grabs all VRAM by default, no equivalent to torch.cuda.empty_cache()). Given our very limited number of TF-based apps, this is acceptable for now.

[keighrim]

this comment includes outdated/incorrect information. See updated comment in the below.

---

Further testing on CUDA/PyTorch behavior suggests we should simplify the VRAM pre-check .

• When VRAM is contested, torch/CUDA blocks the allocation and waits (which was a new finding to me) instead of an immediately OOM. I didn't verify this for tensorflow.
• So actual OOM only occurs if a single request exceeds total VRAM. Concurrent requests queue at first gunicorn level, then the CUDA level.

Plus, the initial VRAM estimation (conservative 80% / historical peak $\times 1.2$ buffer) causes significant capacity waste, can leave significant vram unused while rejecting incoming requests.

Revised VRAM Pre-check: a new runtime param to decide wait-or-die

I propose replacing the availability check with a binary option to provide flexibility and allows users to select between maximum utilization (with unpredictable latency) and predictable latency (with very small capacity waste).

• noWaitGPU (default: false)

This single boolean parameter controls the VRAM pre-check logic for a given request, when off, all incoming request or bash cli.py to be queued, allowing CUDA to handle queuing natively. This is the optimal setting for maximizing VRAM utilization. When on, the sdk checks for remaining memory, and immediately reject a request if there's only <100MB left. If it runs in HTTP mode (app.py) the error view will be returned with 503 code.

gpu_mem_* metadata keys are still there and will be used to compute the worker numbers.

[keighrim]

#266 implements changes to address VRAM stability and capacity (non-)management. Note that the memory behavior observed in the above comment (where I though torch processes was "waiting" for available VRAM vacancy) was due to the NVIDIA HMM feature¹, which uses system RAM as VRAM overflow to prevent immediate OOM errors, at the cost of significant performance slowdown.

App Metadata for GPU Declaration

• Two new fields have been added to the app metadata for declaring VRAM requirements.

VRAM Usage Profiling

• First-time VRAM usage for each unique parameter combination is now recorded and cached locally for bookkeeping and future reference.
• These profiles are stored in the local cache directory under $XDG_CACHE_HOME/clams.

Gunicorn and CUDA

Worker Lifecycle Management

• Worker Discard Policy: All gunicorn workers are now configured to be discarded after processing a single request (max_requests=1).
  • This will prevent VRAM leaks caused by idle (zombie) workers retaining allocated VRAM unnecessarily.
  • But it introduces a model loading time penalty at the start of each new worker process.
  • Developers can override this behavior by setting the max_requests parameter passed to serve_production() in app.py. (Useful for single-model, single-worker scenario)

Worker Count

• The number of gunicorn workers is now automatically set to a reasonably low limit when an app declares GPU usage, based on available VRAM, instead of CPU thread number when no GPU usage is declared.

Footnotes

1. Auto-active under NVIDIA Driver 536.40+ and windows 11 or nvidia 536+ and CUDA 12.2+ and Linux 6.1+ ↩