fix: don't hold VRAM in an idle server (CUDA context at startup)

[lstein]

Summary

Fixes #9413 — since #9263, a freshly started server allocated a CUDA context (~128–256 MiB of VRAM per GPU, size varies by GPU/driver) at startup and held it while idle, before any generation was requested.

Root cause

Empirical probing (checking nvidia-smi compute-apps after each call, not just torch.cuda.is_initialized()) narrowed the VRAM holder down:

• torch.cuda.mem_get_info() creates a VRAM-holding CUDA context (it must, to report free memory).
• torch.cuda.set_device() triggers cuInit and, per CUDA 12 semantics, may eagerly initialize on some drivers.
• get_device_properties() / get_device_name() / current_device() hold no VRAM (device property queries don't need a context).

Two startup paths hit the problematic calls:

1. ModelCache.__init__ sizes the RAM cache via _calc_ram_available_to_model_cache(), which called torch.cuda.mem_get_info() to read total VRAM. This path predates feat: multi-GPU parallel session execution #9263, but feat: multi-GPU parallel session execution #9263 amplified it: build_model_manager now constructs one ModelCache per generation device at startup, creating a context on every GPU. (It was also skipped entirely when the legacy ram: / max_cache_ram_gb setting was set — which is why the reporter's older images sat at 0 MiB.)
2. Session-processor workers (new in feat: multi-GPU parallel session execution #9263) called torch.cuda.set_device() at thread start, i.e. at server boot, on every configured generation device.

Changes

• ModelCache._calc_ram_available_to_model_cache() reads total VRAM from torch.cuda.get_device_properties(...).total_memory instead of mem_get_info(). Verified byte-identical return value, no context created. The runtime mem_get_info() in _get_vram_available() (which genuinely needs free memory during model loads) is untouched.
• Worker threads defer torch.cuda.set_device() until they claim their first queue item. The pin is per-thread and the thread persists, so pinning once before the first item is equivalent to pinning at thread start. The TorchDevice.set_session_device() threadlocal pin — which drives all device-selection logic (choose_torch_device, model loader, nodes) — still happens at thread start, unchanged.
• One test constructed a ModelCache pinned to a nonexistent cuda:1 behind a mem_get_info patch; it now also patches get_device_properties.

Verification

• Live server (fresh root, this branch): absent from nvidia-smi --query-compute-apps after 20+ s idle — while a pre-fix server on the same machine held 128 MiB as a control.
• Enqueued a graph against the fixed server: completed successfully (1 completed, 0 failed), confirming the lazily-pinned worker loop functions.
• All 295 model-cache + session-processor tests pass; ruff clean.

🤖 Generated with Claude Code

[JPPhoto]

Approved. I added some tests to this branch to guarantee behavior and everything is green:

• invokeai/backend/model_manager/load/model_cache/model_cache.py:1166: No regression test proves startup uses get_device_properties() and avoids mem_get_info(); modified existing test mocks both, so restoring old VRAM-holding call still passes. Test: construct CUDA ModelCache, assert get_device_properties(cuda:1) called and mem_get_info() not called during initialization.

• invokeai/app/services/session_processor/session_processor_default.py:676: Lazy CUDA pinning has no automated coverage; future movement before dequeue could silently restore idle VRAM allocation. Test: run CUDA worker against empty queue, then two claimed items; assert set_device() is absent while empty, occurs before first runner call, and occurs exactly once.