Repository navigation
Support discrete CUDA VRAM monitoring for mixed clusters #5214
dgruhin-hrizn
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I tried a pipeline cluster with oMLX 0.8.0rc1 on an M5 Max and a GeForce RTX 5080 passed through to Ubuntu on Proxmox. The worker enrolled and passed a CUDA compute check. The planner placed 20 of Qwen3.8 27B oQ4e’s 64 layers on the 5080 with Lightning MTP off.
Preflight blocked activation with
cuda_memory_guard_unavailable, which reports that discrete CUDA VRAM can’t be monitored during model loading. The same gate is present on current main. The fabric probe also reported thatibv_deviceswas missing on the VM. The cluster stopped before inference. Preflight sourceWould per-device VRAM measurements before and during rank loading fit the existing preflight and rollback design? I’d be happy to test on a 16 GiB GeForce worker and share whether it passes loading and readiness and how first-token and decode speed compare.
All reactions