Repository navigation
AMD GPU fractions with a hard VRAM cap (dmem cgroup): implementation and lab results #4103
manmohanbrahma
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hello Run:ai team,
This is for your information. We have a working implementation of GPU fractions on AMD GPUs with a hard memory cap, and we would like your view on how Run:ai should show it.
What it does
fractiondfrom kai-scheduler/gpu-fractioning, with a new AMD backend (--gpu-vendor=amd), caps each pod's VRAM to its share. It uses the Linuxdmemcgroup controller (Linux 6.14+,amdgpu). This is the AMD counterpart ofNVIDIA_GPU_MEMORY_LIMIT, and the kernel enforces it.docs/amd/README.md).Lab result (Run:ai 3.0.20 with KAI v0.18.3 and the patches above, Radeon PRO W7900, kernel 7.0, project
team-c, node poolamd):Where this touches Run:ai
fractiondrunning on AMD nodes, that status could reflect the AMD path. Related: [bug] GPU nodes counted as not ready when GPU fractioning has no target nodes #3941 (fractions shown unavailable when the daemons have no node).fractiondthe same share becomes a hard limit.Happy to share more detail or open issues for the Run:ai side once KAI settles the design.
All reactions