Repository navigation
Try same-prompt prefill across two CUDA helper hosts #5215
dgruhin-hrizn
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I have two RTX 5080s on separate Proxmox hosts. The CUDA helper already supports a two-GPU layer pipeline inside one helper host. Could two independently paired CUDA machines contribute layers to the same prompt in a future experiment? CUDA helper’s two-GPU documentation
I’d compare Mac alone, one 5080, and both 5080s using identical uncached 16K and 64K prompts. I’d report first-token time and layers per host. Additional layers could help, and the extra network hop could erase the gain. Auto could choose the single-helper path when it estimates that will finish sooner.
All reactions