Skip to content

Studio 4.9: auto-size the hardware split

Choose a tag to compare

View all tags
@mahmoudyassine mahmoudyassine released this 18 Jul 11:32
v4.9.0
615c7ed

Users should not need to derive tensor parallelism by hand. This release adds it as a one-click suggestion.

Auto-size TP & workers (Hardware station)

  • Picks the smallest TP whose slice group fits one full copy of the selected model at the selected precision (keeping ~20% headroom for KV), preferring the NVLink island.
  • Then adds workers until the peak concurrent calls (from the use-case preset or the Little's-law estimator) are admitted at the current batch.
  • If the suggested TP crosses nodes, interconnect efficiency is set to 0.7 and the toast says so. Everything remains editable afterwards.
  • Scripting: window.GPUscale.autoSize().

Clarity

  • The hardware card now always states the fleet layout in plain language: how many replicas exist, that each replica is a full copy of the model on TP GPUs, and that raising TP distributes one copy wider while adding workers creates more copies for more users.

Example: Kimi K3 2.8T FP8 on B300, 377 concurrent calls. Manual TP8 gives 8 overflowing copies (127% on every GPU). One click: TP16, 32 replicas, fits at 66% with 22.8 ms TTFT.

Assets 2
Loading