Why do PDMux Prefill and Decode kernels show no temporal overlap?

[ricklessboy404]

Environment

• GPU: 3080Ti
• Model: Qwen3-1.7B
• CUDA Version: 12.8
• Pytorch Version: 2.9.1
• SGLang Version: 0.5.9

Command

Launch a pdmux server:

nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o pdmux.out --delay 30 \
python3 -m sglang.launch_server \
--model /root/autodl-tmp/models/qwen3_1.7B \
--tp 1 \
--disable-overlap-schedule \
--port 30000 \
--mem-fraction-static 0.75 \
--chunked-prefill-size -1 \
--enable-pdmux \
--pdmux-config-path pdmux_config.yml

Benchmark:

python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 30000 --num-prompts 50 --request-rate 5

pdmux_config.yml:

# pdmux_config.yaml

# Number of SM groups to divide the GPU into.
# This includes two default groups:
#   - Group 0: all SMs for prefill
#   - Last group: all SMs for decode
# So the number of manual divisions (below) must be (sm_group_num - 2).
sm_group_num: 4

# Optional manual divisions of SMs.
# Each entry contains:
#   - prefill_sm: number of SMs allocated for prefill
#   - decode_sm: number of SMs allocated for decode
#   - decode_bs_threshold: minimum decode batch size at which this group should be selected
#
# If provided, the number of entries must equal (sm_group_num - 2).
# manual_divisions:
#   - [72, 8, 1]
#   - [104, 28, 5]
#   - [96, 36, 10]
#   - [80, 52, 15]
#   - [64, 68, 20]
#   - [56, 76, 25]


# Divisor for default stream index calculation.
# Used to adjust sm_group based on decode_bs when no manual_divisions are provided.
# Formula:
#   stream_idx = max(1, min(sm_group_num - 2, decode_bs * (sm_group_num - 2) // decode_bs_divisor))
decode_bs_divisor: 36

# Maximum token budget for split_forward in the prefill stage.
# Determines how many layers are executed per split_forward.
# Formula:
#   forward_count = max(1, forward_token_budget // extend_num_tokens)
split_forward_token_budget: 16384

Phenomenon

I compared performance with and without --enable-pdmux and found that TTFT increases significantly when PDMux is turned on. To diagnose this, I profiled the PDMux-enabled configuration using nsys; the results are shown below.

According to the nsys profile, Green Context 3 is assigned to Prefill and Green Context 4 to Decode. After the initial large-batch Prefill wave finishes, the two contexts appear to execute in a strictly alternating manner with almost no temporal overlap.

Question

Is this expected, or is there a known bottleneck (e.g., launch latency, stream synchronization, or SM scheduling) that prevents them from running concurrently?

[pm32900]

I think the behavior you're seeing can be expected, and --disable-overlap-schedule is probably not the direct reason for the lack of Prefill/Decode kernel overlap.

--disable-overlap-schedule controls SGLang's overlap between the CPU scheduler and the GPU model worker; it is different from the GPU-side concurrency that PDMux is trying to achieve with Green Contexts.

With PDMux, Prefill and Decode can be assigned different Green Contexts / SM partitions, but that does not guarantee that Nsight will show continuous simultaneous kernel execution. The contexts make concurrent execution possible by restricting the SM resources available to each workload, but actual overlap still depends on whether both sides have runnable work at the same time and on synchronization/dependencies between the split forwards.

In your configuration, I would specifically look at two things.

First, you have:

split_forward_token_budget: 16384

PDMux splits the prefill forward pass according to this budget. If those split-forward regions are relatively coarse for Qwen3-1.7B on a 3080 Ti, Decode may only get scheduling opportunities around the boundaries between Prefill chunks. In Nsight this can look much more like alternating Prefill/Decode execution than continuous kernel-level overlap.

Second, the selected SM partition depends on decode_bs. With:

sm_group_num: 4
decode_bs_divisor: 36

and no manual_divisions, the selected group follows the decode-batch-size-based mapping. Your workload (request-rate 5, 50 prompts) may therefore spend much of its time in a partition that is not particularly favorable for this GPU/model combination.

That can also explain why TTFT becomes worse: PDMux deliberately gives some GPU resources/scheduling opportunities to Decode while Prefill is active. On a relatively small model such as Qwen3-1.7B, the benefit from protecting Decode may not compensate for the reduced Prefill throughput and additional split-forward/scheduling overhead.

I would try three comparisons:

1. Profile without --disable-overlap-schedule. This should improve CPU/GPU scheduling overlap, although I would not expect it by itself to guarantee Prefill/Decode kernel overlap.

2. Reduce split_forward_token_budget and check whether Nsight shows finer-grained interleaving/overlap. If the current 16384 budget is producing long Prefill regions, a smaller value should provide more opportunities for Decode to run.

3. Try explicit manual_divisions for the 3080 Ti rather than relying on the default decode_bs_divisor mapping, and compare both TTFT and decode latency. The optimal split is workload- and GPU-dependent.

So I wouldn't interpret the trace as evidence that Green Contexts themselves are failing to provide isolation/concurrency. PDMux provides the resource partitioning needed for Prefill and Decode to coexist, but it doesn't imply that every Prefill kernel should visibly overlap a Decode kernel. The amount of actual overlap is determined by the workload, split-forward granularity, selected SM partition, and synchronization points.

The increased TTFT is therefore also plausible: you're trading some Prefill performance for better Decode isolation/concurrency, and that tradeoff can be unfavorable for this particular Qwen3-1.7B + 3080 Ti workload.