Severe image quality discrepancy between singleton and dynamic batched requests with identical seed, steps and size

[hhhFuture]

Hello everyone, I would like to ask for your assistance regarding an unusual issue I have encountered.

When I launch the online inference service with sglang and configure --batching-max-size and --batching-mode dynamic, there is a severe discrepancy between images generated by singleton requests and batched requests. I have cross-checked the parameters for both request types and confirmed that seed, num_inference_steps, size and all other relevant generation parameters are fully consistent.

I referred to the documentation, which notes:

Batch shape can change kernels, so singleton and dynamic outputs are not expected to be bit-exact.

My understanding of this statement is that singleton and batched generation should only produce negligible floating-point differences at the pixel level, while the visual content and composition of the generated images should remain effectively identical.

I have researched extensively to troubleshoot this issue but have not found a solution. I have already conducted thorough testing and shared the full test details in the following issue comment:
#28502 (comment)

I would greatly appreciate any guidance or support from the community. Thank you all very much in advance!

[rehan243]

Nice catch on the doc note about batch shape changing kernels - that's a crucial detail. I've seen similar issues with batched requests in our production environment at Reallytics.ai, where we're processing 11M+ images. The thing is, even with identical seed, num_inference_steps, and size, batched generation can introduce subtle differences due to kernel optimization and memory allocation. Turns out, when you batch requests, the underlying CUDA kernels or tensor operations might not be exactly the same as singleton requests, even if the params are identical. For instance, torch.backends.cudnn.deterministic can affect this. Fwiw, we had to tweak our sglang config to ensure consistency across batch sizes. Honestly, it's worth checking if you're hitting a similar issue with kernel determinism. Are you using sglang with TensorRT or PyTorch backend? Curious to hear if you've tried tweaking the batching config or backend settings to see if that resolves the discrepancy.

[hhhFuture]

您注意到文档中关于批次形状变化内核的注释，这确实是一个关键细节。我在 Reallytics.ai 的生产环境中也遇到过类似的批量请求问题，我们处理的是超过 1100 万张图像。问题在于，即使参数完全相同，seed批量生成也会因为内核优化和内存分配而引入细微差别。事实证明，即使参数相同，批量请求时底层 CUDA 内核或张量操作也可能与单例请求不完全相同。例如，参数可能会影响这一点。顺便一提，我们不得不调整配置​​以确保不同批次大小下的一致性。说实话，您也应该检查一下是否遇到了类似的内核确定性问题。您使用的是TensorRT还是PyTorch后端？我很想知道您是否尝试过调整批量配置或后端设置来解决这个问题。num_inference_steps``sizetorch.backends.cudnn.deterministic``sglang``sglang

Thank you for your suggestion. I tried setting the backend parameters and tried several different methods, but still encountered the situation mentioned earlier. The configured parameters are as follows:

#!/bin/bash
sglang serve \
   --model-path /root/autodl-fs/data/z-image_turbo/Tongyi-MAI/Z-Image-Turbo \
   --host 0.0.0.0 \
   --port 30000 \
   --attention-backend torch_sdpa \
   --component-attention-backends text_encoder=torch_sdpa,transformer=fa \
   --batching-mode dynamic \
   --batching-max-size 5 \
   --batching-delay-ms 50 \
   --enable-batching-metrics \
   2>&1 | tee sglang_torch_text_encoder.log

[hhhFuture]

@rehan243 Thank you for your reply.

The issue with the generated images goes far beyond minor numerical differences — the outputs no longer match the prompt description at all, suffering from either missing main subjects or incorrectly rendered characters.

In my latest update at #28502 (comment), I conducted further tests. While keeping seed, size, num_inference_steps , same identical prompt, I only modified the value of --batching-max-size. I found a clear pattern: as the value of --batching-max-size increases, the generated images deviate more severely from the prompt, becoming progressively more prone to missing subjects and rendering errors.

The shell script I use to start the service is shown below:

#!/bin/bash
sglang serve \
--model-path /root/autodl-fs/data/z-image_turbo/Tongyi-MAI/Z-Image-Turbo \
--host 0.0.0.0 \
--port 30000 \
--batching-mode dynamic \
--batching-max-size 5 \
--batching-delay-ms 50 \
--enable-batching-metrics \
2>&1 | tee sglang_best.log

After starting the service, I send requests to it via aiohttp and asyncio in Python, which should be running on the PyTorch backend.I haven't tried adjusting the backend parameters yet, but I will try it today. Thank you for your suggestion.

I currently do not know how to resolve this issue, and would sincerely appreciate any help from the community. Thank you again!