Directly get max encoder len from VLLM config in V1

[Sugar-zsg]

Improves performance by getting the max encoder length directly from the initialized vllm_config.scheduler_config. This avoids the expensive lookup and re-computation previously done by MULTIMODAL_REGISTRY.get_encdec_max_encoder_len.

Test Results:

• Environment: H20 GPU
• Data: 10s audio
• Before: Average latency was 1300ms.
• After: Average latency is now 305ms.

Purpose

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request aims to improve performance by optimizing how the maximum encoder length is retrieved. However, the current implementation introduces a critical correctness issue by using an incorrect value, which creates an inconsistency between memory allocation for the KV cache and the metadata used in the attention mechanism. This could lead to out-of-bounds memory access. I've provided a detailed comment explaining the issue and recommending a safer approach.

[gemini-code-assist]

This change introduces a critical correctness issue by using a value for max_encoder_len that is inconsistent with how the cross-attention KV cache is allocated. This can lead to out-of-bounds memory access and other runtime errors.

The new implementation uses scheduler_config.max_num_encoder_input_tokens, which is derived from max_num_batched_tokens. This is a general batching configuration (e.g., 16384) and not the model-specific maximum encoder sequence length (e.g., 1500 for Whisper).

Meanwhile, the memory for the cross-attention KV cache is allocated based on the correct, smaller value via MULTIMODAL_REGISTRY.get_encdec_max_encoder_len in CrossAttentionSpec.max_memory_usage_bytes. Using a much larger max_seq_len in the attention metadata here creates a dangerous discrepancy.

While the performance motivation is valid, a safer solution is to compute the correct value once during engine initialization and cache it in the configuration. For now, it's best to revert to the original implementation to avoid memory corruption issues.

def _get_max_encoder_len(vllm_config: "VllmConfig") -> int:
    return MULTIMODAL_REGISTRY.get_encdec_max_encoder_len(
        vllm_config.model_config)

[DarkLight1337]

LGTM, but best for @russellb @heheda12345 to confirm.

[Sugar-zsg]

@russellb @heheda12345 Could you please review this PR when you have time ? Thx !

[heheda12345]

Before: Average latency was 1300ms.
After: Average latency is now 305ms.

The change make sense to me. But the speedup is much larger than I expect from this change. What is the expensive operation that lead to such a large speedup?

[russellb]

Thanks for the PR! You will also need to update your commit message(s) to include the Signed-off-by header to satisfy the DCO check in CI.

[Sugar-zsg]

Before: Average latency was 1300ms.
After: Average latency is now 305ms.

The change make sense to me. But the speedup is much larger than I expect from this change. What is the expensive operation that lead to such a large speedup?

During decoding, each call to _prepare_inputs triggers the cross-attention builder, which in turn executes MULTIMODAL_REGISTRY.get_encdec_max_encoder_len. In my tests, this method takes around 10ms per call, while the decoder’s forward computation itself only takes about 2ms.

[Sugar-zsg]

@heheda12345 @DarkLight1337 These have been resolved. Please check when you have a chance.

[russellb]

Thanks!

I confirmed that this value in the scheduler_config is initialized from the same code from the multimodal registry:

                self.scheduler_config.max_num_encoder_input_tokens = \
                    MULTIMODAL_REGISTRY.get_encdec_max_encoder_len(self.model_config)

[russellb]

I haven't enabled auto-merge yet in case you wanted to look again @heheda12345

[russellb]

related PR: making the same change in some other areas: #24989