[Bug]: Fix MRL Support Detection for Qwen3-Embedding-8B Model (It Supports MRL per Latest Official Docs)

[Cyberceratops]

Your current environment

Unfortunately, I cannot run the Python script to collect the environment due to certain security considerations and restrictions within the organization.But I can briefly describe the key environmental information.

🐛 Describe the bug

Environment

• vLLM Version: 0.8.5
• Hardware: Unknown
• Python Version: Unknown (Torch 2.6.0)
• Model: Qwen3-Embedding-8B

Description

I'm using the Qwen3-Embedding-8B model deployed via vLLM in a platform (Cherry Studio). When making embedding calls with the "dimensions" parameter included in the request (e.g., {"model": "Qwen3-Embedding-8B", "input": "hi", "dimensions": 1024}), vLLM returns a 400 Bad Request error stating that the model does not support Matryoshka Representation Learning (MRL).

However, according to the latest official technical documentation from Alibaba (released in June 2025), Qwen3-Embedding-8B explicitly supports MRL, allowing dynamic dimension adjustment without quality loss. This indicates a bug in vLLM's model detection logic, where it incorrectly assumes the model lacks MRL support.

This is problematic because:

• In my case, the platform (Cherry Studio) automatically includes "dimensions" in backend requests for optimization, leading to failures in knowledge base embedding and search functionalities.
• The platform developers confirmed: "Other providers ignore extra parameters without issues, but vLLM errors out." They also noted this was fixed before but regressed in a recent change.

Error message from logs:
[KnowledgeQueue] Backend processing error for item ...: 400 Model "Qwen3-Embedding-8B" does not support matryoshka representation, changing output dimensions will lead to poor results.

A simple health check without "dimensions" succeeds (200 OK), but actual workflows fail due to the forced parameter.

Steps to Reproduce

1. Deploy Qwen3-Embedding-8B using vLLM 0.8.5.

2. Send a POST request to the embeddings endpoint with "dimensions" included, e.g.:
   {
   "model": "Qwen3-Embedding-8B",
   "input": "hi",
   "dimensions": 1024 // or any value
   }

3. Observe the 400 error.

Expected Behavior

• vLLM should correctly detect and support MRL for Qwen3-Embedding-8B (as per the June 2025 official docs), allowing dynamic "dimensions" without errors.

Actual Behavior

• vLLM incorrectly enforces lack of MRL support and returns a 400 error when "dimensions" is present, even though the model supports it.

Additional Context

This issue was reported to the Cherry Studio platform, and their developer suggested raising it here: "I suggest you go to vLLM to file an issue, requesting support for Qwen3 embedding's MRL. We'll also think of ways to avoid always passing dimensions."

Any guidance or fixes would be greatly appreciated, as this blocks embedding workflows for Qwen models in integrated platforms.

Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

Manually enable Matryoshka Embeddings

For models that support Matryoshka Embeddings but not recognized by vLLM, please manually override the config using hf_overrides={"is_matryoshka": True}, hf_overrides={"matryoshka_dimensions": []} (offline) or --hf_overrides '{"is_matryoshka": true}', --hf_overrides '{"matryoshka_dimensions": []}'(online).
Here is an example to serve a model with Matryoshka Embeddings enabled.

vllm serve Snowflake/snowflake-arctic-embed-m-v1.5 --hf_overrides '{"matryoshka_dimensions":[256]}'

[oneraghavan]

@noooop Is there a plan to support MRL directly ?

[noooop]

This requires modifying in the official repository, for example

https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5/discussions/8

hf_overrides is Manually adding matryoshka-related parameters to the official config

[xsank]

@Triceratops2017 is there any Qwen3-Embedding-8B official recommend matryoshka dims?

[sirfz]

It seems like using hf_overrides={"is_matryoshka": True} does not yield the correct behavior. If I understand correctly, vllm would apply some kind of pooling (average maybe?) to produce the truncated embeddings while the correct behavior should be simply slicing the output (i.e. embeddings[:128] for 128 dim), please correct me if I'm wrong.

I've tested this with Qwen/Qwen3-Embedding-0.6B using SentenceTransformers(..., truncate_dim=128) vs vllm with model.embed(..., pooling_params=PoolingParams(dimensions=128)) (with hf_overrides={"is_matryoshka": True} of course) but the output of vllm is incorrect. Simply using vllm without matryoshka pooling and just slicing [:128] gives correct outputs.

[noooop]

You can disable ‘normal’ by setting PoolingParams(normalize =False).

Or use override_pooler_config to disable ‘normal’.

refor to #20538

[sirfz]

The result is still incorrect even with normalization disabled. Here's an example:

from vllm import LLM, PoolingConfig

MODEL_NAME = "Qwen/Qwen3-Embedding-0.6B"

model = LLM(MODEL_NAME, runner="pooling", hf_overrides={"is_matryoshka": True})
wrong_1 = model.encode(["hello"], pooling_task="embed", pooling_params=PoolingParams(normalize=False, dimensions=128))[0].outputs.data.float().numpy()
# array([-1.6796875 , -2.09375   , -1.1953125 , -3.9375    ,  0.24023438,
#        -5.09375   , -4.28125   ,  4.125     , -5.6875    ,  0.2734375 , ...])
wrong_2 = model.encode(["hello"], pooling_task="embed", pooling_params=PoolingParams(normalize=True, dimensions=128))[0].outputs.data.float().numpy()
# array([-0.03881836, -0.04858398, -0.02770996, -0.09130859,  0.0055542 ,
#        -0.11816406, -0.09912109,  0.09570312, -0.13183594,  0.00634766, ...])
correct = model.encode(["hello"], pooling_task="embed")[0].outputs.data[:128].float().numpy()
# array([-0.01794434, -0.02233887, -0.01275635, -0.04199219,  0.00256348,
#        -0.05419922, -0.0456543 ,  0.04394531, -0.06054688,  0.00291443, ...])

# reference
MODEL_PARAMS = {
    "model_kwargs": {"attn_implementation": "flash_attention_2", "device_map": "auto", "torch_dtype": "bfloat16"},
    "tokenizer_kwargs": {"padding_side": "left"},
    "truncate_dim": 128,
}

model = SentenceTransformer(MODEL_NAME, **MODEL_PARAMS)
model.encode(["hello"])
# array([-0.01733398, -0.02197266, -0.01263428, -0.04052734,  0.00216675,
#        -0.05322266, -0.04516602,  0.04492188, -0.06054688,  0.00379944, ...])

[noooop]

import numpy as np

from vllm import LLM, PoolingParams

MODEL_NAME = "Qwen/Qwen3-Embedding-0.6B"

model = LLM(MODEL_NAME, runner="pooling", hf_overrides={"is_matryoshka": True}, enforce_eager=True)
out = model.encode(["hello"], pooling_task="embed", pooling_params=PoolingParams(normalize=False))[0].outputs.data.float().numpy()
print(out[:8])

norm = out / np.linalg.norm(out)

print(norm[:8])

"""
out: [-1.6875     -2.03125    -1.1875     -4.03125     0.22949219 -4.90625
 -4.25        4.21875   ]
norm: [-0.01795194 -0.02160882 -0.01263285 -0.0428852   0.00244138 -0.05219361
 -0.0452123   0.04487986]
"""

• SentenceTransformer first normalize then do truncate_dim, the norm of the output vector is not 1.
• vllm first do truncate_dim then normalize, the norm of the output vector is 1.

vllm/vllm/model_executor/layers/pooler.py

Lines 468 to 497 in fecbb7c

	# for matryoshka representation
	dimensions_list = [
	pooling_param.dimensions for pooling_param in pooling_params
	]
	if any(d is not None for d in dimensions_list):
	# change the output dimension
	assert len(pooled_data) == len(dimensions_list)
	if len(set(dimensions_list)) == 1 and not isinstance(
	pooled_data, list):
	# if all dimensions are the same
	d = dimensions_list[0]
	pooled_data = pooled_data[..., :d]
	else:
	pooled_data = [
	vecs if d is None else vecs[..., :d]
	for vecs, d in zip(pooled_data, dimensions_list)
	]
	# for normalize
	flags = [p.normalize for p in pooling_params]
	if len(set(flags)) == 1:
	if flags[0]:
	pooled_data = self.activation(pooled_data)
	else:
	pooled_data = [
	self.activation(vecs) if f else vecs
	for vecs, f in zip(pooled_data, flags)
	]

[sirfz]

I understand, so the solution is to have an option where we can have it normalize before truncation otherwise the only solution right now is to keep normalization enabled and just truncate manually.

[CyrilShch]

any update on the fix?

[sirfz]

@CyrilShch for what it's worth, I believe the vllm behavior is the correct one vs SentenceTransformers.

From the HF blogpost about Matryoshka, this comment from the author quote my eye (emphasis mine):

Good question! Embeddings should be (re-)normalized after the Matryoshka truncation. If you only normalize before truncating, the truncated section won't exactly have the expected mean and standard error, but one that's very slightly off.

Indeed, I've ran a few tests and re-normalizing after truncation is the right way to go for best results.

This means that simply using PoolingParams(dimensions=128) (for example) in vllm yield the correct results (truncates and re-normalizes).

Edit: just realized you might be asking about the original issue in this thread, but I guess worth putting this info out there anyway.

[CyrilShch]

@noooop was it fixed? I still get 400 providing dimensions