fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate

[SuperMarioYL]

fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate

Motivation

Passing an empty/falsy prompt ('', [], {}, ()) to the offline SDK path
(AsyncEngine.generate / pipe.stream_infer) does not raise a clear error. The
prompt is not None, so it slips past the messages is not None XOR guard in
AsyncEngine.generate (async_engine.py:501). The falsy messages then takes the
input_ids else-branch, leaving input_ids at its default None, and the request
later crashes in len(input_ids) with a confusing:

TypeError: object of type 'NoneType' has no len()

at async_engine.py:566 (and the same shape at :424). The HTTP path already
filters empty messages (api_server.py:443-445), so this only affects users of the
offline SDK / pipeline API who pass malformed input — they get an opaque
TypeError deep inside the engine instead of an actionable error at the boundary.

A regression test (tests/test_lmdeploy/serve/test_empty_prompt_guard.py) reproduces
the real generate() path (no model/GPU required) and is red on main (raises
TypeError) / green on this branch (raises a clear ValueError).

Why guard format_prompts + generate() rather than a one-line assert in
pipeline.stream_infer? MultimodalProcessor.format_prompts is the single
pipeline-layer chokepoint that every offline caller (stream_infer/infer/
__call__/chat via _request_generator, see pipeline.py:123,170,343-378)
passes through, so guarding it once fixes the whole offline path. generate() is
the direct-SDK backstop reachable without going through pipeline at all (the
reproduction exercises it directly). A guard only in stream_infer would miss
direct generate() callers and would be strictly redundant with the
format_prompts guard — hence the two non-redundant layers, no third.

Modification

• lmdeploy/serve/processors/multimodal.py: add a shared
  MultimodalProcessor.validate_prompt @staticmethod that rejects None and the
  empty str/list/tuple/dict shapes with a clear ValueError; call it at the top of
  format_prompts (root-cause boundary). A (prompt, image) multimodal pair has
  len == 2 and is intentionally not rejected.
• lmdeploy/serve/core/async_engine.py: call validate_prompt in generate()
  immediately after the existing XOR guard, covering both the messages and
  input_ids falsy shapes (direct-SDK backstop).
• tests/test_lmdeploy/serve/test_empty_prompt_guard.py: new regression test
  (red-on-main / green-on-branch) exercising the real generate() path via
  AsyncEngine.__new__ + minimal mocks (no model/GPU), plus pure-function
  format_prompts('' / [] / None) assertions.

Scope is intentionally offline-SDK only: the HTTP (serve/openai,
serve/anthropic), TurboMind C++, PyTorch engine, pipeline.py, and the reward/ppl
paths are untouched.

BC-breaking (Optional)

No. The change only converts a previously-crashing input (TypeError deep inside
the engine) into an early, clear ValueError at the input boundary. All valid
(non-empty) input flows through unchanged. Downstream projects passing valid prompts
see no behavioral change.

Use cases (Optional)

Offline SDK users who accidentally pass an empty prompt (pipe(''),
engine.generate(messages=''), an empty batch) now get an actionable
ValueError: ... at the boundary instead of an opaque TypeError: object of type 'NoneType' has no len() from inside the engine.

Checklist

1. Lint: ruff check on all changed files — all checks passed (line-length 120,
   E/F/I/W/UP, py310), per .pre-commit-config.yaml.
2. Tests: python -m pytest tests/test_lmdeploy/serve/test_empty_prompt_guard.py -q → 7 passed (green on branch; 7 failed on main); python -m pytest tests/test_lmdeploy/serve/test_session_cleanup.py -q → 9 passed (regression for
   the touched generate() path, no regressions).
3. No new/downstream dependency introduced.
4. No documentation change needed (no user-facing API surface added — only a clearer
   error for previously-crashing invalid input).

[ErenAta16]

The premise checks out — the XOR guard tests identity, so any falsy-but-not-None value walks straight through it:

messages='x'  input_ids=None  -> raise=False   (intended)
messages=None input_ids=[1]   -> raise=False   (intended)
messages='x'  input_ids=[1]   -> raise=True
messages=None input_ids=None  -> raise=True
messages=''   input_ids=None  -> raise=False   <-- empty string passes
messages=[]   input_ids=None  -> raise=False   <-- empty list passes

so generate("") reaches the body with input_ids still None, which is what produces the len(None) TypeError rather than a message naming the actual problem. Worth stating in the PR body, because the guard reads correct at a glance and the reason it isn't is the is not None rather than the XOR.

Two things I'd want settled.

format_prompts now rejects an empty list, and that's a separate behaviour change from the generate guard. Grepping the tree, format_prompts has no caller outside its own definition, so nothing in-repo breaks today — but it's a public @staticmethod on MultimodalProcessor, so external callers passing [] to mean "no prompts in this batch" would start getting a ValueError where they previously got [] back. If that's intended, the PR body should say so; if the goal is only to fix generate, the format_prompts change could be dropped and the guard would still do its job.

validate_prompt enumerates types rather than testing falsiness, which is the right call but for a reason the code doesn't state: a bare if not prompt would also reject 0. input_ids=[0] is a legitimate single-token prompt and not [0] is False, so that's fine, but validate_prompt is also called with name='input_ids' where a future caller might pass a bare int. The explicit per-type checks avoid that class of surprise — worth a one-line comment saying so, otherwise someone will simplify it back to if not prompt later.

Smaller: the test module docstring references "P8-002" and "RED on main, GREEN on branch", which read like internal tracker notes rather than something for this repo's history. Also _make_engine stubs _determine_gen_config to reproduce the crash, so the test is pinning the guard rather than the real call path — that's a reasonable trade given the alternative needs a loaded model, but the docstring should say the stub is what makes it fail on main, not the engine itself.

[ErenAta16]

Read this against main. The gap is real — '' and [] do slip past the XOR guard at async_engine.py:500 because neither is None — and centralising the check is the right shape. Three things I'd want settled before it lands, in decreasing order of importance.

1. The nested empty case, which is the one the motivation describes, still gets through.

validate_prompt is applied per element, and only at the top level:

for prompt in prompts:
    MultimodalProcessor.validate_prompt(prompt, name='prompt')

A (prompt, image) pair is a non-empty tuple, so it passes — including when the prompt inside it is empty:

format_prompts([('', image)])

The tuple has length 2, _is_str_images_pair accepts it (isinstance(_1, str) is true for ''), and it goes on to _re_format_prompt_images_pair with an empty prompt. Same for openai form: [{'role': 'user', 'content': ''}] is a non-empty dict, so it passes validation and _is_openai_message returns it unchanged.

So the guard catches format_prompts('') and format_prompts([]) but not the two structured forms, which are exactly where an empty prompt is most likely to arrive from a caller assembling input programmatically. Worth either recursing into the pair/message shapes or being explicit in the docstring that only the scalar forms are covered.

2. A batch with one empty string now fails as a whole.

format_prompts(['a', '', 'b'])

returned ['a', '', 'b'] before and now raises. That may well be what you want, but it's a behaviour change for batch inference rather than a pure error-message improvement: a caller batching user rows previously got one empty generation and now loses the other two. Worth calling out in the PR description so whoever merges it makes that call knowingly.

3. validate_prompt is a no-op for the other falsy values the title mentions.

0, False, 0.0 aren't None, str, list, tuple or dict, so they pass. They do get rejected further down by the Unsupported prompts: branch, so nothing is broken — but the title says "falsy" and the implementation is "empty container or empty string", and those aren't the same set. Either the type list wants an else for the remaining scalars, or the wording wants narrowing.

Smaller notes:

• The four isinstance branches all reduce to the same thing. if isinstance(prompt, (str, list, tuple, dict)) and not prompt with the type name interpolated from type(prompt).__name__ would give the same messages in a quarter of the lines, and would automatically cover set and bytes.
• MultimodalProcessor.validate_prompt(input_ids, name='input_ids') reads oddly given the method name — input_ids is a token list, not a prompt. Not worth blocking on, but a neutral name like reject_empty_input would fit both call sites.

None of this is an argument against the change; the underlying complaint is legitimate and the offline SDK path should not accept an empty prompt silently. It's the scope that I think needs pinning down, since as written the fix is narrower than the motivation claims.

[ErenAta16]

The problem is real and turning that TypeError from len(input_ids) into a named ValueError at the boundary is the right shape of fix. I loaded validate_prompt off this branch and ran it rather than reading the diff, and the cases the PR targets all behave:

''    rejected      []    rejected      ()    rejected
{}    rejected      None  rejected      ' '   accepted (correctly, a space is a prompt)

I also checked format_prompts rather than assuming the message was wrong, since 'prompts cannot be an empty list' looked like it could fire on a bare string. It cannot: the wrap into a list happens above the check, so '' becomes [''], survives that guard, and is rejected by validate_prompt with prompt string cannot be empty. The message a caller sees is accurate in both paths.

The gap I would want closed before this lands is on the input_ids side. validate_prompt is applied to input_ids as well, and there the realistic zero-length value is not a list:

validate_prompt(np.array([], dtype=int), name='input_ids')          # accepted
validate_prompt(torch.tensor([], dtype=torch.long), name='input_ids')  # accepted
validate_prompt([], name='input_ids')                                # rejected

An empty tensor or ndarray is exactly the zero-length input this guard exists to stop, and both walk straight past it into the same len()-then-index path the PR is fixing. Someone tokenising to a tensor and hitting an empty result gets the original confusing failure, and now with a guard in place that looks like it should have caught it.

The same shape applies to bytes and set, which are less likely to show up but fall through for the same reason: the checks enumerate four concrete types instead of asking whether the value is empty.

Two ways out. Either add len()-based handling for sized objects:

if not isinstance(prompt, (str, bytes)) and hasattr(prompt, '__len__') and len(prompt) == 0:
    raise ValueError(f'{name} cannot be empty')

or keep the type list and add np.ndarray / torch.Tensor explicitly. I would lean toward the first, since it also covers whatever container type shows up next without another edit, and it keeps the docstring honest. Right now the docstring says "reject None / empty / falsy prompt input" and the implementation rejects None plus four specific empty containers, which is a narrower contract than it advertises.

One thing to be deliberate about rather than a defect: '' is now a hard error on the offline path. For a chat model that is clearly right. For a base model, generating unconditionally from an empty prompt is a legitimate thing to ask for, and this closes that off. If lmdeploy does not support that today then there is nothing to do, but it is worth a sentence in the PR description so it is a decision on record rather than a side effect of tightening the guard.

Adding a dedicated tests/test_lmdeploy/serve/test_empty_prompt_guard.py with the RED/GREEN note in the header is good practice, and it makes the regression obvious to whoever touches this next. If you take the __len__ route above, an empty-tensor case in that file would pin the part that is currently uncovered.

[SuperMarioYL]

Thanks for the thorough read @ErenAta16 — really appreciate you actually running validate_prompt against the branch. Pushed bb58b720 addressing the substantive points; notes on the rest below.

Empty tensors / generalise to __len__ (your main ask). Done. The four isinstance branches are collapsed into a single hasattr(prompt, "__len__") and len(prompt) == 0 check, so empty np.ndarray / torch.Tensor / bytes / set are now rejected by name (type(prompt).__name__ interpolation) instead of walking past the guard. A regression case for an empty ndarray input_ids is added to the test module. not [0] staying False (single-token [0] is valid) is the reason this stays __len__-based rather than a bare if not prompt — noted in the docstring so a future simplifier does not collapse it back.

Nested empty case (("", image) pairs / Openai content: ""). Not recursed into in this PR — the structured wrapper is non-empty, so it passes. Documented explicitly in the validate_prompt docstring ("Only scalar and top-level batch forms are validated; an empty prompt nested inside a (prompt, image) pair or an Openai message content field is not recursed into"). Recursing into those shapes felt like a scope expansion beyond the boundary guard this fix is about, but it is a fair follow-up if you would rather open it.

Batch behaviour change. format_prompts(["a", "", "b"]) now raises where it previously returned ["a", "", "b"]. Called out on record: an empty row in a batch is malformed input that previously crashed per-row downstream, so failing the whole batch at the boundary is intentional, but batch callers should be aware.

Empty string as a hard error for base models. Recorded as a deliberate decision rather than a side effect: unconditional generation from an empty prompt is legitimate for a base model, but lmdeploy does not expose that through format_prompts / generate() today, so there is nothing to break. If that ever changes the guard would need an opt-out.

Test module notes. Removed the internal P8-002 / RED on main, GREEN on branch tracker annotations from the header and the inline comments, and clarified the _make_engine docstring so it states the stub is what reproduces the pre-guard crash rather than the engine itself.

Method name (validate_prompt for input_ids). Agreed it reads slightly oddly, but left it as-is to avoid churn — a neutral rename would touch the two call sites and is not blocking.

ruff + docformatter (pre-commit) and the full test_empty_prompt_guard.py suite (8 cases, was 7) pass locally.

[lvhan028]

Hi, @ErenAta16

We are writing on behalf of the lmdeploy project maintainers. We have noticed that you have been posting a very large number of comments across many pull requests in our repository.

While we appreciate your enthusiasm for the project, your activity raises a serious concern: you are not a member of the official lmdeploy team, yet your frequent comments may lead other users to mistakenly perceive your personal opinions as official guidance or recommendations from the project. This can create significant confusion within the community, as your views do not necessarily represent the project’s technical direction or policy.

To avoid misleading others, we kindly ask you to limit your PR comments to only the ones you created. For broader questions or suggestions, please open a dedicated Issue instead. We also encourage you to clearly state in your comments that you are speaking on your own behalf, not as a representative of lmdeploy.

We hope you understand that we value focused, constructive participation. If this pattern of excessive commenting continues, we may need to take further moderation actions to protect the project’s workflow. We trust that you will cooperate and adjust your approach accordingly.

Thank you for your cooperation.

[ErenAta16]

Understood, and thank you for saying it directly rather than just muting me. Your point about readers mistaking an outside comment for project guidance is fair, and it is not something I had weighed properly. Volume was the mistake.

This is my last comment on a PR I did not open here. I will keep to my own (#4811) and use an issue if I have something worth raising. Apologies to the authors whose threads I added noise to.

For what it is worth, @SuperMarioYL, the __len__ change on this one looks right to me and I will leave it there.

[lvhan028]

Hi, @SuperMarioYL

Thanks for your PR. I would like to ask you to move the validation logic out of async_engine.py.

async_engine.py is a core module that should remain interface-agnostic. This kind of request‑specific check belongs better in the Pipeline or api_server layer.

Please revise your PR accordingly. Let us know if you need any clarification.

Thanks for your cooperation.

[lvhan028]

These test cases add little value and may increase maintenance burden without improving real coverage. Could you please remove this file from the PR?

[SuperMarioYL]

Done — removed the file from the PR (63ec775). Thanks for the callout.

[SuperMarioYL]

Thanks @lvhan028. Revised in 63ec775 + b6729eb:

• Moved out of async_engine.py: AsyncEngine.generate no longer calls validate_prompt — the engine stays interface-agnostic. The async_engine.py changes are dropped entirely (the file is back to its base state).
• Validation now lives at the Pipeline layer: MultimodalProcessor.format_prompts (the chokepoint pipeline.py calls) rejects empty/None prompts, and the validate_prompt helper stays in MultimodalProcessor for that layer.
• Test file removed per your feedback (63ec775).

The HTTP/OpenAI path already filters empty input upstream, so only the direct-SDK offline path loses the engine-level backstop — which is the caller's responsibility per your point. Happy to add an explicit guard at the api_server request entry points too if you'd prefer defense-in-depth there.

[lvhan028]

I've tested the following cases with main branch:

from lmdeploy import Pipeline
model_path = 'Qwen/Qwen3.5-35B-A3B-FP8'
pipe = Pipeline(model_path)
answer = pipe.infer(prompts='')  # failed, TypeError: object of type 'NoneType' has no len()
answer = pipe.infer(prompts=[]) # pass, answer is empty list
answer = pipe.infer(prompts=()) # pass. It raised "ValueError: Unsupported prompts: [()]. Only support str, openai message format, or (prompt, image or [images]) or (image or [images], prompt) pair."
answer = pipe.infer(prompts={}) # pass. It raised AssertionError: Each message should be a dict with "role" and "content" keys.

So, I think a better way to deal with the "TypeError: object of type 'NoneType' has no len()" is changing one line code in async_engine.py. That is:

if messages:  ===> if messages is not None:

[SuperMarioYL]

Thanks @lvhan028 for testing main and the proposed one-liner — adopted it in the latest push.

Changes in this update:

• async_engine.py: if messages: → if messages is not None:, so empty/falsy-but-not-None prompts (e.g. '') enter the processing branch instead of skipping it and hitting the downstream len(None) TypeError. None still skips the branch as before.
• Removed the earlier format_prompts/validate_prompt guard in multimodal.py. Keeping it would have made the one-liner dead code for the empty-string case (the guard would raise before reaching async_engine), and your "a better way" framing indicated the one-liner is the intended fix. The PR diff is now just that one line.

Net diff vs base is now a single line. Could you re-run your four cases ('', [], (), {}) against this branch when you get a chance to confirm the TypeError is gone? Happy to adjust if you'd prefer the Pipeline-layer guard retained alongside it.