Skip to content

Disable thinking in LLMs so prompt expansion will work properly with thinking models - #9380

Open
shanedk wants to merge 7 commits into
invoke-ai:mainfrom
shanedk:fix-for-thinking-llms
Open

Disable thinking in LLMs so prompt expansion will work properly with thinking models#9380
shanedk wants to merge 7 commits into
invoke-ai:mainfrom
shanedk:fix-for-thinking-llms

Conversation

@shanedk

@shanedk shanedk commented Jul 23, 2026

Copy link
Copy Markdown

This is my first ever PR, so be gentle! 😊

Summary

A bug fix to disable thinking on text LLM models so prompt expansion will work properly with thinking models.

Related Issues / Discussions

Closes #9379

QA Instructions

  1. Try a thinking model. I was using Goekdeniz-Guelmez/Josiefied-Qwen3-1.7B-abliterated-v1
  2. Type in a prompt
  3. Use it to expand a prompt
  4. Notice that you just get thinking tokens, no prompt
  5. Apply this change
  6. Try it again
  7. Notice that you get a nicely expanded prompt now

Merge Plan

This is just adding an extra parameter, so I'm unaware of any such issues.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions Bot added python PRs that change python files backend PRs that change backend files labels Jul 23, 2026
@Pfannkuchensack Pfannkuchensack self-assigned this Jul 24, 2026
@Pfannkuchensack

Copy link
Copy Markdown
Collaborator

PR #9380 — Disable thinking in LLMs

  • No test coverage, and the existing tests cannot catch a revert. Both TextLLMPipeline suites drive the pipeline with MagicMock tokenizers that swallow **kwargs, and every assertion reads only call_args[0][0] (the messages list). Reverting this PR leaves the suite green. A hermetic test is cheap: invokeai/backend/qwen3/qwen3_tokenizer.py:27 exposes load_bundled_qwen3_tokenizer() and the real Qwen3 chat template ships in invokeai/backend/qwen3/tokenizer/tokenizer_config.json — no network, no weights, no GPU. Assert the rendered prompt ends with <think>\n\n</think>\n\n.
  • The retry branch at invokeai/backend/text_llm_pipeline.py:63-65 has no coverage of any kind. Add a fake tokenizer that raises ValueError("System role not supported") on the first call and assert the second apply_chat_template call also carries enable_thinking=False.
  • Pre-existing, but inside the block being edited: invokeai/backend/text_llm_pipeline.py:74 re-tokenizes the rendered template with add_special_tokens at its default True. transformers' own tokenize path passes False precisely because the template already emits its control tokens — this duplicates BOS for Gemma-family models. Qwen has no BOS, which is why Qwen-based manual testing could not surface it.
  • No docs and no user-facing signal about which models the fix actually covers. All three shipped TextLLM starters are non-thinking models, so the affected population is entirely user-installed.

@shanedk

shanedk commented Aug 6, 2026

Copy link
Copy Markdown
Author

It happened with almost every other model I tried: Qwen/Qwen3-0.6B and 1.7B (and variants), google/gemma-3-1b-it (and variants), and a few others.

I haven't had any issue with Qwen3 or gemma3-based models since I added this, and it doesn't seem to have created any problems with Llama-3.2 or SmolLM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files python PRs that change python files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: Thinking models currently unusable for prompt expansion

2 participants