Thinking preamble

Thinking preamble

Symptom

Reasoning-capable models may prepend text such as Thinking Process: or emit <think>...</think> before the requested JSON. Free-text distillation wastes output budget; Cognee's schema-constrained graph extraction fails validation, retries repeatedly, and may roll back the document.

Root cause

Qwen3-family chat templates default thinking mode to enabled. enable_thinking is a chat-template argument, not a normal top-level OpenAI parameter. Sending this is commonly ignored:

{"enable_thinking": false}

The hard switch is:

{"chat_template_kwargs": {"enable_thinking": false}}

This survives compatible MLX quantization because behavior is selected by the tokenizer chat template, not model weight precision.

Connector fix

Every distillation request includes:

{
  "temperature": 0.1,
  "chat_template_kwargs": {"enable_thinking": false}
}

The strict template switch is more reliable than /no_think, prompt wording, regex stripping, or token penalties.

Cognee / LiteLLM fix

LLM_ARGS: '{"chat_template_kwargs": {"enable_thinking": false}}'
STRUCTURED_OUTPUT_FRAMEWORK: instructor
LLM_INSTRUCTOR_MODE: json_schema_mode

LLM_ARGS forwards the template argument through Cognee and LiteLLM to the local server. Instructor validates the final model output against Cognee's schema.

Five-minute verification

Positive test:

curl -s "$LLM_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"<model-id>",
    "messages":[{"role":"user","content":"Reply with exactly: OK"}],
    "max_tokens":10,
    "chat_template_kwargs":{"enable_thinking":false}
  }'

Pass: content is exactly OK, with no reasoning field or preamble.

Control test: remove chat_template_kwargs. On an affected model this should reproduce the preamble, proving that the switch — not prompt luck — caused the fix.

Then ingest one isolated Cognee document and verify:

• pipeline status completed;
• zero validation retry messages;
• graph completion returns the expected fact.

Alternatives

Option	Reliability	Trade-off
server-wide model setting enable_thinking: false	high	affects every client; may require reload
/no_think suffix	medium	soft instruction; template may still emit tags
post-hoc regex	low for structured output	parsing fails before the caller can clean it
non-reasoning model	high	potentially different speed/quality profile

Never publish raw chain-of-thought from source sessions or model output as knowledge.