fix(hf): filter model_options for apply_chat_template via Jinja AST allowlist

[planetf1]

The problem

When you pass generation options like temperature, max_new_tokens, or do_sample to LocalHFBackend, those options were being forwarded into HuggingFace's apply_chat_template. The method accepts arbitrary **kwargs and passes them straight into the Jinja template's variable namespace.

Standard HF templates silently ignore variables they don't recognise, so nothing breaks for today's models. But this is a latent correctness hazard: any model whose Jinja template happens to define a variable named temperature, num_beams, or any other GenerationConfig parameter would silently receive the caller's generation setting instead of the template author's intended value. It also makes the calling contract of model_options opaque — there's no principled boundary between what goes to apply_chat_template and what goes to generate().

How other backends handle this

For comparison, the OpenAI backend uses inspect.signature(Completions.create) to build a self-maintaining filter: it forwards only kwargs that appear in the API client's call signature. The HF backend's equivalent interface is the Jinja template itself — template variables are the "declared parameters" for apply_chat_template.

The fix

Rather than maintaining a hand-written denylist of GenerationConfig parameter names (which can never be complete — transformers has 60+ generation parameters), this PR uses the Jinja2 template as the source of truth.

At both apply_chat_template call sites, a new _filter_for_chat_template method is applied. It:

1. Parses the tokenizer's Jinja2 template with jinja2.meta.find_undeclared_variables to get the exact set of variables the template references.
2. Subtracts _HF_INTERNAL_TEMPLATE_VARS — the 5 variables HuggingFace injects automatically (see below).
3. Forwards only the keys that are in the resulting allowlist.

Generation-only options like temperature are dropped not because they appear on any list, but because no standard HF chat template ever references them as Jinja variables. The allowlist is computed once per backend instance (via functools.cached_property) and adapts automatically as models evolve.

The exclusion set — please review carefully

_HF_INTERNAL_TEMPLATE_VARS is the small set of variables that HuggingFace injects into the template namespace unconditionally. These must be excluded from the allowlist to avoid "got multiple values for keyword argument" TypeErrors or silent replacement of injected callables.

Variable	Source	Why excluded
messages	render_jinja_template named param	always injected as the first positional arg
tools	render_jinja_template named param	our call sites pass tools=convert_tools_to_json(...) explicitly
add_generation_prompt	render_jinja_template named param	passed as add_generation_prompt=True at the standard-generation call site
raise_exception	_compile_jinja_template Jinja env global	find_undeclared_variables cannot see env globals; treats this as undeclared
strftime_now	_compile_jinja_template Jinja env global	same as above

Relevant transformers source locations (verified against installed version):

• Named params: PreTrainedTokenizerBase.apply_chat_template — the render_jinja_template(...) call
• Jinja globals: transformers.utils.chat_template_utils._compile_jinja_template — the jinja_env.globals["raise_exception"] and jinja_env.globals["strftime_now"] assignments

⚠️ If transformers is upgraded, verify that this set is still accurate. The test test_hf_internal_template_vars_contents fails immediately if the constant changes without a corresponding test update — making the exclusion set an explicit, auditable contract.

Behaviour changes reviewers should be aware of

1. add_generation_prompt on the KV-cache path: because add_generation_prompt is in _HF_INTERNAL_TEMPLATE_VARS, any user-supplied value is now dropped rather than forwarded. This is correct: the KV-cache path formats context (not a generation turn), so HF's default of False is the right behaviour. A comment at the call site documents this explicitly.

2. Graceful degradation for unsupported template extensions: templates using {% generation %} / {% endgeneration %} (DeepSeek-R1, Qwen3, transformers ≥ 4.47) or unusual Jinja extensions that cannot be parsed by a plain jinja2.Environment return frozenset() — the filter forwards nothing to apply_chat_template rather than crashing. {% break %} / {% continue %} are handled correctly via jinja2.ext.loopcontrols.

3. Non-string chat_template: some tokenizers store chat_template as a list of named template dicts. The filter now guards against this with an isinstance(str) check and returns frozenset() for non-string values, leaving apply_chat_template to handle that format itself.

Changes

• mellea/backends/huggingface.py

  • New module-level constant _HF_INTERNAL_TEMPLATE_VARS (5 entries, documented above)
  • New _chat_template_allowlist cached property — parses the Jinja AST once per backend instance, with loopcontrols extension registered and graceful fallback on parse failure
  • New _filter_for_chat_template method — renames sentinels then filters by allowlist
  • Both apply_chat_template call sites updated to use _filter_for_chat_template
  • _filter_generate_only_options removed (the 21-key denylist is gone)
  • jinja2 and jinja2.meta imported at module level

• test/backends/test_huggingface_filter_options.py — 21 unit tests (no GPU needed):

  • Strict equality check on _HF_INTERNAL_TEMPLATE_VARS contents
  • Allowlist: Jinja AST parsing, exclusion of all 5 HF-internal vars, absence of generate-only names in a realistic template, caching, empty template
  • Robustness: {% generation %} tag (graceful degradation), {% break %} (loopcontrols), non-string template (isinstance guard)
  • _filter_for_chat_template: sentinel renaming, generate-only drop, template-var pass-through, empty input, unknown key drop
  • _filter_chat_template_only_options: regression guard on pre-existing method
  • Integration tests (marked huggingface): Granite tokenizer allowlist includes think + guardian_config, excludes generate-only keys and HF-internal vars

Testing

uv run pytest test/backends/test_huggingface_filter_options.py -m "not huggingface" -v
# 18 passed

The e2e test test_generate_only_options_do_not_break_generation is @pytest.mark.qualitative and requires the real model fixture.

Where this fits

Standalone bugfix. No other phases.

Closes #1154