feat(openai): add /v1/responses support for gpt-5 (with adapter hardening)

[gadievron]

What

Add the OpenAI /v1/responses path for gpt-5 models to the LLM adapter, plus hardening fixes on that new path found in a follow-up bug hunt — several settled against the live gpt-5 API.

Why

gpt-5-generation models reject tools + reasoning on /v1/chat/completions, which the agentic-enhance and attacker-sim verify phases need — so the adapter routes ^gpt-[5-9] to /v1/responses (_complete_responses) while gpt-4o and the o-series stay on the unchanged chat path. The four follow-up fixes close correctness gaps on that path; the most consequential is a reliability issue specific to a SAST tool — a truncated or abnormal model response could read to the pipeline as a clean, finding-free verdict.

How

Feature (commit 79fb6e0):

• _complete_responses + _responses_to_unified Message↔responses-item translation, store=False, a reasoning-aware output-token floor, and a status-before-content parser so a failed/cancelled/content-filtered response raises a typed error instead of an empty pass.
• Union _is_reasoning_model with ^gpt-[5-9] so the endpoint choice and the token-param/role choice cannot disagree.
• Dependency bump in pyproject.toml (the openai SDK version that ships the Responses API).

Hardening (this branch — all confined to utilities/llm/providers/openai.py plus its tests):

• 664a9a9 — in _responses_to_unified, an abnormal status / unknown incomplete-reason carrying partial content is relabelled stop_reason="max_tokens" and warned once (_warn_unknown_responses_status), instead of a silent end_turn. Mirrors the chat path's unknown-finish_reason warning.
• b6eab1f — validate() pings with the same reasoning={"effort": …} the scan will send, so a model-rejected effort fails at init instead of 400-ing every unit.
• 457d0b5 — _is_gpt5_responses_family (^gpt-[5-9]([.-]|$) and no -chat), used by both _use_responses_api and _is_reasoning_model, so the non-reasoning gpt-5-chat/gpt-5-chat-latest no longer route to responses+reasoning, gpt-50 is not the family, and ids are trimmed. Coupling invariant preserved.
• 9d5f405 — corrects the _complete_responses docstring: reasoning items are emitted and deliberately dropped; the loop survives because function_calls are re-sent with call_id only, never the server fc_ item id. Adds a latent-trap note.
• d6e178a — extends the non-reasoning exclusion from -chat to -chat+-search, so gpt-5-search-api (a web-search model served on Chat Completions, like gpt-4o-search-preview) is no longer swept onto responses+reasoning. Reasoning ids are guarded by a companion test.
• 270fd9c — decouples the endpoint decision from the token-param/role decision (superseding part of 457d0b5): _is_gpt5_responses_family stays the endpoint class (chat for -chat/-search), and a new _is_gpt5_completion_token_family drives the token-param — the whole gpt-5 generation except -search. Live-verified against the real API: gpt-5.2-chat-latest rejects max_tokens and needs max_completion_tokens; gpt-5-search-api accepts max_tokens. Net: -chat → chat + max_completion_tokens, -search → chat + max_tokens.
• 82ed244 — the chat-path analog of 664a9a9: _response_to_unified defaulted an unknown finish_reason to stop_reason="end_turn"; it now defaults to "max_tokens" so a proxy/future-value truncation on the chat path isn't laundered into a clean completion. Known reasons (stop/tool_calls/length/content_filter) are unchanged.

Tests

# adapter file
pytest -q tests/test_llm_openai_adapter.py | tail -1   ->  71 passed   (was 60; grep -c "def test_" = 45)
# wider llm + agent suite
pytest -q tests/ -k "llm or openai or provider or adapter or agent" | tail -1  ->  409 passed, 4 skipped, 0 failed
# full suite (whole module) green in CI on macos/ubuntu/windows (the only local reds are a zig-parser subprocess PYTHONPATH quirk, absent in CI; 0 parser files in this diff)

Each behavioral fix has a git-history red/green receipt (/rebugz). Additionally, the adapter was run end-to-end against the real gpt-5 API through openant's own config→registry path: gpt-5-nano (responses), gpt-5.2-chat-latest (chat + max_completion_tokens), and gpt-5-search-api (chat + max_tokens) all return valid completions; the abnormal-status and reasoning-drop behaviors were confirmed on the live API.

Compatibility

No breaking change; gpt-4o and the o-series paths are untouched (verified by the existing gate/coupling tests, still green).

Reachability — this path is dormant in shipped config. No gpt-5 id is registered (config/ + utilities/model_config.py carry only gpt-4o/4.1/o1/o3/o4), and build_adapter in utilities/llm/registry.py passes only api_key+base_url, so the new use_responses_api/reasoning_effort ctor args are not settable from config (effort is env-only, default "medium"). This PR hardens a path nothing in shipped config reaches yet — it activates only when an operator registers a gpt-5 id or points a base_url proxy at one. Registering a gpt-5 model is intentionally out of scope here. (The path itself is proven working: the adapter was run end-to-end against the real gpt-5 API — see Tests — so the dormancy is purely a config choice, not an untested path.)

Author notes

• Q1 (which code implements it): feature in _complete_responses + _responses_to_unified; the fixes in _responses_to_unified (abnormal-status branch), validate() (reasoning ping), the routing predicates (_is_gpt5_responses_family for the endpoint + _is_gpt5_completion_token_family for the token-param), and the _complete_responses docstring.
• Q2 (concrete input previously handled wrong): a Responses object status="incomplete", incomplete_details.reason=None, output [message: "…no issues fou"] returned stop_reason="end_turn" (a clean verdict) before 664a9a9; it now returns max_tokens + a stderr warning. Reproduced first-hand and as a regression test.
• Q3 (likely reviewer pushback + answer): "these paths aren't reachable yet, why fix them?" — the fixes are cheap, single-file, and land before gpt-5 is ever enabled; the abnormal-status one is the silent-false-negative class a SAST tool must not ship. The one deeper item is out of scope (below).

Follow-ups

• Verify-stage stop_reason gating (stacked PR). finding_verifier.py accepts a finish tool block regardless of stop_reason, so a max_tokens-truncated reply carrying a well-formed finish(agree=false, "safe") could be read as a completed verdict (a narrow silent-FN; malformed-mid-JSON truncation is already caught). BUG-7 makes the adapter report truncation honestly; the verifier must then gate finish-acceptance on stop_reason != "max_tokens" (+ a missing record_call on one cost-undercount path at finding_verifier.py:425-432). Different subsystem than this adapter PR → stacked follow-up.

Resolved during review (were open questions, now settled): the dropped-reasoning-item concern is live-confirmed benign (3/3 on gpt-5-nano — the stateless replay re-sends function_call by call_id, which the API accepts; docstring corrected in 9d5f405), and the non-reasoning-variant token-param is live-verified and fixed (270fd9c). (The enhancer degenerate-exit handling was checked and already classifies these as INCOMPLETE — no change needed there.)