Skip to content

v1.1.0

Latest

Choose a tag to compare

@shaifulshabuj shaifulshabuj released this 11 Jun 21:03

Added

  • ollama adapter — local-model backend (internal/ai/ollamacli.go, closes #30).
    teststop now ships a first-class ollama adapter that calls the ollama HTTP API at
    localhost:11434. No subprocess, no API key, no account quota.

    • Default model: qwen3.6:latest (36B Q4_K_M). Override with TESTSTOP_MODEL.
    • stream: false, think: false, num_ctx: 32768 per request.
    • Automatic <think>…</think> block stripping for qwen3 models on older ollama builds.
    • JSON-only output constraint appended to the mandate for local models (local models are
      less instruction-following than cloud models; cloud adapters are unaffected).
    • IsOllamaAvailable() pings localhost:11434 with a 2s timeout for fast detection.
  • Auto-detection precedence changed: ollama → claude → copilot.
    TESTSTOP_CLI=auto (the default) now prefers ollama when the local server is reachable.
    Cloud CLIs (claude, copilot) are still fully supported but opt-in:
    TESTSTOP_CLI=claude or TESTSTOP_CLI=copilot. TESTSTOP_CLI=ollama forces ollama only.

    Quality tradeoff (measured, against waymark API project):

    Model Scenarios Time Notes
    qwen3.6:latest 29–31 ~3–4 min High specificity; IDOR, race conditions, token expiry
    gemma4:latest 52 ~8 min Most thorough; covers auth, concurrency, edge inputs
    qwen3:4b Not viable; outputs reasoning prose, never generates JSON

    Scenario depth and edge-case creativity from local models are moderately lower than
    claude's output. For production runs where quota is not a concern,
    TESTSTOP_CLI=claude remains the highest-quality choice.

  • Resilient JSON parsing for local models (ParseScenariosFromJSON). Local models
    produce sloppier output than cloud CLIs. The parser now uses a three-pass strategy:
    (1) direct unmarshal; (2) extract […] from prose-wrapped output (qwen3:4b pattern —
    reasoning before JSON); (3) sanitize invalid JSON escape sequences then retry (gemma4
    pattern — emits \xNN hex notation which is not valid JSON). Cloud adapter output
    is unaffected (pass 1 always succeeds for claude/copilot).

  • Design document (docs/design/ollama-adapter.md). Records the HTTP-vs-CLI decision,
    num_ctx rationale, think: false approach, and JSON-suffix strategy.