Skip to content

v0.3.0 — the LLM-experience release: llm check / show / runs

Choose a tag to compare

@Mrgoudan Mrgoudan released this 10 Jul 16:16

Making the model behind agent.run operable like infrastructure — whether it's a CLI agent, a hosted API, or a model served on localhost. Schema v3 (auto-migrates).

Highlights

  • forgeflow llm check — one live probe per agent binding and per embedding model: endpoint reachable, auth valid, model loaded, and the model can follow the fenced-JSON output contract. Exit 1 on failure — croneable. Replay bindings report available recordings.
  • forgeflow llm show ROLE --data '{...}' — the exact assembled prompt a step would send, plus the prompt_sha the audit trail pins.
  • forgeflow llm runs + per-model metrics — verdicts, wall-time, and re-ask counts per run and per model. Rising re-asks = your early warning that a model update started fumbling the contract.
  • Three-stage binding validation, all fail-loud: structure at pack load, environment at engine start (CLI resolved and pinned, secrets present — after any --replay-from wrap, so replay CI never demands the live backend), live chain via llm check.
  • params: passthrough for openai-compat (temperature, max_tokens, response_format — the Ollama/vLLM/llama.cpp knobs) and max_turns / extra_args for the agentic CLI backend.
  • docs/LLM.md — copy-paste recipes for Ollama, vLLM, llama.cpp, LM Studio, gateways, the Claude CLI, and record/replay, plus the error-class troubleshooting table.

185 tests. Full details in CHANGELOG.md.