CI Gate

CI Gate

Docs-site pages: Scenarios · Hermetic replay · CI gate CLI. Action reference: .github/actions/tracely-gate. The 2-minute "re-break" demo: guides/DEMO.md.

You need two secrets: your Tracely API URL and an ingest key (the key is the workspace). Pick the option that matches how CI can reach your agent.

Option A — Tracely drives your agent (any language, no agent code in CI)

Register the agent's HTTP endpoint once (Scenarios → Agent endpoint), write scenarios (scripted turns, or an adversarial goal), and the action drives them:

- uses: Jwuthri/Tracely/.github/actions/tracely-gate@master
  with:
    api: https://tracely.your-co.dev
    key: ${{ secrets.TRACELY_KEY }}
    # agent: planner,support-agent   ← subset; omit to gate EVERY agent with an enabled scenario
    # min-pass-rate: "0.9"           ← adversarial suites land some probes by design

Omitting agent derives the list from enabled scenarios: a new agent is gated the day someone writes its first scenario. Each agent runs as its own gate, but they share one commit status and one PR comment; one red agent fails the job. Enabled scenarios + no endpoint → NO_COVERAGE, which blocks.

Your service must forward the traceparent header into tracely.trace(traceparent=…), otherwise the gate sees only text in / text out and tool expectations report SKIP.

Option B — grade the traces CI already emits

If your pipeline runs the agent instrumented with env="ci":

permissions:
  contents: read
  statuses: write          # the blocking commit status
  pull-requests: write     # the upserted results comment
steps:
  - uses: actions/checkout@v4
  # … your step that runs the agent and emits env=ci traces …
  - uses: Jwuthri/Tracely/.github/actions/tracely-gate@master
    with:
      mode:  gate
      agent: planner
      api:   https://tracely.your-co.dev
      key:   ${{ secrets.TRACELY_KEY }}

Option C — hermetic replay of promoted cases ($0, offline)

Re-runs your agent on each promoted case's recorded input, serving the recorded tool/LLM outputs as fixtures — no API keys, no model spend, deterministic:

- run: pip install tracely-ai
- run: tracely replay planner --entrypoint my_pkg.agent:run        # Python
  # or any language: tracely replay planner --cmd "node run.js"     # reads $TRACELY_INPUT
  env:
    TRACELY_API: https://tracely.your-co.dev
    TRACELY_KEY: ${{ secrets.TRACELY_KEY }}

Hermetic coverage: @observe(as_type="tool") tools, OpenAI chat.completions, Anthropic messages, plus Gemini / Mistral / LiteLLM since SDK 0.3.3. Elsewhere route through the call_tool / call_llm seam, or add --live.

How the verdict works

• A conversation/case fails iff a non-advisory evaluator scored FAIL.
• Ungraded conversations count against the pass rate; an all-ungraded suite is NO_COVERAGE and blocks. A green gate that tested nothing is treated as a bug.
• Adversarial scenarios are inverted: goal achieved = attack succeeded = FAIL. With no LLM key the scenario is skipped, not passed.
• Exit code 2 means "never got an answer" (timeout, unreachable API, server-side error) — not a pass.
• Driving and grading are separate worker tasks so the customer's own spans can land before grading.

Dogfooding

The repo's own tracely-gate.yml replays the bundled weather_agent example on every PR — it uses an in-repo pip install ./sdk, which is why it doesn't look like the snippets above.