raggate-action

Fail the PR when your LLM/RAG answers regress — one composite Action that runs raggate against a YAML suite and enforces a trusted baseline.

- uses: actions/checkout@v4
- uses: mutton-dev/raggate-action@v1
  with:
    config: path/to/suite.yaml
    baseline: path/to/baseline.json   # commit from main / trusted store

日本語: README.ja.md · raggate intro (Zenn): https://zenn.dev/mutton/articles/b0e17496a34b13

Inputs

Name	Required	Default	Description
config	yes	—	Path to the raggate suite YAML.
baseline	no	""	Path to a trusted baseline results JSON. When empty, the action only runs the suite (no gate).
raggate-version	no	v0.1.2	raggate module version to install (verified via the Go checksum database).
comment	no	false	Post the compare report as a PR comment (same-repository pull requests only; needs pull-requests: write).

Outputs

Name	Description
results-path	Path to the run results JSON.
compare-markdown-path	Path to the rendered compare report (empty when no baseline was given).
gate-passed	"true" when thresholds passed (or no baseline), "false" when the quality gate failed.

Minimal workflow

name: rag-gate
on:
  pull_request:
permissions:
  contents: read
  # pull-requests: write   # only if comment: true
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: mutton-dev/raggate-action@v1
        with:
          config: testdata/suite.yaml
          baseline: testdata/baseline.json
          # comment: "true"

Exit behavior: the step fails when raggate’s quality gate fails (compare exit code 3). Read gate-passed for explicit asserts in follow-up steps.

Security

Treat suite YAML and baselines as code. This action will call whatever base_url values appear in the suite (HTTP/HTTPS only, per raggate).

Fork pull requests and comments

• On pull_request from a fork, GitHub issues a read-only GITHUB_TOKEN for the base repository. Commenting on the PR is not available with that token, and secrets are not exposed to the fork workflow.
• This action’s comment: true path only runs when
  github.event.pull_request.head.repo.full_name == github.repository
  (same-repository PRs). That avoids writing comments (or relying on write tokens) for untrusted fork workflows.
• Grant permissions: pull-requests: write only when you enable comment: true on same-repo PRs.

Do not use pull_request_target for this gate

pull_request_target runs with the base repository’s privileges while checking out untrusted PR code is a common footgun (“pwn request”). Prefer plain pull_request and a baseline stored on the base branch or another trusted location.

Reference: Preventing pwn requests (GitHub Security Lab).

Trusted baselines (do not let the PR relax its own gate)

• Keep baseline.json on the default branch, a release artifact, or an internal store you control.
• A PR that regenerates and commits its own baseline can hide regressions.
• raggate compare defaults to thresholds from the baseline (--thresholds baseline), so a PR cannot quietly loosen min_score / max_score_drop by editing only the suite used for the current run. Align your workflow with that default: always pass a baseline produced outside the untrusted change set.

raggate install integrity

The action installs raggate with Go module tooling (go install …@version). Module contents are verified against the public Go checksum database (sum.golang.org), which mitigates tampered module proxies for the published module path.

Versioning

• Use the moving major tag v1 for the latest compatible 1.x Action (uses: mutton-dev/raggate-action@v1).
• Pin a full release tag or commit SHA when you need bit-for-bit reproducibility.
• The raggate-version input pins the CLI independently of the Action tag.

Local fixture (this repo)

# Terminal A
go run github.com/mutton-dev/raggate/examples/mockrag@v0.1.2 --port 18080

# Terminal B
go run github.com/mutton-dev/raggate/cmd/raggate@v0.1.2 run -c testdata/suite.yaml -o /tmp/out.json

Use --degrade on mockrag to force a quality drop (citations stripped, generic answer).

Links

• raggate CLI: https://github.com/mutton-dev/raggate
• Zenn article (ja): https://zenn.dev/mutton/articles/b0e17496a34b13
• License: MIT