comprehension-gate

🇧🇷 Leia em português

A GitHub Action that asks pull request authors 2–4 specific questions about their own diff before merge — questions only someone who understood the change can answer. The author responds in a PR comment; the Action checks the answer against the diff and clears or flags the merge through a Check Run.

Why

Before AI-assisted coding, producing a pull request required understanding the code well enough to write it. That effort acted as an unintentional quality filter. Generating code got cheap; reviewing it did not. comprehension-gate restores that filter by charging in the currency that actually matters — comprehension, not authorship.

This is not an AI-content detector

Read this section before adopting the tool; it explains the one design decision everything else follows from.

comprehension-gate never tries to infer whether code was written by a human or a model. There is no provenance scoring, no stylometry, nothing that looks at how a diff was produced. The only thing it measures is whether a human who can explain this specific change is attached to the pull request. Code generated by an LLM and understood by its author is a legitimate, fully accepted contribution here — the check passes exactly the same way a hand-written change would.

Two practical consequences follow from this:

• False positives are the expensive failure mode. Wrongly flagging a good-faith contributor destroys trust immediately, and is treated as far costlier than letting a borderline case through. The default is informative-only (see Blocking vs. informative), and both the prompts and the schema constraints in this repo are written to prefer a pass when the model is uncertain.
• The bar is understanding, not English fluency. Questions and evaluation are tolerant of informal phrasing, typos, and non-native language, and the question language is configurable (see language in the config table). A filter that only rewards people who write well in English measures the wrong thing.

How it works

Two stateless phases, no database, no server — everything runs inside your own GitHub Actions and all state lives in PR comments, via a hidden HTML marker.

Phase 1 — a PR is opened or updated (pull_request: opened, synchronize)

1. Fetch the PR's changed files.
2. Check the exemption rules. If any match, the check completes immediately as passing — nothing further happens.
3. Build a diff context for the LLM, respecting a token budget (large diffs are truncated to the most-changed files; the questions comment says so when that happens).
4. Ask the configured LLM for 2–4 questions about the diff.
5. Post a comment with the questions (with a hidden marker carrying them, so phase 2 doesn't need to re-derive anything from the visible text).
6. Create a Check Run in a pending state.

Phase 2 — the author replies (issue_comment: created)

1. Confirm the comment is on a pull request and was written by the PR's author (this also means the Action's own comments are never mistaken for an answer — the bot's own login never matches the author's).
2. Find the questions comment via its marker.
3. Re-fetch the diff fresh and ask the LLM to evaluate the answer against it.
4. Update the Check Run: success or a non-blocking-by-default neutral (see below), with a text explanation. No numeric score, ever.
5. The author can retry as many times as needed — there is no limit and no time pressure.

Installation

1. Add a secret to your repository with an API key for one of the supported LLM providers (see LLM provider below).
2. Add the workflow below.
3. Optionally add .github/comprehension.yml to customize behavior — every key is optional and has a sensible default (see the config table).

Example workflow

# .github/workflows/comprehension-gate.yml
name: Comprehension Gate

on:
  # pull_request_target (not pull_request) so PRs from forks get a token
  # that can comment and manage check runs. This is safe here ONLY because
  # this workflow never checks out or executes the PR's code — it just
  # calls the Action, which reads PR metadata/diff over the API. Do not add
  # an actions/checkout step for the PR's head ref to this job.
  pull_request_target:
    types: [opened, synchronize]
  issue_comment:
    types: [created]

permissions:
  pull-requests: write # post comments
  checks: write        # create/update the check run
  contents: read        # read .github/comprehension.yml

concurrency:
  group: comprehension-gate-${{ github.event.pull_request.number || github.event.issue.number }}
  cancel-in-progress: false

jobs:
  comprehension-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: erickdevz/comprehension-gate@v1
        with:
          llm-api-key: ${{ secrets.GEMINI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

That's it — no actions/checkout step is required, since the Action only talks to the GitHub API and the LLM provider, never to your working tree.

Configuration

All of .github/comprehension.yml is optional; unset keys use the defaults below.

Key	Default	Meaning
provider	gemini	anthropic | openai | gemini. Gemini is the suggested default — its free tier lets a maintainer with no budget use this tool for free.
model	provider-specific fast/cheap tier	Overrides the provider's default model. Check the provider's current docs/pricing; the built-in defaults are a snapshot and may lag.
language	auto	Question/feedback language. auto detects from the diff, falling back to the repo's apparent language. Accepts any BCP-47 tag or plain language name, e.g. pt-BR, en, japanese.
questions	3	Number of questions asked per PR, 2–4.
blocking	false	See Blocking vs. informative.
min_diff_lines	10	PRs with fewer changed lines (summed over non-skip_paths files) are exempt.
trusted_after	3	Authors with this many previously passed comprehension checks in this repo are exempt from future ones. Set to 0 to disable the check for everyone (an explicit escape hatch, not a typo).
skip_paths	["**/*.md", "docs/**", "**/*.lock"]	Glob patterns; a PR touching only these paths is exempt, and these files never count toward min_diff_lines either way.
skip_labels	["skip-comprehension"]	A PR with any of these labels is exempt.

# .github/comprehension.yml — every key below is optional
provider: gemini
model: gemini-2.5-flash
language: auto
questions: 3
blocking: false
min_diff_lines: 10
trusted_after: 3
skip_paths: ["**/*.md", "docs/**", "**/*.lock"]
skip_labels: ["skip-comprehension"]

Security note: this file is always read from the PR's base branch, never from the PR's own branch — a contributor cannot raise their own min_diff_lines or set trusted_after: 0 to exempt themselves by editing this file inside their own PR.

Exemptions

The check is skipped entirely (and the Check Run completes as passing immediately) when any of the following is true:

• The PR author is a bot (dependabot, renovate, or any account of type Bot).
• The PR author has write or admin permission on the repository.
• The PR only touches files matched by skip_paths.
• The diff (outside of skip_paths files) is smaller than min_diff_lines.
• The PR has one of the skip_labels labels.
• The author has already passed trusted_after or more comprehension checks in this repository.

Blocking vs. informative

blocking: false (the default) means the Check Run never reports failure, even when the LLM judges the answer as not demonstrating understanding — it reports neutral instead, with the real feedback in the check's text. neutral cannot satisfy or fail a required status check, so even a maintainer who adds "Comprehension Gate" to required checks without reading this doc can't accidentally block a merge because of it.

blocking: true makes a failed check report conclusion: failure and fails the Action run. Setting this is necessary but not sufficient to actually block merges — you must also add "Comprehension Gate" as a required status check in the repository's branch protection settings. comprehension-gate does not, and will not, modify branch protection itself.

Either way, an infrastructure fault — the LLM call failing, hitting a rate limit, timing out, or the API key being missing or invalid — always completes the check as neutral with a warning. A broken integration must never be the reason someone's PR is stuck.

LLM provider

BYO key: you provide an API key for one provider as a repository secret; this project never sees your code or your key. All three adapters send only the (possibly truncated) diff and the questions/answers — nothing else — to the provider you configure.

Provider	Suggested for	Notes
Gemini (default)	Anyone, especially maintainers with no budget	Has a usable free tier.
Anthropic	Existing Anthropic users	Uses the Messages API.
OpenAI	Existing OpenAI users	Uses Chat Completions with response_format: json_object.

Model IDs change over time; the built-in default per provider is a best-effort snapshot (see DEFAULT_MODELS in src/config.ts) and is always overridable via model: in .github/comprehension.yml.

Permissions

The calling workflow needs:

permissions:
  pull-requests: write # post the questions/evaluation comments
  checks: write         # create and update the Check Run
  contents: read         # read .github/comprehension.yml from the base branch

The Action never logs the diff or the API key.

The prompts are the product

src/prompts.ts holds the two prompts that decide everything this tool does — what counts as a good question, and how tolerant the evaluation is. They ship with a working v0.1 starting point, but they are explicitly meant to be hand-tuned by whoever runs this tool, against real diffs from real work. "What does this function do?" is a bad question anyone can answer by reading the diff; "why did you use X instead of Y here, and what breaks if you remove it?" is a good one. If you're adopting this tool, read that file before you rely on it.

Development

npm ci
npm run typecheck
npm run lint
npm run test:coverage
npm run build   # bundles src/main.ts into dist/ — commit the result

dist/ is committed and must stay in sync with src/; CI fails the build if it isn't. This repository also runs comprehension-gate on itself (see .github/workflows/self.yml), using the code in each PR rather than the last release.

Built with AI — on purpose, and disclosed

The initial implementation of this repository (v0.1) was built with Claude Code, Anthropic's AI coding agent, working from a detailed spec. Every commit in the history carries a Co-Authored-By: Claude trailer saying so — nothing here is hidden.

That's not a contradiction of this tool's own thesis, it's the thesis in practice: comprehension-gate does not care whether code was written by a human or generated by a model. It cares whether a human understands what's being shipped. The author reviewed, tested, and can explain every part of this codebase — which is exactly the bar this tool holds your contributors to. Once self.yml is live with a real API key, future changes to this repository go through comprehension-gate itself, same as any other project using it.

If you use AI tools to write code you submit here or anywhere else, that's fine. Understand what you're shipping — that's the whole point.

License

Apache License 2.0.