Add doublecheck plugin: three-layer verification pipeline for AI output#978
Conversation
Adds a new plugin that helps users verify AI-generated content before acting on it. Designed for sensitive contexts (legal, medical, financial, compliance) where hallucinations carry real consequences. Three verification layers: - Self-Audit: extracts verifiable claims, checks internal consistency - Source Verification: web searches per claim, produces URLs for human review - Adversarial Review: assumes errors exist, checks hallucination patterns Supports persistent mode (auto-verifies every factual response inline) and one-shot mode (full report on specific text). Confidence ratings: VERIFIED, PLAUSIBLE, UNVERIFIED, DISPUTED, FABRICATION RISK. Includes: - Skill (skills/doublecheck/) with bundled report template - Agent (agents/doublecheck.agent.md) for interactive verification - Plugin package (plugins/doublecheck/) bundling both Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new doublecheck plugin/skill/agent intended to help users verify AI-generated content via a three-layer pipeline (self-audit → source verification → adversarial review), plus documentation/registry updates so it shows up in the repo’s listings and marketplace.
Changes:
- Adds the
doublecheckskill and a verification report template asset. - Adds a
doublecheckagent and adoublecheckplugin definition (plugin.json + README) and registers it in marketplace/docs. - Updates docs index tables (
README.skills.md,README.plugins.md,README.agents.md) and.github/plugin/marketplace.json.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/doublecheck/assets/verification-report-template.md | Adds a structured markdown template for full verification reports. |
| skills/doublecheck/SKILL.md | Defines the three-layer verification skill and persistent/one-shot behavior. |
| plugins/doublecheck/skills/doublecheck/assets/verification-report-template.md | Copies the report template into the plugin package (currently committed). |
| plugins/doublecheck/skills/doublecheck/SKILL.md | Copies the skill into the plugin package (currently committed). |
| plugins/doublecheck/agents/doublecheck.md | Copies the agent into the plugin package (currently committed). |
| plugins/doublecheck/README.md | Documents plugin purpose, layers, ratings, and usage patterns. |
| plugins/doublecheck/.github/plugin/plugin.json | Declares the plugin metadata and references bundled agent/skill paths. |
| docs/README.skills.md | Adds doublecheck to the skills index table. |
| docs/README.plugins.md | Adds doublecheck to the plugins index table. |
| docs/README.agents.md | Adds Doublecheck to the agents index table with install links. |
| agents/doublecheck.agent.md | Adds the root agent definition for interactive verification. |
| .github/plugin/marketplace.json | Registers the doublecheck plugin in the marketplace list. |
You can also share your feedback on Copilot code review. Take the survey.
- Fix tools frontmatter in agents/doublecheck.agent.md to use standard YAML list format instead of flow sequence with trailing comma - Remove plugins/doublecheck/agents/ and plugins/doublecheck/skills/ from tracking; these paths are in .gitignore as CI-materialized artifacts that should not be committed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
You can also share your feedback on Copilot code review. Take the survey.
| # Doublecheck | ||
|
|
||
| A three-layer verification pipeline for AI-generated output. Extracts verifiable claims, finds sources via web search, runs adversarial review for hallucination patterns, and produces a structured report with source links so humans can verify before acting. | ||
|
|
||
| ## Why This Exists | ||
|
|
||
| AI hallucinations are a model-level problem. No plugin can fix them. But the *consequences* of hallucinations -- acting on fabricated citations, relying on made-up statistics, citing nonexistent case law -- can be mitigated by making verification fast and structured. | ||
|
|
||
| Doublecheck doesn't tell you what's true. It extracts every verifiable claim from AI output, searches for sources you can check independently, and flags anything that matches known hallucination patterns. You make the final call. | ||
|
|
There was a problem hiding this comment.
This plugin README is missing the standard sections that every other plugin README in this repo includes (notably Installation with the copilot plugin install ...@awesome-copilot command, plus Source and License). Without these, users have no canonical install instructions and the plugin README is inconsistent with the rest of plugins/*/README.md.
| | Component | Type | Description | | ||
| |-----------|------|-------------| | ||
| | `doublecheck` | Skill | The core verification pipeline. Runs three layers and produces a structured report. | | ||
| | `Doublecheck` | Agent | Interactive verification mode for follow-up questions and deeper investigation. | |
There was a problem hiding this comment.
In the "What's Included" table, the agent is listed as Doublecheck, but throughout this repo plugin READMEs list agents by their identifier/filename (e.g., azure-principal-architect). Since the agent file is agents/doublecheck.agent.md (and the plugin spec path is ./agents/doublecheck.md), the identifier here should likely be doublecheck to avoid confusion about how to reference/install it.
| | `Doublecheck` | Agent | Interactive verification mode for follow-up questions and deeper investigation. | | |
| | `doublecheck` | Agent | Interactive verification mode for follow-up questions and deeper investigation. | |
|
#codeowner |
|
I've prepared the CODEOWNERS update for this PR, but encountered authentication limitations preventing automatic PR creation. The following entries should be added to the CODEOWNERS file: These entries assign A maintainer with push access can create the CODEOWNERS update PR manually, or the workflow can be re-triggered with appropriate git credentials.
|
What this adds
A new plugin called doublecheck that helps users verify AI-generated content before acting on it. Designed for sensitive contexts where mitigations for potential hallucinations are called for.
How it works
Three verification layers run in sequence:
Self-Audit -- Extracts every verifiable claim from the target text. Categorizes them (factual, statistical, citation, entity, causal, temporal). Checks for internal contradictions.
Source Verification -- Runs web searches for each claim. Produces clickable URLs so the user can verify independently. Gives extra scrutiny to citations, which are the highest-risk hallucination category.
Adversarial Review -- Assumes the output contains errors and actively tries to find them. Checks against a hallucination pattern checklist: fabricated citations, unsourced statistics, confident specificity on uncertain topics, temporal confusion, overgeneralization, missing qualifiers.
Each claim gets a confidence rating: VERIFIED, PLAUSIBLE, UNVERIFIED, DISPUTED, or FABRICATION RISK.
Two modes
use doublecheck): stays active for the conversation, auto-appends inline verification summaries to factual/analytical responses. Skips code and casual chat.Key design principle
The tool provides source links, not verdicts. "Here's where you can check" is trustworthy. "I checked and it's fine" is just more AI output. The human makes the final call.
What's included
skills/doublecheck/SKILL.mdskills/doublecheck/assets/verification-report-template.mdagents/doublecheck.agent.mdplugins/doublecheck/Validation
npm run plugin:validate-- all plugins passnpm run skill:validate-- all skills passnpm run build-- README and marketplace.json updatedbash scripts/fix-line-endings.sh-- line endings normalized