Skip to content

Add batched AI false positive detection with check-specific prompts#1278

Draft
davidperezgar wants to merge 9 commits intotrunkfrom
1106-add-ai-integration-to-detect-false-positives-in-plugin-checks
Draft

Add batched AI false positive detection with check-specific prompts#1278
davidperezgar wants to merge 9 commits intotrunkfrom
1106-add-ai-integration-to-detect-false-positives-in-plugin-checks

Conversation

@davidperezgar
Copy link
Copy Markdown
Member

What?

Closes #1106

Refactors the AI false positive detection system to use a batched, check-specific approach inspired by the internal scanner, replacing the previous one-issue-at-a-time implementation. Now uses WordPress 7.0 core AI client (wp_ai_client_prompt()) instead of the old Settings_Page/wp-ai-client library.

Why?

The previous AI implementation analyzed each issue individually with a generic prompt, which was:

  • Inefficient: one API call per issue, leading to high token usage and slow execution.
  • Imprecise: a single generic prompt couldn't provide the nuanced context needed for different check types (escaping vs. sanitization vs. DB queries, etc.).
  • Outdated: relied on the old Settings_Page + \WordPress\AI_Client\AI_Client approach instead of WordPress 7.0 core AI.

The internal scanner already solved this with batched requests and check-specific prompts — this PR brings that proven pattern to PCP.

How?

Refactored AI_Analyzer trait (AI_Analyzer.php)

  • Batched approach: Issues are grouped by check code prefix (e.g., all WordPress.Security.EscapeOutput.* issues go together), then sent in batches of up to 12 cases per AI request.
  • Check-specific prompts: Each group uses a dedicated prompt template from prompts/ai-review-*.md that describes the specific issue type and what constitutes a false positive.
  • WP 7.0 core AI: Uses wp_ai_client_prompt() with using_model_preference() — same pattern as AI_Check_Names.
  • Severity filtering: Only analyzes issues with severity < threshold (default: 7 for errors, 6 for warnings), configurable via wp_plugin_check_ai_severity_threshold filter.
  • Code context: Each case includes ±10 lines of code context with a >>> marker on the flagged line.

New prompt templates (prompts/ai-review-*.md)

File Covers
ai-review-late-escaping.md WordPress.Security.EscapeOutput, PluginCheck.CodeAnalysis.EscapeOutput
ai-review-nonce-verification.md WordPress.Security.NonceVerification
ai-review-sanitization.md WordPress.Security.ValidatedSanitizedInput
ai-review-direct-db-queries.md WordPress.DB.DirectDatabaseQuery, WordPress.DB.PreparedSQL
ai-review-code-obfuscation.md PluginCheck.CodeAnalysis.Obfuscation
ai-review-setting-sanitization.md PluginCheck.CodeAnalysis.SettingSanitization
ai-review-plugin-updater.md PluginCheck.CodeAnalysis.PluginUpdater
ai-review-generic.md Fallback for unmapped check codes

CLI enhancements (Plugin_Check_Command.php)

  • New --ai-model=<model> option to select AI model (e.g., openai::gpt-4o).
  • New display_ai_summary() method showing a formatted summary with ✨ markers for false positives.

Runner updates (Abstract_Check_Runner.php)

  • Removed Settings_Page dependency.
  • Added $ai_model_preference property and set_ai_model_preference() setter.
  • Passes model preference through to analyze_results_with_ai().

Testing Instructions

  1. Ensure you are running WordPress 7.0+ with an AI connector configured (e.g., OpenAI).
  2. Install and activate Plugin Check.
  3. Run a check with AI enabled:
    wp plugin check <plugin-slug> --use-ai
  4. Verify the output shows the standard check results followed by an AI summary section:
    ────────────────────────────────────────────────────────────
    ✨ AI False Positive Analysis
    ────────────────────────────────────────────────────────────
    Issues analyzed: X
    False positives detected: Y
    Tokens spent: Z
    
  5. Test with a specific model:
    wp plugin check <plugin-slug> --use-ai --ai-model=openai::gpt-4o
  6. Test with a plugin that has no issues — should show "No errors found" without AI section.
  7. Test without --use-ai flag — should behave exactly as before (no AI analysis).

AI Usage Disclosure

  • This PR was created without the help of AI tools
  • This PR includes AI-assisted code or content

If AI tools were used, please describe how they were used:
Used GitHub Copilot (Claude) to assist with the implementation of the batched AI analysis approach, prompt template creation, and CLI output formatting. The architecture and prompt content were designed based on the existing internal scanner implementation patterns.

@davidperezgar davidperezgar linked an issue Apr 26, 2026 that may be closed by this pull request
Co-authored-by: Copilot <copilot@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add AI Integration to Detect False Positives in Plugin Checks

1 participant