feat(deslop): add buzzword inflation detection - #113
Conversation
Detect quality claims (production-ready, secure, scalable, etc.) in documentation that lack supporting code evidence. This completes the AI slop research implementation roadmap (Priority 3, Task #7). - Add analyzeBuzzwordInflation() function with claim extraction - Support 6 buzzword categories: production, enterprise, security, scale, reliability, completeness - Search for evidence patterns: tests, error handling, logging, auth, validation, encryption, async, cache, pool - Distinguish positive claims from TODOs/FIXMEs (reduce false positives) - Add 50+ tests for new functionality
Summary of ChangesHello @avifenesh, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates a sophisticated "Buzzword Inflation Detection" mechanism into the /deslop-around tool. Its primary goal is to enhance code quality and documentation integrity by automatically identifying and flagging instances where high-level quality claims (such as "secure" or "scalable") are made in project documentation or comments without sufficient, verifiable evidence within the codebase itself. This helps maintain a realistic and honest representation of a project's capabilities. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e3b67481ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const { files: sourceFiles } = countSourceFiles(repoPath, options); | ||
|
|
||
| // Extract all claims from claim source files |
There was a problem hiding this comment.
The evidence scan is driven by countSourceFiles, which filters out test files via isTestFile, but the new evidence patterns explicitly look for test filenames to substantiate production/reliability claims. As a result, repositories where tests are the primary evidence (e.g., only *.test.* files and no logging/error-handling patterns) will be incorrectly flagged as having insufficient evidence. This makes the analyzer systematically report false positives unless tests are also searched.
Useful? React with 👍 / 👎.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new 'buzzword inflation' analyzer, a significant feature that detects unsubstantiated quality claims in documentation by searching for code evidence. The implementation is comprehensive and includes an extensive test suite. My review focuses on a critical bug in the file exclusion logic that could lead to incorrect analysis, along with several suggestions to improve performance and maintainability in the core analyzer functions and tests. Once these issues are addressed, this will be a solid addition to the tool.
| if (shouldExclude(entry.name)) continue; | ||
|
|
||
| const fullPath = path.join(dir, entry.name); | ||
| const relativePath = path.relative(repoPath, fullPath); |
There was a problem hiding this comment.
There's a bug in findClaimSourceFiles. The shouldExclude function is called with entry.name instead of the full relative path. shouldExclude works by splitting the path, so it needs the full path from the repository root to correctly identify and exclude nested directories (e.g., src/vendor). Using just the entry name will cause the exclusion logic to fail for any excluded directory not at the top level of the walk.
| if (shouldExclude(entry.name)) continue; | |
| const fullPath = path.join(dir, entry.name); | |
| const relativePath = path.relative(repoPath, fullPath); | |
| const fullPath = path.join(dir, entry.name); | |
| const relativePath = path.relative(repoPath, fullPath); | |
| if (shouldExclude(relativePath)) continue; |
References
- Always validate input paths to prevent path traversal vulnerabilities. Ensure all file operations are constrained within the intended project directory.
| const fs = require('fs'); | ||
| const path = require('path'); | ||
| const os = require('os'); |
There was a problem hiding this comment.
The fs, path, and os modules are required multiple times within different it blocks in the 'analyzeBuzzwordInflation (integration)' test suite (e.g., lines 1761-1763, 1795-1797, 1825-1827). To improve code clarity and avoid repetition, consider moving these require statements to the top of the describe('analyzeBuzzwordInflation (integration)', ...) block (around line 1759).
| for (const [category, buzzwords] of Object.entries(buzzwordCategories)) { | ||
| for (const buzzword of buzzwords) { | ||
| // Build case-insensitive word-boundary regex | ||
| const buzzwordRegex = new RegExp(`\\b${escapeRegex(buzzword)}\\b`, 'i'); | ||
| const match = buzzwordRegex.exec(line); | ||
|
|
There was a problem hiding this comment.
The extractClaims function has a performance issue. It iterates through all categories and buzzwords for each line of content, creating a new RegExp object inside the innermost loop. For large files, this can be very inefficient.
A more performant approach would be to build a single, comprehensive regex from all buzzwords outside the loop, and then use it to find all matches on each line. This avoids repeated iterations and regex compilations.
For example, you could build a single regex like this:
const allBuzzwords = Object.values(buzzwordCategories).flat();
const allBuzzwordsRegex = new RegExp(`\\b(${allBuzzwords.map(escapeRegex).join('|')})\\b`, 'gi');Then, for each line, you can find all matches using line.matchAll(allBuzzwordsRegex) and process them.
References
- When matching keywords within strings, such as for categorization based on labels, use word-boundary regular expressions instead of simple substring inclusion to prevent false positives (e.g., matching 'bug' in 'debug').
| for (const [evidenceType, regexes] of Object.entries(patterns)) { | ||
| for (const regex of regexes) { | ||
| // For file path patterns (like test file detection), test the path | ||
| if (regex.source.includes('\\.[jt]sx?$') || regex.source.includes('_test\\.')) { |
There was a problem hiding this comment.
The logic in searchEvidence to differentiate between file path patterns and content patterns is brittle. It relies on checking if the regex source string contains \\.test\\.[jt]sx?$ or _test\\.. This is not robust; if a new file path-based pattern is added without these specific substrings, it will be incorrectly treated as a content pattern.
A more maintainable approach would be to structure the EVIDENCE_PATTERNS object to explicitly separate path and content patterns. For example:
const EVIDENCE_PATTERNS = {
production: {
tests: {
path: [/\\.test\\.[jt]sx?$/],
},
errorHandling: {
content: [/try\\s*\\{|catch\\s*\\(/],
},
},
// ...
};This would make the logic in searchEvidence cleaner and less prone to errors when new patterns are added.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
All 7 tasks from AI-SLOP-RESEARCH.md are now implemented: - Priority 1: placeholder detection, doc/code ratio, phantom refs - Priority 2: generic naming, verbosity detection - Priority 3: over-engineering metrics, buzzword inflation (PR #113) Ready for v2.7.0 release.
There was a problem hiding this comment.
Pull request overview
This PR implements buzzword inflation detection for the /deslop-around command, completing the final task (#7) from the AI-SLOP-RESEARCH.md roadmap. The feature detects quality claims (like "production-ready", "secure", "scalable") in documentation and comments without supporting code evidence.
Changes:
- Added
buzzword_inflationpattern configuration with 6 categories and evidence patterns - Implemented
analyzeBuzzwordInflation()with claim extraction, evidence search, and gap detection logic - Added 50+ comprehensive tests including integration tests and ReDoS safety tests
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
| lib/patterns/slop-patterns.js | Added buzzword_inflation pattern configuration with multi-pass analysis requirements |
| lib/patterns/slop-analyzers.js | Implemented complete buzzword inflation detection system with helper functions, constants, and main analyzer |
| tests/slop-analyzers.test.js | Added comprehensive test suite covering claim extraction, evidence search, gap detection, integration scenarios, and ReDoS safety |
| CHANGELOG.md | Documented the new feature with detailed explanation of categories, evidence patterns, and behavior |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (regex.test(content)) { | ||
| if (!evidence.categories[evidenceType]) { | ||
| evidence.categories[evidenceType] = []; | ||
| } | ||
| if (!evidence.categories[evidenceType].includes(file)) { | ||
| evidence.categories[evidenceType].push(file); | ||
| evidence.total++; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The evidence counting logic has a subtle behavior that may not be intuitive. A file that matches multiple regex patterns within the same evidence type (e.g., both "try...catch" and "except:" for errorHandling) only counts as 1 evidence match. However, if the same file matches patterns across different evidence types (e.g., errorHandling and logging), each counts separately. While this appears intentional based on the per-file deduplication within each category, it could lead to unexpected results. Consider documenting this behavior clearly, or reconsidering whether each distinct pattern match should count separately to better quantify the actual evidence present.
| buzzword_inflation: { | ||
| pattern: null, // Requires multi-pass analysis | ||
| exclude: ['*.test.*', '*.spec.*', '**/tests/**', '**/node_modules/**'], | ||
| severity: 'high', | ||
| autoFix: 'flag', // Cannot auto-fix documentation claims | ||
| language: null, // Universal - all languages | ||
| description: 'Quality claims (production-ready, secure, scalable) without supporting code evidence', | ||
| requiresMultiPass: true, | ||
| // Minimum evidence matches required to substantiate a claim | ||
| minEvidenceMatches: 2 | ||
| } |
There was a problem hiding this comment.
The new buzzword_inflation pattern should have tests in tests/slop-patterns.test.js similar to the tests for over_engineering_metrics pattern. These tests should verify that the pattern is properly included in getMultiPassPatterns(), getUniversalPatterns(), getPatternsBySeverity('high'), and getPatternsByAutoFix('flag'). While the analyzer has comprehensive tests, the pattern metadata itself should also be tested to ensure it's properly configured and discoverable.
| for (const regex of regexes) { | ||
| // For file path patterns (like test file detection), test the path | ||
| if (regex.source.includes('\\.[jt]sx?$') || regex.source.includes('_test\\.')) { | ||
| if (regex.test(file)) { | ||
| if (!evidence.categories[evidenceType]) { | ||
| evidence.categories[evidenceType] = []; | ||
| } | ||
| if (!evidence.categories[evidenceType].includes(file)) { | ||
| evidence.categories[evidenceType].push(file); | ||
| evidence.total++; | ||
| } | ||
| } | ||
| } else { | ||
| // For content patterns, test the content | ||
| if (regex.test(content)) { | ||
| if (!evidence.categories[evidenceType]) { | ||
| evidence.categories[evidenceType] = []; | ||
| } | ||
| if (!evidence.categories[evidenceType].includes(file)) { | ||
| evidence.categories[evidenceType].push(file); | ||
| evidence.total++; | ||
| } | ||
| } |
There was a problem hiding this comment.
The heuristic to distinguish file path patterns from content patterns is fragile. It checks if the regex source contains specific strings like '\.[jt]sx?$' or '_test\.', but this approach is brittle and may incorrectly classify patterns. For example, a content pattern that happens to match these substrings would be incorrectly treated as a file path pattern. Consider adding an explicit metadata flag to each pattern definition to indicate whether it should test file paths or content.
| for (const regex of regexes) { | |
| // For file path patterns (like test file detection), test the path | |
| if (regex.source.includes('\\.[jt]sx?$') || regex.source.includes('_test\\.')) { | |
| if (regex.test(file)) { | |
| if (!evidence.categories[evidenceType]) { | |
| evidence.categories[evidenceType] = []; | |
| } | |
| if (!evidence.categories[evidenceType].includes(file)) { | |
| evidence.categories[evidenceType].push(file); | |
| evidence.total++; | |
| } | |
| } | |
| } else { | |
| // For content patterns, test the content | |
| if (regex.test(content)) { | |
| if (!evidence.categories[evidenceType]) { | |
| evidence.categories[evidenceType] = []; | |
| } | |
| if (!evidence.categories[evidenceType].includes(file)) { | |
| evidence.categories[evidenceType].push(file); | |
| evidence.total++; | |
| } | |
| } | |
| for (const patternDef of regexes) { | |
| // Support both bare RegExp and objects with metadata { regex: RegExp, target: 'path' | 'content' } | |
| let regex; | |
| let target = 'content'; // default to content-based matching | |
| if (patternDef instanceof RegExp) { | |
| regex = patternDef; | |
| } else if (patternDef && typeof patternDef === 'object') { | |
| regex = patternDef.regex; | |
| if (patternDef.target === 'path' || patternDef.target === 'content') { | |
| target = patternDef.target; | |
| } | |
| } | |
| if (!(regex instanceof RegExp)) { | |
| continue; | |
| } | |
| const testValue = target === 'path' ? file : content; | |
| if (regex.test(testValue)) { | |
| if (!evidence.categories[evidenceType]) { | |
| evidence.categories[evidenceType] = []; | |
| } | |
| if (!evidence.categories[evidenceType].includes(file)) { | |
| evidence.categories[evidenceType].push(file); | |
| evidence.total++; | |
| } |
| }, | ||
| completeness: { | ||
| edgeCases: [/edge.?case|boundary|corner.?case/i], | ||
| errorHandling: [/invalid|error|exception|fail/i], |
There was a problem hiding this comment.
The errorHandling pattern for the 'completeness' category is overly broad. The regex /invalid|error|exception|fail/i will match common words like "fail", "error", "exception", and "invalid" even when they appear in variable names, comments, or unrelated contexts. This could lead to false positives where the mere presence of these common words is counted as evidence of comprehensive error handling. Consider making this pattern more specific by requiring these words to appear in specific contexts like function names, class names, or error handling blocks.
| errorHandling: [/invalid|error|exception|fail/i], | |
| errorHandling: [ | |
| /\b(error|exception|failure)s?\s+handl(?:er|ing|ed|es)\b/i, // "error handling", "errors handled", "exception handler" | |
| /\b(handle|handles|handled|handling)\s+(all\s+)?(errors?|exceptions?|failures?)\b/i, // "handle all errors", "handles exceptions" | |
| /\b(all\s+)?(errors?|exceptions?|failures?)\s+(are|is)\s+(handled|caught|managed)\b/i // "all errors are handled" | |
| ], |
| for (const file of filesToSearch) { | ||
| let content; | ||
| try { | ||
| const fullPath = path.isAbsolute(file) ? file : path.join(repoPath, file); | ||
| content = fs.readFileSync(fullPath, 'utf8'); | ||
| } catch { | ||
| continue; // Skip unreadable files | ||
| } |
There was a problem hiding this comment.
The searchEvidence function reads file content for every file in filesToSearch, even for patterns that only need to test file paths. This is inefficient because file path patterns (like test file detection) don't require reading the file content at all. Consider restructuring the logic to separate file path checks from content checks, and only read file content when necessary. This would significantly improve performance when analyzing large repositories.
| /\bprovides?\s+/i, // "provides secure" | ||
| /\boffers?\s+/i, // "offers robust" | ||
| /\bfeatures?\s+/i, // "features comprehensive" | ||
| /\bwith\s+/i, // "with enterprise-grade" |
There was a problem hiding this comment.
The CLAIM_INDICATORS pattern /\bwith\s+/i is too broad and will match many common phrases that aren't actually claims. For example, "compatible with enterprise-grade systems" or "works with secure connections" would be flagged as positive claims even though they're describing compatibility, not making quality claims about the current codebase. Consider making this pattern more restrictive by combining it with other indicators or requiring specific sentence structures.
| /\bwith\s+/i, // "with enterprise-grade" |
| completeness: { | ||
| edgeCases: [/edge.?case|boundary|corner.?case/i], | ||
| errorHandling: [/invalid|error|exception|fail/i], | ||
| documentation: [/\/\*\*|\/\/\/|#.*docstring|"""/] |
There was a problem hiding this comment.
The regex pattern for completeness documentation evidence //**|///|#.*docstring|"""/ has a potential issue with the docstring portion. The pattern #.*docstring will match any line containing a '#' followed by any characters and then "docstring", which could lead to false positives. For example, "# This is not a docstring" would match. Consider anchoring this pattern more carefully or making it more specific to actual Python docstring syntax.
| documentation: [/\/\*\*|\/\/\/|#.*docstring|"""/] | |
| documentation: [/\/\*\*|\/\/\/|"""|'''/] |
| */ | ||
| buzzword_inflation: { | ||
| pattern: null, // Requires multi-pass analysis | ||
| exclude: ['*.test.*', '*.spec.*', '**/tests/**', '**/node_modules/**'], |
There was a problem hiding this comment.
The buzzword_inflation pattern configuration specifies an exclude property with patterns like '.test.', '.spec.', '/tests/', but the analyzeBuzzwordInflation function doesn't use this property. Instead, it relies on the hardcoded exclusion logic in countSourceFiles and isTestFile. This creates a disconnect between the pattern configuration and the actual implementation. Either the exclude property should be removed from the pattern configuration (since it's not used), or the analyzer should be modified to respect the pattern's exclude property for consistency with other patterns in the codebase.
| exclude: ['*.test.*', '*.spec.*', '**/tests/**', '**/node_modules/**'], | |
| exclude: [], // Exclusions handled by analyzer logic (e.g., isTestFile/countSourceFiles) |
| /\bFIXME\b/i, | ||
| /\bshould\s+be\b/i, | ||
| /\bwill\s+be\b/i, | ||
| /\bmake\s+(it\s+)?/i, |
There was a problem hiding this comment.
The NOT_CLAIM_INDICATORS pattern /\bmake\s+(it\s+)?/i is too broad and will incorrectly match legitimate statements. For example, "We make secure applications" or "Our tools make production-ready code" are positive claims, not aspirational statements, but would be flagged as non-claims. Consider refining this pattern to specifically match patterns like "make it more" or "make this" which are more clearly aspirational.
| /\bmake\s+(it\s+)?/i, | |
| /\bmake\s+(?:it\s+(?:more|less|better|easier|faster|slower|work)|this\b)/i, |
- Fix shouldExclude using entry.name instead of full relative path - Include test files in evidence search (tests are evidence) - Optimize extractClaims with single combined regex + Map lookup - Add isFilePathPattern() helper for robust path vs content detection - Move require statements to top of describe block in tests
|
Addressed all review feedback in commit 661e4b2: ✅ High: shouldExclude bug - Fixed to use full relative path instead of entry.name |
Code reviewNo issues found in the code changes. Checked for bugs and CLAUDE.md compliance. Summary:
Note: Per CLAUDE.md section "PR Auto-Review Process", this PR should receive reviews from all 4 automated reviewers (Copilot, Claude, Gemini, Codex). Currently, only 3 of 4 reviewers have commented (missing: Claude/Anthropic AI reviewer). However, this is a process/infrastructure observation rather than a code issue. |
- Add buzzword_inflation pattern tests to slop-patterns.test.js - Fix overly broad errorHandling pattern in completeness category - Fix documentation regex (remove #.*docstring false positive) - Remove overly broad /with/ from CLAIM_INDICATORS - Refine NOT_CLAIM_INDICATORS /make/ pattern to be more specific - Update exclude property to empty array with explanatory comment - Optimize searchEvidence to skip file reads for path-only patterns
|
Addressed Copilot review feedback in commit c4da202: ✅ Pattern tests - Added 9 tests for buzzword_inflation pattern in slop-patterns.test.js |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const EVIDENCE_PATTERNS = { | ||
| production: { | ||
| tests: [/\.test\.[jt]sx?$|\.spec\.[jt]sx?$|__tests__|test_.*\.py$|_test\.go$|_test\.rs$/], | ||
| errorHandling: [/try\s*\{|catch\s*\(|\.catch\s*\(|except\s*:|if\s+let\s+Err|match.*Err\(/], |
There was a problem hiding this comment.
The regex pattern /match.*Err\(/ contains a potentially dangerous .* quantifier that could cause catastrophic backtracking (ReDoS) with malicious inputs like "match" followed by many characters that don't result in "Err(". Consider using a more specific pattern like /match[^E]*Err\(/ or use a non-greedy quantifier /match.*?Err\(/ to reduce backtracking risk.
| errorHandling: [/try\s*\{|catch\s*\(|\.catch\s*\(|except\s*:|if\s+let\s+Err|match.*Err\(/], | |
| errorHandling: [/try\s*\{|catch\s*\(|\.catch\s*\(|except\s*:|if\s+let\s+Err|match.*?Err\(/], |
| }, | ||
| enterprise: { | ||
| auth: [/authenticat|authorization|permission|rbac|acl|role/i], | ||
| audit: [/audit|track.*event|event.*log|activity.*log/i], |
There was a problem hiding this comment.
The regex pattern /track.*event/ and /event.*log/ contain potentially dangerous .* quantifiers that could cause catastrophic backtracking (ReDoS). With inputs containing "track" or "event" followed by many characters without the expected end pattern, these could cause performance issues. Consider using non-greedy quantifiers (.*?) or more specific patterns to reduce backtracking risk.
| audit: [/audit|track.*event|event.*log|activity.*log/i], | |
| audit: [/audit|track.*?event|event.*?log|activity.*log/i], |
Summary
This completes the AI slop research implementation roadmap from
docs/AI-SLOP-RESEARCH.md.Test plan
Closes the final task from AI-SLOP-RESEARCH.md