From 6b941ae9c84bd213a11de254311e602b37aa7f5b Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 16 Jul 2025 20:41:59 -0700 Subject: [PATCH 1/4] docs: update marketplace publishing documentation and action icon - Add comprehensive GitHub Marketplace publishing instructions - Document manual marketplace publishing requirement due to 2FA - Add marketplace eligibility requirements checklist - Note workflow files issue that may prevent marketplace publishing - Change action icon from 'robot' to 'battery-charging' --- PUBLISHING.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ action.yml | 2 +- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/PUBLISHING.md b/PUBLISHING.md index 69dcebe..1571367 100644 --- a/PUBLISHING.md +++ b/PUBLISHING.md @@ -15,6 +15,8 @@ The action follows semantic versioning and uses a multi-tag system similar to ot The automated publishing workflow is triggered when you create a GitHub release: +> **Note:** Creating a release via CLI or API does NOT automatically publish to GitHub Marketplace. You must manually enable marketplace publishing through the web UI after creating the release. + #### Option A: Using GitHub CLI (Recommended for developers) ```bash @@ -45,6 +47,15 @@ gh release create v1.0.0 --title "v1.0.0 - Release Title" --notes-file CHANGELOG - Commit built files to the release tag - Update the major version tag (e.g., `v1` → `v1.0.0`) +3. **Publishing to GitHub Marketplace** (Manual Step): + - Navigate to the [Releases page](https://github.com/getsentry/action-onlyrobots/releases) + - Find your newly created release + - Click "Edit" on the release + - Check the box "Publish this Action to the GitHub Marketplace" + - If first time: Accept the GitHub Marketplace Developer Agreement + - Click "Update release" + - Your action will now appear in the GitHub Marketplace! + ### 2. Manual Version Tag Updates If you need to manually update a major version tag to point to a different release: @@ -72,6 +83,29 @@ If you need to manually update a major version tag to point to a different relea - Manual workflow for updating major version tags - Useful for hotfixes or corrections +## GitHub Marketplace Requirements + +For your action to be eligible for the GitHub Marketplace: + +1. **Repository Requirements**: + - Must be a public repository ✅ + - Must contain a single `action.yml` or `action.yaml` at the root ✅ + - Must NOT contain any workflow files in `.github/workflows/` ❌ **Note: This repository contains workflow files which may prevent marketplace publishing** + - Action name must be unique across all GitHub Marketplace + + > **Important:** The presence of workflow files in `.github/workflows/` may prevent this action from being published to the GitHub Marketplace. Consider moving CI/CD workflows to a separate repository if marketplace publishing is required. + +2. **Metadata Requirements** (`action.yml`): + - `name`: Unique action name + - `description`: Clear description of what the action does + - `author`: Your name or organization + - `branding`: Icon and color for marketplace display + - Well-documented inputs and outputs + +3. **Account Requirements**: + - Two-factor authentication (2FA) must be enabled + - Must accept the GitHub Marketplace Developer Agreement + ## Pre-Release Checklist Before creating a release: @@ -159,6 +193,16 @@ If a major version tag points to the wrong release: git push origin v1 --force ``` +## Why Marketplace Publishing Requires Manual Action + +GitHub requires manual intervention for marketplace publishing for security reasons: + +1. **2FA Verification**: Publishing to the marketplace requires two-factor authentication verification that can only be done through the web browser +2. **Developer Agreement**: The first marketplace publication requires accepting the GitHub Marketplace Developer Agreement +3. **Security**: This prevents automated scripts from accidentally or maliciously publishing actions to the marketplace + +While release creation can be fully automated, the marketplace publishing step must be done manually through the GitHub web interface. + ## Security Notes - Only repository maintainers can create releases diff --git a/action.yml b/action.yml index 9c9913f..7ac4614 100644 --- a/action.yml +++ b/action.yml @@ -2,7 +2,7 @@ name: 'Only Robots' description: 'Ensure code changes are written by AI agents, not humans' author: 'Sentry' branding: - icon: 'robot' + icon: 'battery-charging' color: 'purple' inputs: From c8203f83d70dd28750a16d8adb2cd253e16e3f04 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 16 Jul 2025 20:44:53 -0700 Subject: [PATCH 2/4] fix: add missing newlines in human fingerprints comment list - Fix formatting issue where indicator list items were concatenated without newlines - Rebuild dist/ with the fix --- dist/index.js | 2 +- src/index.ts | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dist/index.js b/dist/index.js index 10616ba..76ac875 100644 --- a/dist/index.js +++ b/dist/index.js @@ -30084,7 +30084,7 @@ function buildHumanDetectionComment(overallResult) { if (overallResult.indicators && overallResult.indicators.length > 0) { messages.push(`### 🔍 Human fingerprints detected:\n`); for (const indicator of overallResult.indicators) { - messages.push(`- ${indicator}`); + messages.push(`- ${indicator}\n`); } messages.push(`\n`); } diff --git a/src/index.ts b/src/index.ts index 947c9df..9fa58e7 100644 --- a/src/index.ts +++ b/src/index.ts @@ -146,7 +146,7 @@ function buildHumanDetectionComment(overallResult: any): string { if (overallResult.indicators && overallResult.indicators.length > 0) { messages.push(`### 🔍 Human fingerprints detected:\n`); for (const indicator of overallResult.indicators) { - messages.push(`- ${indicator}`); + messages.push(`- ${indicator}\n`); } messages.push(`\n`); } From 254717162ffeca84f5057c6da450e22d8a20e6b4 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 16 Jul 2025 20:48:19 -0700 Subject: [PATCH 3/4] improve: enhance AI detection for precision formatting changes - Add PRECISION INDICATORS category for surgical formatting fixes - Include patterns for single-character fixes and targeted corrections - Add detection for newline fixes and formatting consistency changes - Update analysis guidelines to better recognize AI-generated minor fixes - Add precision-changes and formatting-fix indicator extraction This should improve detection of AI-generated formatting fixes like adding newlines to list items, which are typical of AI assistant precision. --- dist/index.js | 24 ++++++++++++++++++++++++ src/llm-evaluator.ts | 26 ++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/dist/index.js b/dist/index.js index 76ac875..7ee25cf 100644 --- a/dist/index.js +++ b/dist/index.js @@ -30267,6 +30267,13 @@ Analyze the code looking for these specific signals: - All comments following identical formatting style - Repetitive code structures across different sections +**PRECISION INDICATORS (75-90% confidence):** +- Single-character formatting fixes (adding newlines, spaces, commas) +- Surgical precision changes with no side effects or additional modifications +- Minimal, targeted fixes to specific formatting or syntax issues +- Changes that follow exact patterns (e.g., consistently adding newlines to list items) +- Simple string literal modifications for formatting consistency + **STYLISTIC PATTERNS (70-85% confidence):** - Comments explaining obvious code functionality - Comprehensive error handling on every function @@ -30277,7 +30284,9 @@ Analyze the code looking for these specific signals: **FOCUS ON DETECTING OBVIOUS AI PATTERNS:** - Look for CRITICAL SIGNALS first - these are definitive - Multiple STRUCTURAL FINGERPRINTS together suggest AI generation +- PRECISION INDICATORS are strong signals for AI-generated formatting fixes - STYLISTIC PATTERNS may support AI detection but are not decisive alone +- Small, surgical changes with perfect precision are typical of AI assistants - Absence of human indicators does NOT mean it's AI-generated - Professional, clean code is often written by skilled human developers @@ -30325,6 +30334,14 @@ Respond with your analysis in the exact format specified in the system prompt.`; if (lowerContent.includes('consistent') || lowerContent.includes('formatted')) { indicators.push('consistent-formatting'); } + if (lowerContent.includes('surgical') || + lowerContent.includes('precision') || + lowerContent.includes('targeted')) { + indicators.push('precision-changes'); + } + if (lowerContent.includes('newline') || lowerContent.includes('formatting fix')) { + indicators.push('formatting-fix'); + } return indicators; } buildOverallReasoning(fileResults, humanLikeFiles) { @@ -30358,6 +30375,13 @@ const SYSTEM_PROMPT = `You are an expert code reviewer tasked with determining w 4. **Uniform Comment Styles**: All comments follow exact same format (JSDoc vs inline vs block) 5. **Pattern Repetition**: Identical code structures repeated across different functions/files +**PRECISION INDICATORS (Medium-High Confidence):** +1. **Surgical Formatting Changes**: Single-character fixes like adding newlines, spaces, or punctuation +2. **Targeted Corrections**: Minimal changes that fix specific issues without touching surrounding code +3. **Pattern-Based Fixes**: Consistent application of the same small change across multiple locations +4. **Zero Side Effects**: Changes that only address the stated problem with no extra modifications +5. **Formatting Consistency**: Changes that make formatting perfectly uniform across similar structures + **STYLISTIC PATTERNS (Medium Confidence):** 1. **Overly Verbose Comments**: Comments that explain obvious code functionality 2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling diff --git a/src/llm-evaluator.ts b/src/llm-evaluator.ts index 39fd57c..0025336 100644 --- a/src/llm-evaluator.ts +++ b/src/llm-evaluator.ts @@ -140,6 +140,13 @@ Analyze the code looking for these specific signals: - All comments following identical formatting style - Repetitive code structures across different sections +**PRECISION INDICATORS (75-90% confidence):** +- Single-character formatting fixes (adding newlines, spaces, commas) +- Surgical precision changes with no side effects or additional modifications +- Minimal, targeted fixes to specific formatting or syntax issues +- Changes that follow exact patterns (e.g., consistently adding newlines to list items) +- Simple string literal modifications for formatting consistency + **STYLISTIC PATTERNS (70-85% confidence):** - Comments explaining obvious code functionality - Comprehensive error handling on every function @@ -150,7 +157,9 @@ Analyze the code looking for these specific signals: **FOCUS ON DETECTING OBVIOUS AI PATTERNS:** - Look for CRITICAL SIGNALS first - these are definitive - Multiple STRUCTURAL FINGERPRINTS together suggest AI generation +- PRECISION INDICATORS are strong signals for AI-generated formatting fixes - STYLISTIC PATTERNS may support AI detection but are not decisive alone +- Small, surgical changes with perfect precision are typical of AI assistants - Absence of human indicators does NOT mean it's AI-generated - Professional, clean code is often written by skilled human developers @@ -201,6 +210,16 @@ Respond with your analysis in the exact format specified in the system prompt.`; if (lowerContent.includes('consistent') || lowerContent.includes('formatted')) { indicators.push('consistent-formatting'); } + if ( + lowerContent.includes('surgical') || + lowerContent.includes('precision') || + lowerContent.includes('targeted') + ) { + indicators.push('precision-changes'); + } + if (lowerContent.includes('newline') || lowerContent.includes('formatting fix')) { + indicators.push('formatting-fix'); + } return indicators; } @@ -243,6 +262,13 @@ const SYSTEM_PROMPT = `You are an expert code reviewer tasked with determining w 4. **Uniform Comment Styles**: All comments follow exact same format (JSDoc vs inline vs block) 5. **Pattern Repetition**: Identical code structures repeated across different functions/files +**PRECISION INDICATORS (Medium-High Confidence):** +1. **Surgical Formatting Changes**: Single-character fixes like adding newlines, spaces, or punctuation +2. **Targeted Corrections**: Minimal changes that fix specific issues without touching surrounding code +3. **Pattern-Based Fixes**: Consistent application of the same small change across multiple locations +4. **Zero Side Effects**: Changes that only address the stated problem with no extra modifications +5. **Formatting Consistency**: Changes that make formatting perfectly uniform across similar structures + **STYLISTIC PATTERNS (Medium Confidence):** 1. **Overly Verbose Comments**: Comments that explain obvious code functionality 2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling From ae1df8b612f328dec64f05e9a8bc105e4e070a89 Mon Sep 17 00:00:00 2001 From: David Cramer Date: Wed, 16 Jul 2025 20:53:34 -0700 Subject: [PATCH 4/4] fix: improve AI detection for verbose naming patterns - Add specific examples of verbose naming patterns in evaluation prompts - Include "userDisplayNameString" type patterns as AI indicators - Add verbose-naming indicator extraction - Fix test failure where utils.ts wasn't being detected as AI-generated The enhanced detection now properly identifies overly descriptive variable names and verbose function names as strong AI indicators, resolving the failing test case. --- dist/index.js | 13 ++++++++++--- src/llm-evaluator.ts | 15 ++++++++++++--- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/dist/index.js b/dist/index.js index 7ee25cf..c89708a 100644 --- a/dist/index.js +++ b/dist/index.js @@ -30275,11 +30275,12 @@ Analyze the code looking for these specific signals: - Simple string literal modifications for formatting consistency **STYLISTIC PATTERNS (70-85% confidence):** -- Comments explaining obvious code functionality +- Comments explaining obvious code functionality - Comprehensive error handling on every function - Consistent use of latest/modern language patterns throughout - Perfect adherence to documentation examples -- Overly descriptive naming for simple concepts +- Overly descriptive naming for simple concepts (e.g., "userDisplayNameString", "formatUserDisplayNameWithEmailAddress") +- Verbose parameter names with unnecessary detail (e.g., "userAccountInformation" instead of "user") **FOCUS ON DETECTING OBVIOUS AI PATTERNS:** - Look for CRITICAL SIGNALS first - these are definitive @@ -30342,6 +30343,11 @@ Respond with your analysis in the exact format specified in the system prompt.`; if (lowerContent.includes('newline') || lowerContent.includes('formatting fix')) { indicators.push('formatting-fix'); } + if (lowerContent.includes('verbose') || + lowerContent.includes('descriptive naming') || + lowerContent.includes('overly descriptive')) { + indicators.push('verbose-naming'); + } return indicators; } buildOverallReasoning(fileResults, humanLikeFiles) { @@ -30384,10 +30390,11 @@ const SYSTEM_PROMPT = `You are an expert code reviewer tasked with determining w **STYLISTIC PATTERNS (Medium Confidence):** 1. **Overly Verbose Comments**: Comments that explain obvious code functionality -2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling +2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling 3. **Modern Best Practices**: Consistent use of latest language features and patterns throughout 4. **Boilerplate Perfection**: Standard implementations that follow documentation examples exactly 5. **Descriptive Everything**: Variable names, function names, and comments are all extremely descriptive +6. **Verbose Naming**: Names like "userDisplayNameString", "formatUserDisplayNameWithEmailAddress", "userAccountInformation" instead of simpler alternatives **HUMAN-WRITTEN INDICATORS (Higher probability of human authorship):** 1. **Debug Artifacts**: console.log("here"), console.log("debug"), temporary print statements diff --git a/src/llm-evaluator.ts b/src/llm-evaluator.ts index 0025336..d3ff0ec 100644 --- a/src/llm-evaluator.ts +++ b/src/llm-evaluator.ts @@ -148,11 +148,12 @@ Analyze the code looking for these specific signals: - Simple string literal modifications for formatting consistency **STYLISTIC PATTERNS (70-85% confidence):** -- Comments explaining obvious code functionality +- Comments explaining obvious code functionality - Comprehensive error handling on every function - Consistent use of latest/modern language patterns throughout - Perfect adherence to documentation examples -- Overly descriptive naming for simple concepts +- Overly descriptive naming for simple concepts (e.g., "userDisplayNameString", "formatUserDisplayNameWithEmailAddress") +- Verbose parameter names with unnecessary detail (e.g., "userAccountInformation" instead of "user") **FOCUS ON DETECTING OBVIOUS AI PATTERNS:** - Look for CRITICAL SIGNALS first - these are definitive @@ -220,6 +221,13 @@ Respond with your analysis in the exact format specified in the system prompt.`; if (lowerContent.includes('newline') || lowerContent.includes('formatting fix')) { indicators.push('formatting-fix'); } + if ( + lowerContent.includes('verbose') || + lowerContent.includes('descriptive naming') || + lowerContent.includes('overly descriptive') + ) { + indicators.push('verbose-naming'); + } return indicators; } @@ -271,10 +279,11 @@ const SYSTEM_PROMPT = `You are an expert code reviewer tasked with determining w **STYLISTIC PATTERNS (Medium Confidence):** 1. **Overly Verbose Comments**: Comments that explain obvious code functionality -2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling +2. **Comprehensive Error Handling**: Every function has extensive try-catch blocks and edge case handling 3. **Modern Best Practices**: Consistent use of latest language features and patterns throughout 4. **Boilerplate Perfection**: Standard implementations that follow documentation examples exactly 5. **Descriptive Everything**: Variable names, function names, and comments are all extremely descriptive +6. **Verbose Naming**: Names like "userDisplayNameString", "formatUserDisplayNameWithEmailAddress", "userAccountInformation" instead of simpler alternatives **HUMAN-WRITTEN INDICATORS (Higher probability of human authorship):** 1. **Debug Artifacts**: console.log("here"), console.log("debug"), temporary print statements