From f8240e8a1a9578de61d33c3f5618268ed82acc6f Mon Sep 17 00:00:00 2001 From: NiloCK Date: Sat, 5 Jul 2025 10:14:07 -0300 Subject: [PATCH 01/14] Add cron automation for nightly CI health checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add nightly-ci-check.sh script for automated CI failure analysis - Create dedicated worktree per CI failure for isolated fixes - Generate assessment.md for all failures, resolution.md for fixes - Use Claude Code for autonomous root cause analysis and fixes - Conservative approach: >75% confidence threshold for auto-fixes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- cron/README.md | 118 +++++++++++++++++++++++++++++++ cron/nightly-ci-check.sh | 149 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 267 insertions(+) create mode 100644 cron/README.md create mode 100755 cron/nightly-ci-check.sh diff --git a/cron/README.md b/cron/README.md new file mode 100644 index 000000000..57d35a7bc --- /dev/null +++ b/cron/README.md @@ -0,0 +1,118 @@ +# Cron Automation Scripts + +This directory contains dev-local cron automation scripts for agentic quality control and CI/CD monitoring. + +## Purpose + +These scripts enable automated, AI-driven quality control by: +- Monitoring CI/CD pipeline health +- Performing automated root cause analysis of failures +- Attempting autonomous fixes for high-confidence issues +- Generating detailed reports for manual review + +## Scripts + +### `nightly-ci-check.sh` + +**Purpose**: Automated nightly CI health check and fix attempt + +**Schedule**: Runs at 2:00 AM daily via cron + +**Workflow**: +1. Checks recent GitHub workflow runs using `gh` CLI +2. For each failed workflow, creates a dedicated worktree using `nt` command +3. Performs root cause analysis using Claude Code +4. Always creates `assessment--YYYYMMDD.md` with analysis +5. For high-confidence fixes (>75%), attempts fix and creates `resolution--YYYYMMDD.md` +6. For deferred fixes, creates `status--YYYYMMDD.md` with reasoning + +**Dependencies**: +- GitHub CLI (`gh`) - for workflow and log access +- Claude Code CLI (`claude`) - for AI analysis and fixes +- Local `nt` command - for worktree management (defined in `~/.colin.bashrc`) + +## Setup + +### 1. Install Dependencies + +```bash +# Install GitHub CLI +sudo apt install gh # or equivalent for your OS + +# Install Claude Code +npm install -g @anthropic-ai/claude-code + +# Authenticate with GitHub +gh auth login +``` + +### 2. Configure Cron Job + +Add to your crontab (`crontab -e`): + +```cron +# Nightly CI health check at 2:00 AM +0 2 * * * /path/to/your/repo/cron/nightly-ci-check.sh +``` + +### 3. Directory Structure + +``` +cron/ +├── README.md # This file +├── nightly-ci-check.sh # Main automation script +└── reports/ # Analysis reports and logs + ├── nightly-ci-YYYYMMDD.log # Execution logs + ├── assessment--YYYYMMDD.md # Always created + ├── resolution--YYYYMMDD.md # If fix attempted + └── status--YYYYMMDD.md # If fix deferred +``` + +## Configuration + +Edit `nightly-ci-check.sh` to customize: +- `CLAUDE_TIMEOUT`: Max time for Claude operations (default: 1 hour) +- Log retention policies +- Confidence thresholds for auto-fixes +- Notification preferences + +## Security Considerations + +- Scripts run with your local user permissions +- GitHub access uses your authenticated `gh` CLI session +- Claude Code uses your API configuration +- All operations are logged for audit purposes +- Fix branches are created locally and not automatically pushed + +## Monitoring + +Check logs in `cron/reports/` for: +- Execution status and timing +- CI analysis results +- Fix attempts and outcomes +- Error conditions and debugging info + +## Troubleshooting + +**Common Issues**: +- `gh` authentication expired: Run `gh auth login` +- `claude` not found: Ensure Claude Code is installed and in PATH +- Permission denied: Ensure script is executable (`chmod +x`) +- Timeout issues: Increase `CLAUDE_TIMEOUT` value + +**Debug Mode**: +Run script manually to test: +```bash +cd /path/to/repo +./cron/nightly-ci-check.sh +``` + +## Philosophy + +This automation follows the principle of "assisted autonomy": +- High-confidence, low-risk fixes are automated +- Complex or risky issues generate detailed analysis for human review +- All actions are logged and auditable +- Conservative approach prioritizes safety over speed + +The goal is to handle routine CI failures automatically while escalating complex issues to human developers with comprehensive analysis. \ No newline at end of file diff --git a/cron/nightly-ci-check.sh b/cron/nightly-ci-check.sh new file mode 100755 index 000000000..54c4d8b6d --- /dev/null +++ b/cron/nightly-ci-check.sh @@ -0,0 +1,149 @@ +#!/bin/bash + +# Nightly CI Check Script for Claude Code Automation +# This script runs at 2 AM daily to check for CI failures and attempt automated fixes + +set -euo pipefail + +# Configuration +REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +REPORT_DIR="$REPO_DIR/cron/reports" +LOG_FILE="$REPORT_DIR/nightly-ci-$(date +%Y%m%d).log" +CLAUDE_TIMEOUT=3600 # 1 hour timeout for Claude operations + +# Ensure report directory exists +mkdir -p "$REPORT_DIR" + +# Logging function +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE" +} + +# Error handling +cleanup() { + log "Script interrupted or completed. Cleaning up..." + # Add any cleanup operations here +} +trap cleanup EXIT + +# Main execution +main() { + log "Starting nightly CI check for $(pwd)" + + # Change to repo directory + cd "$REPO_DIR" + + # Ensure we're in a git repository + if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + log "ERROR: Not in a git repository. Exiting." + exit 1 + fi + + # Check if gh CLI is available + if ! command -v gh >/dev/null 2>&1; then + log "ERROR: GitHub CLI (gh) not found. Please install it first." + exit 1 + fi + + # Check if claude is available + if ! command -v claude >/dev/null 2>&1; then + log "ERROR: Claude Code CLI not found. Please install it first." + exit 1 + fi + + log "Starting automated CI analysis with Claude Code..." + + # Run Claude Code with comprehensive CI analysis prompt + timeout "$CLAUDE_TIMEOUT" claude --non-interactive <<'EOF' +# Nightly CI Analysis and Auto-Fix Task + +You are running an automated nightly CI health check. Your task is to: + +1. **Analyze CI Status**: Use `gh` to check recent workflow runs and identify failures +2. **Per-Failure Processing**: For EACH failed workflow, create a separate worktree and analysis: + - Use `nt cc-resolve-$(date +%Y%m%d)-` for each failure + - Find the last known-good run of the same workflow + - Pull logs and perform root cause analysis + - List PRs/commits between good and bad runs + - Assess likelihood of successful autonomous fix (0-100%) + +## Analysis Framework + +### Step 1: Get CI Status +```bash +gh run list --limit 20 --json status,conclusion,workflowName,createdAt,headSha,url +``` + +### Step 2: For Each Failure (Separate Worktrees) +```bash +# Create dedicated worktree for this failure +nt cc-resolve-$(date +%Y%m%d)- +cd ../cc-resolve-$(date +%Y%m%d)- + +# Get detailed run info +gh run view --json jobs,conclusion,workflowName,headSha,url +gh run view --log +gh run list --workflow= --status=success --limit 1 +``` + +### Step 3: Root Cause Analysis +Analyze: +- Error messages in logs +- Changed files between good/bad commits +- Common failure patterns (deps, tests, build, lint) +- Environmental factors + +### Step 4: Fix Assessment & Documentation +Rate confidence (0-100%) based on: +- **High confidence (80-100%)**: Simple dependency updates, known lint fixes, obvious typos +- **Medium confidence (60-79%)**: Test failures with clear fixes, build config issues +- **Low confidence (0-59%)**: Complex logic errors, environmental issues, unclear failures + +### Step 5: Create Reports (in each worktree) +For each failure, create in `cron/reports/`: + +**Always create**: `assessment--$(date +%Y%m%d).md` +- Failure summary and root cause analysis +- Commit/PR range analysis +- Fix confidence assessment + +**If fix attempted**: `resolution--$(date +%Y%m%d).md` +- Fix implementation details +- Test results and verification +- Success/failure status + +**If fix deferred**: `status--$(date +%Y%m%d).md` +- Reasons for deferral +- Manual intervention required +- Next steps recommendation + +### Step 6: Auto-Fix Execution (if confidence >75%) +1. Implement fix in the dedicated worktree +2. Run relevant tests to verify +3. Commit with descriptive message +4. Document in resolution.md + +## Important Notes +- Each failure gets its own worktree for isolation +- Always create assessment.md, plus resolution.md OR status.md +- Conservative approach: when in doubt, defer to manual review +- Use proper git hygiene and descriptive commits + +Begin analysis now. +EOF + + local claude_exit_code=$? + + if [ $claude_exit_code -eq 0 ]; then + log "Claude Code analysis completed successfully" + elif [ $claude_exit_code -eq 124 ]; then + log "WARNING: Claude Code analysis timed out after ${CLAUDE_TIMEOUT}s" + else + log "ERROR: Claude Code analysis failed with exit code $claude_exit_code" + fi + + log "Nightly CI check completed. See full log at: $LOG_FILE" +} + +# Run main function +main "$@" \ No newline at end of file From 63c3d4e8299a4fc1bafe9350491d3443c0769477 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Sun, 6 Jul 2025 11:41:14 -0300 Subject: [PATCH 02/14] chop --- cron/README.md | 115 ------------------------------------------------- 1 file changed, 115 deletions(-) diff --git a/cron/README.md b/cron/README.md index 57d35a7bc..828c400ca 100644 --- a/cron/README.md +++ b/cron/README.md @@ -1,118 +1,3 @@ # Cron Automation Scripts This directory contains dev-local cron automation scripts for agentic quality control and CI/CD monitoring. - -## Purpose - -These scripts enable automated, AI-driven quality control by: -- Monitoring CI/CD pipeline health -- Performing automated root cause analysis of failures -- Attempting autonomous fixes for high-confidence issues -- Generating detailed reports for manual review - -## Scripts - -### `nightly-ci-check.sh` - -**Purpose**: Automated nightly CI health check and fix attempt - -**Schedule**: Runs at 2:00 AM daily via cron - -**Workflow**: -1. Checks recent GitHub workflow runs using `gh` CLI -2. For each failed workflow, creates a dedicated worktree using `nt` command -3. Performs root cause analysis using Claude Code -4. Always creates `assessment--YYYYMMDD.md` with analysis -5. For high-confidence fixes (>75%), attempts fix and creates `resolution--YYYYMMDD.md` -6. For deferred fixes, creates `status--YYYYMMDD.md` with reasoning - -**Dependencies**: -- GitHub CLI (`gh`) - for workflow and log access -- Claude Code CLI (`claude`) - for AI analysis and fixes -- Local `nt` command - for worktree management (defined in `~/.colin.bashrc`) - -## Setup - -### 1. Install Dependencies - -```bash -# Install GitHub CLI -sudo apt install gh # or equivalent for your OS - -# Install Claude Code -npm install -g @anthropic-ai/claude-code - -# Authenticate with GitHub -gh auth login -``` - -### 2. Configure Cron Job - -Add to your crontab (`crontab -e`): - -```cron -# Nightly CI health check at 2:00 AM -0 2 * * * /path/to/your/repo/cron/nightly-ci-check.sh -``` - -### 3. Directory Structure - -``` -cron/ -├── README.md # This file -├── nightly-ci-check.sh # Main automation script -└── reports/ # Analysis reports and logs - ├── nightly-ci-YYYYMMDD.log # Execution logs - ├── assessment--YYYYMMDD.md # Always created - ├── resolution--YYYYMMDD.md # If fix attempted - └── status--YYYYMMDD.md # If fix deferred -``` - -## Configuration - -Edit `nightly-ci-check.sh` to customize: -- `CLAUDE_TIMEOUT`: Max time for Claude operations (default: 1 hour) -- Log retention policies -- Confidence thresholds for auto-fixes -- Notification preferences - -## Security Considerations - -- Scripts run with your local user permissions -- GitHub access uses your authenticated `gh` CLI session -- Claude Code uses your API configuration -- All operations are logged for audit purposes -- Fix branches are created locally and not automatically pushed - -## Monitoring - -Check logs in `cron/reports/` for: -- Execution status and timing -- CI analysis results -- Fix attempts and outcomes -- Error conditions and debugging info - -## Troubleshooting - -**Common Issues**: -- `gh` authentication expired: Run `gh auth login` -- `claude` not found: Ensure Claude Code is installed and in PATH -- Permission denied: Ensure script is executable (`chmod +x`) -- Timeout issues: Increase `CLAUDE_TIMEOUT` value - -**Debug Mode**: -Run script manually to test: -```bash -cd /path/to/repo -./cron/nightly-ci-check.sh -``` - -## Philosophy - -This automation follows the principle of "assisted autonomy": -- High-confidence, low-risk fixes are automated -- Complex or risky issues generate detailed analysis for human review -- All actions are logged and auditable -- Conservative approach prioritizes safety over speed - -The goal is to handle routine CI failures automatically while escalating complex issues to human developers with comprehensive analysis. \ No newline at end of file From af37166b4a88e68d5e9e230950329eff99cff885 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Sun, 6 Jul 2025 11:50:48 -0300 Subject: [PATCH 03/14] add info --- cron/README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/cron/README.md b/cron/README.md index 828c400ca..6c4b56928 100644 --- a/cron/README.md +++ b/cron/README.md @@ -1,3 +1,28 @@ # Cron Automation Scripts This directory contains dev-local cron automation scripts for agentic quality control and CI/CD monitoring. + +## Scripts + +### `nightly-ci-check.sh` + +**Purpose**: Automated nightly CI health check and fix attempt + +**Schedule**: Runs at 2:00 AM daily via cron + +**Workflow**: +1. Checks recent GitHub workflow runs using `gh` CLI +2. For each failed workflow, creates a dedicated worktree using `nt` command +3. Performs root cause analysis using Claude Code +4. Always creates `assessment--YYYYMMDD.md` with analysis +5. For high-confidence fixes (>75%), attempts fix and creates `resolution--YYYYMMDD.md` +6. For deferred fixes, creates `status--YYYYMMDD.md` with reasoning + +### Configure Cron Job + +Add to your crontab (`crontab -e`): + +```cron +# Nightly CI health check at 2:00 AM +0 2 * * * /path/to/your/repo/cron/nightly-ci-check.sh +``` From a5b9f630d4fa09c4f398405ac26cfd30212d8df6 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Sun, 6 Jul 2025 11:50:54 -0300 Subject: [PATCH 04/14] nightlies only --- cron/nightly-ci-check.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cron/nightly-ci-check.sh b/cron/nightly-ci-check.sh index 54c4d8b6d..47abf6916 100755 --- a/cron/nightly-ci-check.sh +++ b/cron/nightly-ci-check.sh @@ -59,7 +59,7 @@ main() { You are running an automated nightly CI health check. Your task is to: -1. **Analyze CI Status**: Use `gh` to check recent workflow runs and identify failures +1. **Analyze CI Status**: Use `gh` to check recent schedule-triggered workflow runs and identify failures 2. **Per-Failure Processing**: For EACH failed workflow, create a separate worktree and analysis: - Use `nt cc-resolve-$(date +%Y%m%d)-` for each failure - Find the last known-good run of the same workflow @@ -69,9 +69,9 @@ You are running an automated nightly CI health check. Your task is to: ## Analysis Framework -### Step 1: Get CI Status +### Step 1: Get CI Status (Schedule-triggered runs only) ```bash -gh run list --limit 20 --json status,conclusion,workflowName,createdAt,headSha,url +gh run list --limit 20 --json status,conclusion,workflowName,createdAt,headSha,url,event --jq '.[] | select(.event == "schedule")' ``` ### Step 2: For Each Failure (Separate Worktrees) @@ -83,7 +83,7 @@ cd ../cc-resolve-$(date +%Y%m%d)- # Get detailed run info gh run view --json jobs,conclusion,workflowName,headSha,url gh run view --log -gh run list --workflow= --status=success --limit 1 +gh run list --workflow= --status=success --limit 1 --json status,conclusion,workflowName,createdAt,headSha,url,event --jq '.[] | select(.event == "schedule")' ``` ### Step 3: Root Cause Analysis From cf6d53e1faafd7b9178763b6c8fb4339d429f47f Mon Sep 17 00:00:00 2001 From: NiloCK Date: Sun, 6 Jul 2025 13:36:44 -0300 Subject: [PATCH 05/14] use ts --- cron/README.md | 29 ++- cron/nightly-ci-check.sh | 149 ------------ cron/nightly-ci-check.ts | 510 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 531 insertions(+), 157 deletions(-) delete mode 100755 cron/nightly-ci-check.sh create mode 100755 cron/nightly-ci-check.ts diff --git a/cron/README.md b/cron/README.md index 6c4b56928..4cf102942 100644 --- a/cron/README.md +++ b/cron/README.md @@ -4,19 +4,25 @@ This directory contains dev-local cron automation scripts for agentic quality co ## Scripts -### `nightly-ci-check.sh` +### `nightly-ci-check.ts` **Purpose**: Automated nightly CI health check and fix attempt +**Requirements**: +- Node.js with `tsx` for TypeScript execution +- GitHub CLI (`gh`) authenticated +- Claude Code CLI (`claude`) +- Git 2.5+ with worktree support + **Schedule**: Runs at 2:00 AM daily via cron **Workflow**: -1. Checks recent GitHub workflow runs using `gh` CLI -2. For each failed workflow, creates a dedicated worktree using `nt` command -3. Performs root cause analysis using Claude Code -4. Always creates `assessment--YYYYMMDD.md` with analysis -5. For high-confidence fixes (>75%), attempts fix and creates `resolution--YYYYMMDD.md` -6. For deferred fixes, creates `status--YYYYMMDD.md` with reasoning +1. Checks recent GitHub workflow runs using GitHub CLI +2. Filters for scheduled workflows with unresolved failures/cancellations +3. For each failure, creates a dedicated worktree using `nt` command +4. Collects failure data: logs, metadata, commit ranges, PR information +5. Invokes Claude Code for root cause analysis in isolated worktree +6. Generates assessment reports and attempts fixes based on confidence level ### Configure Cron Job @@ -24,5 +30,12 @@ Add to your crontab (`crontab -e`): ```cron # Nightly CI health check at 2:00 AM -0 2 * * * /path/to/your/repo/cron/nightly-ci-check.sh +0 2 * * * cd /path/to/your/repo && npx tsx cron/nightly-ci-check.ts ``` + +### Manual Testing + +```bash +cd /path/to/your/repo +npx tsx cron/nightly-ci-check.ts +``` \ No newline at end of file diff --git a/cron/nightly-ci-check.sh b/cron/nightly-ci-check.sh deleted file mode 100755 index 47abf6916..000000000 --- a/cron/nightly-ci-check.sh +++ /dev/null @@ -1,149 +0,0 @@ -#!/bin/bash - -# Nightly CI Check Script for Claude Code Automation -# This script runs at 2 AM daily to check for CI failures and attempt automated fixes - -set -euo pipefail - -# Configuration -REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -REPORT_DIR="$REPO_DIR/cron/reports" -LOG_FILE="$REPORT_DIR/nightly-ci-$(date +%Y%m%d).log" -CLAUDE_TIMEOUT=3600 # 1 hour timeout for Claude operations - -# Ensure report directory exists -mkdir -p "$REPORT_DIR" - -# Logging function -log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE" -} - -# Error handling -cleanup() { - log "Script interrupted or completed. Cleaning up..." - # Add any cleanup operations here -} -trap cleanup EXIT - -# Main execution -main() { - log "Starting nightly CI check for $(pwd)" - - # Change to repo directory - cd "$REPO_DIR" - - # Ensure we're in a git repository - if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - log "ERROR: Not in a git repository. Exiting." - exit 1 - fi - - # Check if gh CLI is available - if ! command -v gh >/dev/null 2>&1; then - log "ERROR: GitHub CLI (gh) not found. Please install it first." - exit 1 - fi - - # Check if claude is available - if ! command -v claude >/dev/null 2>&1; then - log "ERROR: Claude Code CLI not found. Please install it first." - exit 1 - fi - - log "Starting automated CI analysis with Claude Code..." - - # Run Claude Code with comprehensive CI analysis prompt - timeout "$CLAUDE_TIMEOUT" claude --non-interactive <<'EOF' -# Nightly CI Analysis and Auto-Fix Task - -You are running an automated nightly CI health check. Your task is to: - -1. **Analyze CI Status**: Use `gh` to check recent schedule-triggered workflow runs and identify failures -2. **Per-Failure Processing**: For EACH failed workflow, create a separate worktree and analysis: - - Use `nt cc-resolve-$(date +%Y%m%d)-` for each failure - - Find the last known-good run of the same workflow - - Pull logs and perform root cause analysis - - List PRs/commits between good and bad runs - - Assess likelihood of successful autonomous fix (0-100%) - -## Analysis Framework - -### Step 1: Get CI Status (Schedule-triggered runs only) -```bash -gh run list --limit 20 --json status,conclusion,workflowName,createdAt,headSha,url,event --jq '.[] | select(.event == "schedule")' -``` - -### Step 2: For Each Failure (Separate Worktrees) -```bash -# Create dedicated worktree for this failure -nt cc-resolve-$(date +%Y%m%d)- -cd ../cc-resolve-$(date +%Y%m%d)- - -# Get detailed run info -gh run view --json jobs,conclusion,workflowName,headSha,url -gh run view --log -gh run list --workflow= --status=success --limit 1 --json status,conclusion,workflowName,createdAt,headSha,url,event --jq '.[] | select(.event == "schedule")' -``` - -### Step 3: Root Cause Analysis -Analyze: -- Error messages in logs -- Changed files between good/bad commits -- Common failure patterns (deps, tests, build, lint) -- Environmental factors - -### Step 4: Fix Assessment & Documentation -Rate confidence (0-100%) based on: -- **High confidence (80-100%)**: Simple dependency updates, known lint fixes, obvious typos -- **Medium confidence (60-79%)**: Test failures with clear fixes, build config issues -- **Low confidence (0-59%)**: Complex logic errors, environmental issues, unclear failures - -### Step 5: Create Reports (in each worktree) -For each failure, create in `cron/reports/`: - -**Always create**: `assessment--$(date +%Y%m%d).md` -- Failure summary and root cause analysis -- Commit/PR range analysis -- Fix confidence assessment - -**If fix attempted**: `resolution--$(date +%Y%m%d).md` -- Fix implementation details -- Test results and verification -- Success/failure status - -**If fix deferred**: `status--$(date +%Y%m%d).md` -- Reasons for deferral -- Manual intervention required -- Next steps recommendation - -### Step 6: Auto-Fix Execution (if confidence >75%) -1. Implement fix in the dedicated worktree -2. Run relevant tests to verify -3. Commit with descriptive message -4. Document in resolution.md - -## Important Notes -- Each failure gets its own worktree for isolation -- Always create assessment.md, plus resolution.md OR status.md -- Conservative approach: when in doubt, defer to manual review -- Use proper git hygiene and descriptive commits - -Begin analysis now. -EOF - - local claude_exit_code=$? - - if [ $claude_exit_code -eq 0 ]; then - log "Claude Code analysis completed successfully" - elif [ $claude_exit_code -eq 124 ]; then - log "WARNING: Claude Code analysis timed out after ${CLAUDE_TIMEOUT}s" - else - log "ERROR: Claude Code analysis failed with exit code $claude_exit_code" - fi - - log "Nightly CI check completed. See full log at: $LOG_FILE" -} - -# Run main function -main "$@" \ No newline at end of file diff --git a/cron/nightly-ci-check.ts b/cron/nightly-ci-check.ts new file mode 100755 index 000000000..a5b635625 --- /dev/null +++ b/cron/nightly-ci-check.ts @@ -0,0 +1,510 @@ +#!/usr/bin/env tsx + +import { execSync, spawn } from 'child_process'; +import { writeFileSync, mkdirSync, existsSync } from 'fs'; +import { join, dirname } from 'path'; +import { fileURLToPath } from 'url'; + +// Configuration +const CONFIG = { + CLAUDE_TIMEOUT: 300000, // 5 minutes in milliseconds for testing + REPO_DIR: join(dirname(fileURLToPath(import.meta.url)), '..'), + REPORTS_DIR: '', + LOG_FILE: '', + DATE_STAMP: new Date().toISOString().slice(0, 10).replace(/-/g, '') +}; + +// Initialize paths +CONFIG.REPORTS_DIR = join(CONFIG.REPO_DIR, 'cron', 'reports'); +CONFIG.LOG_FILE = join(CONFIG.REPORTS_DIR, `nightly-ci-${CONFIG.DATE_STAMP}.log`); + +// Types +interface WorkflowRun { + databaseId: number; + workflowName: string; + conclusion: string | null; + headSha: string; + url: string; + createdAt: string; + event: string; +} + +interface FailureInfo { + runId: number; + workflowName: string; + headSha: string; + url: string; +} + +// Logging utility +function log(message: string): void { + const timestamp = new Date().toISOString().replace('T', ' ').slice(0, 19); + const logLine = `[${timestamp}] ${message}`; + + // Write to log file + try { + writeFileSync(CONFIG.LOG_FILE, logLine + '\n', { flag: 'a' }); + } catch (err) { + // If log file doesn't exist yet, create the directory and try again + mkdirSync(CONFIG.REPORTS_DIR, { recursive: true }); + writeFileSync(CONFIG.LOG_FILE, logLine + '\n', { flag: 'a' }); + } + + // Also output to stderr for real-time feedback + console.error(logLine); +} + +// Execute shell command and return output +function execCommand(command: string, options: { cwd?: string; silent?: boolean } = {}): string { + try { + const result = execSync(command, { + encoding: 'utf8', + cwd: options.cwd || CONFIG.REPO_DIR, + stdio: options.silent ? 'pipe' : ['inherit', 'pipe', 'inherit'] + }); + return result.trim(); + } catch (error: any) { + throw new Error(`Command failed: ${command}\n${error.message}`); + } +} + +// Clean workflow name for file/directory names +function cleanWorkflowName(name: string): string { + return name.replace(/[^a-zA-Z0-9]/g, '-').replace(/-+/g, '-').replace(/^-|-$/g, ''); +} + +// Get scheduled workflow failures that haven't been superseded by recent successes +async function getScheduledFailures(): Promise { + log("Checking for scheduled workflow failures..."); + + // Get recent scheduled runs + const runsJson = execCommand( + 'gh run list --limit 50 --json status,conclusion,workflowName,createdAt,headSha,url,event,databaseId', + { silent: true } + ); + + const allRuns: WorkflowRun[] = JSON.parse(runsJson); + const scheduledRuns = allRuns.filter(run => run.event === 'schedule'); + + log(`DEBUG: Found ${scheduledRuns.length} scheduled runs`); + + // Group by workflow and find most recent run per workflow + const workflowGroups = new Map(); + scheduledRuns.forEach(run => { + if (!workflowGroups.has(run.workflowName)) { + workflowGroups.set(run.workflowName, []); + } + workflowGroups.get(run.workflowName)!.push(run); + }); + + const failures: FailureInfo[] = []; + + for (const [workflowName, runs] of workflowGroups) { + // Sort by creation date (newest first) + runs.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime()); + const mostRecent = runs[0]; + + log(`DEBUG: Processing workflow: ${workflowName}`); + log(`DEBUG: Latest run has conclusion: ${mostRecent.conclusion}`); + + if (mostRecent.conclusion === 'failure' || mostRecent.conclusion === 'cancelled') { + log(`DEBUG: Adding ${workflowName} to failures list`); + failures.push({ + runId: mostRecent.databaseId, + workflowName: mostRecent.workflowName, + headSha: mostRecent.headSha, + url: mostRecent.url + }); + } + } + + log(`DEBUG: Final failures list has ${failures.length} entries`); + + if (failures.length === 0) { + log("No unresolved scheduled workflow failures found"); + return []; + } + + log(`Found ${failures.length} unresolved scheduled workflow failures`); + return failures; +} + +// Setup worktree for a specific failure +async function setupWorktreeForFailure(runId: number, workflowName: string): Promise { + const cleanWorkflow = cleanWorkflowName(workflowName); + const worktreeName = `cc-resolve-${CONFIG.DATE_STAMP}-${cleanWorkflow}`; + + log(`Setting up worktree for ${workflowName} failure (run: ${runId})`); + + try { + // Create worktree using git worktree command directly (force override if exists) + const worktreePath = join('..', worktreeName); + execCommand(`git worktree add -f "${worktreePath}"`, { silent: true }); + + // Create reports directory in worktree + mkdirSync(join(worktreePath, 'cron', 'reports'), { recursive: true }); + + log(`Created worktree: ${worktreePath}`); + return worktreePath; + } catch (error: any) { + log(`ERROR: Failed to create worktree ${worktreeName}: ${error.message}`); + throw error; + } +} + +// Collect failure data for analysis +async function collectFailureData( + runId: number, + workflowName: string, + headSha: string, + runUrl: string, + worktreePath: string +): Promise { + log(`Collecting failure data for ${workflowName} (run: ${runId})`); + + const cleanWorkflow = cleanWorkflowName(workflowName); + const reportsDir = join(worktreePath, 'cron', 'reports'); + + try { + // Get failure logs + log("Fetching failure logs..."); + const failureLogs = execCommand(`gh run view ${runId} --log`, { silent: true }); + writeFileSync(join(reportsDir, `failure-logs-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), failureLogs); + + // Get detailed run info + const runDetails = execCommand( + `gh run view ${runId} --json jobs,conclusion,workflowName,headSha,url,createdAt`, + { silent: true } + ); + writeFileSync(join(reportsDir, `failure-details-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json`), runDetails); + + // Find last successful run of same workflow + log(`Finding last successful run of ${workflowName}...`); + try { + const successfulRunsJson = execCommand( + `gh run list --workflow="${workflowName}" --status=success --limit 1 --json status,conclusion,workflowName,createdAt,headSha,url,event,databaseId`, + { silent: true } + ); + + const successfulRuns: WorkflowRun[] = JSON.parse(successfulRunsJson); + const lastGoodRun = successfulRuns.find(run => run.event === 'schedule'); + + if (lastGoodRun) { + const goodSha = lastGoodRun.headSha; + const goodRunId = lastGoodRun.databaseId; + + log(`Found last good run: ${goodRunId} (sha: ${goodSha})`); + + // Get commits between good and bad + log(`Analyzing commits between ${goodSha} and ${headSha}...`); + try { + // Basic commit range + const commitRange = execCommand( + `git log --oneline --pretty=format:'%h|%s|%an|%ad' --date=short ${goodSha}..${headSha}`, + { silent: true } + ); + writeFileSync(join(reportsDir, `commit-range-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), commitRange); + + // Detailed git log with full messages + const detailedLog = execCommand( + `git log --pretty=format:'%H%n%an <%ae>%n%ad%n%s%n%n%b%n---COMMIT-END---' --date=iso ${goodSha}..${headSha}`, + { silent: true } + ); + writeFileSync(join(reportsDir, `detailed-commits-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), detailedLog); + + // Git diff summary (files changed) + const diffStat = execCommand( + `git diff --stat ${goodSha}..${headSha}`, + { silent: true } + ); + writeFileSync(join(reportsDir, `diff-stat-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), diffStat); + + // Git diff --name-status (files and change types) + const nameStatus = execCommand( + `git diff --name-status ${goodSha}..${headSha}`, + { silent: true } + ); + writeFileSync(join(reportsDir, `diff-name-status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), nameStatus); + + // Full git diff (careful - this could be large) + try { + const fullDiff = execCommand( + `git diff ${goodSha}..${headSha}`, + { silent: true } + ); + // Only write if diff is reasonable size (< 1MB) + if (fullDiff.length < 1024 * 1024) { + writeFileSync(join(reportsDir, `full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), fullDiff); + } else { + writeFileSync(join(reportsDir, `full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), + `Diff too large (${Math.round(fullDiff.length / 1024)}KB) - skipped for performance`); + } + } catch (diffError) { + writeFileSync(join(reportsDir, `full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "Could not generate full diff"); + } + + } catch (gitError) { + log("Could not get commit range - continuing without it"); + writeFileSync(join(reportsDir, `commit-range-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No commit range available"); + writeFileSync(join(reportsDir, `detailed-commits-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No detailed commits available"); + writeFileSync(join(reportsDir, `diff-stat-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No diff stat available"); + writeFileSync(join(reportsDir, `diff-name-status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No name status available"); + writeFileSync(join(reportsDir, `full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No full diff available"); + } + + // Get PR information for commits in range + const prInfoFile = join(reportsDir, `pr-info-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json`); + try { + const commits = execCommand( + `git log --pretty=format:'%H' ${goodSha}..${headSha}`, + { silent: true } + ).split('\n').filter(sha => sha.trim()); + + const prInfo = []; + for (const commitSha of commits) { + try { + const prData = execCommand( + `gh pr list --search "${commitSha}" --json number,title,author,mergedAt,url --limit 1`, + { silent: true } + ); + const prs = JSON.parse(prData); + if (prs.length > 0) { + prInfo.push({ + commit: commitSha, + pr: prs[0] + }); + } + } catch (prError) { + // Skip if can't find PR for this commit + continue; + } + } + writeFileSync(prInfoFile, JSON.stringify(prInfo, null, 2)); + } catch (prError) { + writeFileSync(prInfoFile, '[]'); + } + + writeFileSync(join(reportsDir, `last-good-run-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), `${goodSha}|${goodRunId}`); + } else { + log(`WARNING: No successful scheduled run found for ${workflowName}`); + writeFileSync(join(reportsDir, `last-good-run-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No successful run found"); + } + } catch (error) { + log(`WARNING: Could not find successful runs for ${workflowName}`); + writeFileSync(join(reportsDir, `last-good-run-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt`), "No successful run found"); + } + + // Create metadata file + const metadata = { + runId, + workflowName, + headSha, + runUrl, + analysisDate: new Date().toISOString(), + worktreePath + }; + writeFileSync(join(reportsDir, `metadata-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json`), JSON.stringify(metadata, null, 2)); + + log(`Data collection completed for ${workflowName}`); + } catch (error: any) { + log(`ERROR: Failed to collect data for ${workflowName}: ${error.message}`); + throw error; + } +} + +// Invoke Claude analysis in worktree +async function invokeClaudeAnalysis(workflowName: string, worktreePath: string): Promise { + const cleanWorkflow = cleanWorkflowName(workflowName); + log(`Invoking Claude analysis for ${workflowName} in ${worktreePath}`); + + const claudePrompt = `# Automated CI Failure Analysis + +You are analyzing a CI failure in a dedicated worktree. The failure data has been pre-collected in ./cron/reports/. + +Your task is to: +1. **Analyze the failure** using the pre-collected data: + - Read failure-logs-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for error details + - Read failure-details-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json for run metadata + - Read commit-range-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for basic commit list + - Read detailed-commits-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for full commit messages + - Read diff-stat-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for file change summary + - Read diff-name-status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for specific file changes + - Read full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for complete code changes + - Read pr-info-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json for PR context + - Read last-good-run-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for baseline reference + +2. **Perform root cause analysis** and assess fix confidence (0-100%): + - **High confidence (80-100%)**: Simple dependency updates, lint fixes, obvious typos + - **Medium confidence (60-79%)**: Test failures with clear fixes, build config issues + - **Low confidence (0-59%)**: Complex logic errors, environmental issues + +3. **Create analysis report** in ./cron/reports/: + - **Always create**: assessment-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md + - **If attempting fix**: resolution-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md + - **If deferring**: status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md + +4. **If confidence >75%**: Implement fix, test, and commit with clear message + +**Important**: Work only with the pre-collected data. Focus on analysis and solution, not data gathering. + +Begin analysis now.`; + + try { + // Write prompt to file for debugging + writeFileSync(join(worktreePath, 'claude-prompt.txt'), claudePrompt); + log(`DEBUG: Wrote Claude prompt to ${join(worktreePath, 'claude-prompt.txt')}`); + + // Use Claude in print mode (-p) which is better for programmatic usage + const claude = spawn('claude', ['-p', claudePrompt], { + cwd: worktreePath, + stdio: 'pipe', + env: process.env // Pass full environment including auth configs + }); + + let claudeOutput = ''; + let claudeError = ''; + + claude.stdout.on('data', (data) => { + const chunk = data.toString(); + claudeOutput += chunk; + log(`DEBUG: Claude stdout chunk: ${chunk.slice(0, 200)}...`); + }); + + claude.stderr.on('data', (data) => { + const chunk = data.toString(); + claudeError += chunk; + log(`DEBUG: Claude stderr chunk: ${chunk.slice(0, 200)}...`); + }); + + // Wait for Claude to complete with timeout + const result = await new Promise((resolve, reject) => { + const timeout = setTimeout(() => { + claude.kill(); + reject(new Error('Claude analysis timed out')); + }, CONFIG.CLAUDE_TIMEOUT); + + claude.on('close', (code) => { + clearTimeout(timeout); + resolve(code || 0); + }); + + claude.on('error', (error) => { + clearTimeout(timeout); + reject(error); + }); + }); + + if (result === 0) { + log(`Claude analysis completed successfully for ${workflowName}`); + return true; + } else { + log(`ERROR: Claude analysis failed for ${workflowName} with exit code ${result}`); + if (claudeError) { + log(`Claude stderr: ${claudeError}`); + } + if (claudeOutput) { + log(`Claude stdout: ${claudeOutput}`); + } + return false; + } + } catch (error: any) { + if (error.message.includes('timed out')) { + log(`WARNING: Claude analysis timed out for ${workflowName} after ${CONFIG.CLAUDE_TIMEOUT / 1000}s`); + } else { + log(`ERROR: Claude analysis failed for ${workflowName}: ${error.message}`); + } + return false; + } +} + +// Main execution +async function main(): Promise { + log(`Starting nightly CI check for ${CONFIG.REPO_DIR}`); + + try { + // Ensure we're in a git repository + execCommand('git rev-parse --is-inside-work-tree', { silent: true }); + } catch (error) { + log("ERROR: Not in a git repository. Exiting."); + process.exit(1); + } + + // Check if required commands are available + const requiredCommands = ['gh', 'claude']; + for (const cmd of requiredCommands) { + try { + execCommand(`command -v ${cmd}`, { silent: true }); + } catch (error) { + log(`ERROR: ${cmd} command not found. Please ensure it's available in PATH.`); + process.exit(1); + } + } + + // Check if git worktree is available + try { + execCommand('git worktree --help', { silent: true }); + } catch (error) { + log("ERROR: git worktree not available. Please ensure you have Git 2.5+."); + process.exit(1); + } + + log("Starting automated CI analysis..."); + + try { + // Get scheduled workflow failures + const failures = await getScheduledFailures(); + + if (failures.length === 0) { + log("No scheduled workflow failures to process. Exiting."); + return; + } + + let failureCount = 0; + let successCount = 0; + + // Process each failure + for (const failure of failures) { + failureCount++; + log(`Processing failure ${failureCount}: ${failure.workflowName} (run: ${failure.runId})`); + + try { + // Setup worktree for this failure + const worktreePath = await setupWorktreeForFailure(failure.runId, failure.workflowName); + + // Collect failure data + await collectFailureData( + failure.runId, + failure.workflowName, + failure.headSha, + failure.url, + worktreePath + ); + + // Invoke Claude analysis + const success = await invokeClaudeAnalysis(failure.workflowName, worktreePath); + if (success) { + successCount++; + log(`Successfully completed analysis for ${failure.workflowName}`); + } else { + log(`ERROR: Analysis failed for ${failure.workflowName}`); + } + } catch (error: any) { + log(`ERROR: Failed to process ${failure.workflowName}: ${error.message}`); + } + } + + log(`Nightly CI check completed. Processed ${failureCount} failures, ${successCount} successful analyses.`); + log(`See full log at: ${CONFIG.LOG_FILE}`); + } catch (error: any) { + log(`ERROR: Main execution failed: ${error.message}`); + process.exit(1); + } +} + +// Run if this is the main module +if (import.meta.url === `file://${process.argv[1]}`) { + main().catch((error) => { + console.error('Unhandled error:', error); + process.exit(1); + }); +} \ No newline at end of file From 58335d1b79113eb0bdcb6476290252e7db717323 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 10:42:15 -0300 Subject: [PATCH 06/14] udpate - park --- cron/nightly-ci-check.ts | 265 ++++++++++++++++++++++----------------- 1 file changed, 147 insertions(+), 118 deletions(-) diff --git a/cron/nightly-ci-check.ts b/cron/nightly-ci-check.ts index a5b635625..1297a4b3c 100755 --- a/cron/nightly-ci-check.ts +++ b/cron/nightly-ci-check.ts @@ -77,56 +77,61 @@ function cleanWorkflowName(name: string): string { async function getScheduledFailures(): Promise { log("Checking for scheduled workflow failures..."); - // Get recent scheduled runs - const runsJson = execCommand( - 'gh run list --limit 50 --json status,conclusion,workflowName,createdAt,headSha,url,event,databaseId', - { silent: true } - ); - - const allRuns: WorkflowRun[] = JSON.parse(runsJson); - const scheduledRuns = allRuns.filter(run => run.event === 'schedule'); - - log(`DEBUG: Found ${scheduledRuns.length} scheduled runs`); + try { + // Get recent scheduled runs + const runsJson = execCommand( + 'gh run list --limit 50 --json status,conclusion,workflowName,createdAt,headSha,url,event,databaseId', + { silent: true } + ); + + const allRuns: WorkflowRun[] = JSON.parse(runsJson); + const scheduledRuns = allRuns.filter(run => run.event === 'schedule'); - // Group by workflow and find most recent run per workflow - const workflowGroups = new Map(); - scheduledRuns.forEach(run => { - if (!workflowGroups.has(run.workflowName)) { - workflowGroups.set(run.workflowName, []); + log(`DEBUG: Found ${scheduledRuns.length} scheduled runs`); + + // Group by workflow and find most recent run per workflow + const workflowGroups = new Map(); + scheduledRuns.forEach(run => { + if (!workflowGroups.has(run.workflowName)) { + workflowGroups.set(run.workflowName, []); + } + workflowGroups.get(run.workflowName)!.push(run); + }); + + const failures: FailureInfo[] = []; + + for (const [workflowName, runs] of workflowGroups) { + // Sort by creation date (newest first) + runs.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime()); + const mostRecent = runs[0]; + + log(`DEBUG: Processing workflow: ${workflowName}`); + log(`DEBUG: Latest run has conclusion: ${mostRecent.conclusion}`); + + if (mostRecent.conclusion === 'failure' || mostRecent.conclusion === 'cancelled') { + log(`DEBUG: Adding ${workflowName} to failures list`); + failures.push({ + runId: mostRecent.databaseId, + workflowName: mostRecent.workflowName, + headSha: mostRecent.headSha, + url: mostRecent.url + }); + } } - workflowGroups.get(run.workflowName)!.push(run); - }); - - const failures: FailureInfo[] = []; - - for (const [workflowName, runs] of workflowGroups) { - // Sort by creation date (newest first) - runs.sort((a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime()); - const mostRecent = runs[0]; - log(`DEBUG: Processing workflow: ${workflowName}`); - log(`DEBUG: Latest run has conclusion: ${mostRecent.conclusion}`); + log(`DEBUG: Final failures list has ${failures.length} entries`); - if (mostRecent.conclusion === 'failure' || mostRecent.conclusion === 'cancelled') { - log(`DEBUG: Adding ${workflowName} to failures list`); - failures.push({ - runId: mostRecent.databaseId, - workflowName: mostRecent.workflowName, - headSha: mostRecent.headSha, - url: mostRecent.url - }); + if (failures.length === 0) { + log("No unresolved scheduled workflow failures found"); + return []; } - } - - log(`DEBUG: Final failures list has ${failures.length} entries`); - - if (failures.length === 0) { - log("No unresolved scheduled workflow failures found"); + + log(`Found ${failures.length} unresolved scheduled workflow failures`); + return failures; + } catch (error: any) { + log(`ERROR: Failed to get scheduled failures: ${error.message}`); return []; } - - log(`Found ${failures.length} unresolved scheduled workflow failures`); - return failures; } // Setup worktree for a specific failure @@ -139,7 +144,15 @@ async function setupWorktreeForFailure(runId: number, workflowName: string): Pro try { // Create worktree using git worktree command directly (force override if exists) const worktreePath = join('..', worktreeName); - execCommand(`git worktree add -f "${worktreePath}"`, { silent: true }); + + // Remove existing worktree if it exists + try { + execCommand(`git worktree remove -f "${worktreePath}"`, { silent: true }); + } catch (e) { + // Ignore errors if worktree doesn't exist + } + + execCommand(`git worktree add "${worktreePath}"`, { silent: true }); // Create reports directory in worktree mkdirSync(join(worktreePath, 'cron', 'reports'), { recursive: true }); @@ -317,102 +330,109 @@ async function invokeClaudeAnalysis(workflowName: string, worktreePath: string): const cleanWorkflow = cleanWorkflowName(workflowName); log(`Invoking Claude analysis for ${workflowName} in ${worktreePath}`); - const claudePrompt = `# Automated CI Failure Analysis + const claudePrompt = `# CI Failure Analysis Task + +You are in a dedicated worktree for analyzing a GitHub Actions workflow failure. + +## Your Task + +1. **Analyze the failure data** in the cron/reports/ directory +2. **Identify the root cause** of the ${workflowName} workflow failure +3. **Create a detailed assessment** report +4. **Suggest concrete fixes** if the confidence level is high +5. **Write your findings** to a file called "analysis-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md" + +## Available Data + +The cron/reports/ directory contains: +- failure-logs-*.txt: Full workflow failure logs +- failure-details-*.json: Structured failure information +- commit-range-*.txt: Commits between last success and failure +- detailed-commits-*.txt: Full commit messages and details +- diff-stat-*.txt: Files changed summary +- full-diff-*.txt: Complete code changes +- pr-info-*.json: Related pull request information +- metadata-*.json: Run metadata -You are analyzing a CI failure in a dedicated worktree. The failure data has been pre-collected in ./cron/reports/. +## Analysis Framework -Your task is to: -1. **Analyze the failure** using the pre-collected data: - - Read failure-logs-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for error details - - Read failure-details-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json for run metadata - - Read commit-range-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for basic commit list - - Read detailed-commits-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for full commit messages - - Read diff-stat-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for file change summary - - Read diff-name-status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for specific file changes - - Read full-diff-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for complete code changes - - Read pr-info-${cleanWorkflow}-${CONFIG.DATE_STAMP}.json for PR context - - Read last-good-run-${cleanWorkflow}-${CONFIG.DATE_STAMP}.txt for baseline reference +Please structure your analysis as follows: -2. **Perform root cause analysis** and assess fix confidence (0-100%): - - **High confidence (80-100%)**: Simple dependency updates, lint fixes, obvious typos - - **Medium confidence (60-79%)**: Test failures with clear fixes, build config issues - - **Low confidence (0-59%)**: Complex logic errors, environmental issues +### 1. Executive Summary +- Brief description of the failure +- Impact assessment +- Confidence level in diagnosis (High/Medium/Low) -3. **Create analysis report** in ./cron/reports/: - - **Always create**: assessment-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md - - **If attempting fix**: resolution-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md - - **If deferring**: status-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md +### 2. Root Cause Analysis +- Primary cause of failure +- Contributing factors +- Timeline of events -4. **If confidence >75%**: Implement fix, test, and commit with clear message +### 3. Code Analysis +- Specific changes that triggered the failure +- Code quality issues identified +- Test coverage gaps -**Important**: Work only with the pre-collected data. Focus on analysis and solution, not data gathering. +### 4. Recommendations +- Immediate fixes needed +- Long-term improvements +- Prevention strategies -Begin analysis now.`; +### 5. Implementation Plan +- Step-by-step fix instructions +- Testing recommendations +- Risk assessment + +Focus on actionable insights that will help prevent similar failures.`; try { + // Test write permissions in worktree + try { + writeFileSync(join(worktreePath, 'write-test.tmp'), 'test'); + execCommand(`rm -f "${join(worktreePath, 'write-test.tmp')}"`, { silent: true }); + log(`DEBUG: Write permissions confirmed in ${worktreePath}`); + } catch (permError) { + log(`WARNING: Write permission issues in ${worktreePath}: ${permError}`); + } + // Write prompt to file for debugging writeFileSync(join(worktreePath, 'claude-prompt.txt'), claudePrompt); log(`DEBUG: Wrote Claude prompt to ${join(worktreePath, 'claude-prompt.txt')}`); - // Use Claude in print mode (-p) which is better for programmatic usage - const claude = spawn('claude', ['-p', claudePrompt], { - cwd: worktreePath, - stdio: 'pipe', - env: process.env // Pass full environment including auth configs - }); - - let claudeOutput = ''; - let claudeError = ''; + // Invoke Claude for actual CI analysis + log(`Starting CI failure analysis for ${workflowName}...`); - claude.stdout.on('data', (data) => { - const chunk = data.toString(); - claudeOutput += chunk; - log(`DEBUG: Claude stdout chunk: ${chunk.slice(0, 200)}...`); - }); - - claude.stderr.on('data', (data) => { - const chunk = data.toString(); - claudeError += chunk; - log(`DEBUG: Claude stderr chunk: ${chunk.slice(0, 200)}...`); - }); - - // Wait for Claude to complete with timeout - const result = await new Promise((resolve, reject) => { - const timeout = setTimeout(() => { - claude.kill(); - reject(new Error('Claude analysis timed out')); - }, CONFIG.CLAUDE_TIMEOUT); + try { + // Use Claude CLI to analyze the failure with all collected data + const analysisResult = execSync(`claude -p "${claudePrompt.replace(/"/g, '\\"')}"`, { + encoding: 'utf8', + cwd: worktreePath, + env: { ...process.env }, + stdio: 'pipe', + timeout: CONFIG.CLAUDE_TIMEOUT + }).trim(); - claude.on('close', (code) => { - clearTimeout(timeout); - resolve(code || 0); - }); + log(`Claude analysis completed for ${workflowName}`); + log(`Analysis result length: ${analysisResult.length} characters`); + + // Write the analysis result to a file + const analysisFile = join(worktreePath, 'cron', 'reports', `claude-analysis-${cleanWorkflow}-${CONFIG.DATE_STAMP}.md`); + writeFileSync(analysisFile, analysisResult); + log(`Analysis written to: ${analysisFile}`); - claude.on('error', (error) => { - clearTimeout(timeout); - reject(error); - }); - }); - - if (result === 0) { - log(`Claude analysis completed successfully for ${workflowName}`); return true; - } else { - log(`ERROR: Claude analysis failed for ${workflowName} with exit code ${result}`); - if (claudeError) { - log(`Claude stderr: ${claudeError}`); + } catch (claudeError: any) { + log(`ERROR: Claude analysis failed: ${claudeError.message}`); + if (claudeError.stderr) { + log(`ERROR: Claude stderr: ${claudeError.stderr}`); } - if (claudeOutput) { - log(`Claude stdout: ${claudeOutput}`); + if (claudeError.stdout) { + log(`ERROR: Claude stdout: ${claudeError.stdout}`); } return false; } } catch (error: any) { - if (error.message.includes('timed out')) { - log(`WARNING: Claude analysis timed out for ${workflowName} after ${CONFIG.CLAUDE_TIMEOUT / 1000}s`); - } else { - log(`ERROR: Claude analysis failed for ${workflowName}: ${error.message}`); - } + log(`ERROR: Claude analysis failed for ${workflowName}: ${error.message}`); return false; } } @@ -495,6 +515,15 @@ async function main(): Promise { log(`Nightly CI check completed. Processed ${failureCount} failures, ${successCount} successful analyses.`); log(`See full log at: ${CONFIG.LOG_FILE}`); + + if (successCount > 0) { + log(`Analysis reports generated in worktree directories:`); + for (const failure of failures) { + const cleanWorkflow = cleanWorkflowName(failure.workflowName); + const worktreePath = join('..', `cc-resolve-${CONFIG.DATE_STAMP}-${cleanWorkflow}`); + log(` - ${worktreePath}/cron/reports/`); + } + } } catch (error: any) { log(`ERROR: Main execution failed: ${error.message}`); process.exit(1); From b9bcdc3d8788e7db30ab3fa96f5e0a6f25d0b0e4 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 11:06:14 -0300 Subject: [PATCH 07/14] do dev-time installations 'at once' --- packages/cli/package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/cli/package.json b/packages/cli/package.json index 14cbf845b..ca5943550 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -27,7 +27,7 @@ "lint": "npx eslint .", "lint:fix": "npx eslint . --fix", "lint:check": "npx eslint . --max-warnings 0", - "try:init": "node dist/cli.js init testproject --dangerously-clobber --no-interactive --data-layer static --import-course-data --import-server-url http://localhost:5984 --import-username admin --import-password password --import-course-ids 2aeb8315ef78f3e89ca386992d00825b && cd testproject && npm i && npm install --save-dev @vue-skuilder/cli@file:.. && npm install @vue-skuilder/db@file:../../db && npm install @vue-skuilder/courses@file:../../courses && npm install @vue-skuilder/common-ui@file:../../common-ui" + "try:init": "node dist/cli.js init testproject --dangerously-clobber --no-interactive --data-layer static --import-course-data --import-server-url http://localhost:5984 --import-username admin --import-password password --import-course-ids 2aeb8315ef78f3e89ca386992d00825b && cd testproject && npm i && npm install --save-dev @vue-skuilder/cli@file:.. && npm install @vue-skuilder/db@file:../../db @vue-skuilder/courses@file:../../courses @vue-skuilder/common-ui@file:../../common-ui" }, "keywords": [ "cli", From 7ca8ae238d38d4594922874afa77781962da7c97 Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 12:45:55 -0300 Subject: [PATCH 08/14] implement static getCourseTagStubs --- packages/db/src/impl/static/courseDB.ts | 80 +++++++++++++++++++++---- 1 file changed, 70 insertions(+), 10 deletions(-) diff --git a/packages/db/src/impl/static/courseDB.ts b/packages/db/src/impl/static/courseDB.ts index 38ab595eb..9ef5230e5 100644 --- a/packages/db/src/impl/static/courseDB.ts +++ b/packages/db/src/impl/static/courseDB.ts @@ -15,6 +15,7 @@ import { DataLayerResult } from '../../core/types/db'; import { ContentNavigationStrategyData } from '../../core/types/contentNavigationStrategy'; import { ScheduledCard } from '../../core/types/user'; import { Navigators } from '../../core/navigators'; +import { logger } from '../../util/logger'; export class StaticCourseDB implements CourseDBInterface { constructor( @@ -41,10 +42,9 @@ export class StaticCourseDB implements CourseDBInterface { } async getCourseInfo(): Promise { - // This would need to be pre-computed in the manifest return { - cardCount: 0, // Would come from manifest - registeredUsers: 0, + cardCount: this.manifest.documentCount || 0, + registeredUsers: 0, // Always 0 in static mode }; } @@ -160,6 +160,7 @@ export class StaticCourseDB implements CourseDBInterface { async getAppliedTags(_cardId: string): Promise> { // Would need to query the tag index + logger.warn(`getAppliedTags not implemented`); return { total_rows: 0, offset: 0, @@ -188,12 +189,71 @@ export class StaticCourseDB implements CourseDBInterface { } async getCourseTagStubs(): Promise> { - // Would query all tag documents - return { - total_rows: 0, - offset: 0, - rows: [], - }; + try { + const tagsIndex = await this.unpacker.getTagsIndex(); + + if (!tagsIndex || !tagsIndex.byTag) { + logger.warn('Tags index not found or empty'); + return { + total_rows: 0, + offset: 0, + rows: [], + }; + } + + // Create tag stubs from the index + const tagNames = Object.keys(tagsIndex.byTag); + const rows = await Promise.all( + tagNames.map(async (tagName) => { + const cardIds = tagsIndex.byTag[tagName] || []; + const tagId = `${DocType.TAG}-${tagName}`; + + try { + // Try to get the full tag document + const tagDoc = await this.unpacker.getDocument(tagId); + return { + id: tagId, + key: tagId, + value: { rev: '1-static' }, + doc: tagDoc, + }; + } catch (error) { + // If tag document not found, create a minimal stub + logger.warn(`Tag document not found for ${tagName}, creating stub`); + const stubDoc = { + _id: tagId, + _rev: '1-static', + course: this.courseId, + docType: DocType.TAG, + name: tagName, + snippet: `Tag: ${tagName}`, + wiki: '', + taggedCards: cardIds, + author: 'system', + }; + return { + id: tagId, + key: tagId, + value: { rev: '1-static' }, + doc: stubDoc, + }; + } + }) + ); + + return { + total_rows: rows.length, + offset: 0, + rows, + }; + } catch (error) { + logger.error('Failed to get course tag stubs:', error); + return { + total_rows: 0, + offset: 0, + rows: [], + }; + } } async addNote( @@ -256,7 +316,7 @@ export class StaticCourseDB implements CourseDBInterface { } // Attachment helper methods (internal use, not part of interface) - + /** * Get attachment URL for a document and attachment name * Internal helper method for static attachment serving From 7ab7d829c7ca973b517afe9e5da351cd10e74f8d Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 12:50:46 -0300 Subject: [PATCH 09/14] count cards, not all docs, for cardCount --- packages/db/src/impl/static/courseDB.ts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/packages/db/src/impl/static/courseDB.ts b/packages/db/src/impl/static/courseDB.ts index 9ef5230e5..b4cf47f09 100644 --- a/packages/db/src/impl/static/courseDB.ts +++ b/packages/db/src/impl/static/courseDB.ts @@ -42,8 +42,14 @@ export class StaticCourseDB implements CourseDBInterface { } async getCourseInfo(): Promise { + // Count only cards, not all documents + // Use chunks metadata to count card documents specifically + const cardCount = this.manifest.chunks + .filter(chunk => chunk.docType === DocType.CARD) + .reduce((total, chunk) => total + chunk.documentCount, 0); + return { - cardCount: this.manifest.documentCount || 0, + cardCount, registeredUsers: 0, // Always 0 in static mode }; } From 08f5caaf1eb8230957208a4fee1d8f5439c3645f Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 13:30:16 -0300 Subject: [PATCH 10/14] rm old working doc --- agent/a.1.assessment.md | 211 ---------------------------------------- 1 file changed, 211 deletions(-) delete mode 100644 agent/a.1.assessment.md diff --git a/agent/a.1.assessment.md b/agent/a.1.assessment.md deleted file mode 100644 index 869cbf5c5..000000000 --- a/agent/a.1.assessment.md +++ /dev/null @@ -1,211 +0,0 @@ -# Assessment: CLI Studio Express Integration - -## Current State Analysis - -### How CLI Currently Handles Studio-UI - -The CLI's `studio` command currently: - -1. **Bundled Static Assets**: Studio-UI is built as a static Vue.js app and bundled into the CLI package - - Built via `npm run build:studio-ui` in CLI build process - - Assets copied to `dist/studio-ui-assets/` via `embed:studio-ui` script - - Served via Node.js `serve-static` middleware with dynamic config injection - -2. **Process Management**: CLI manages processes directly in Node.js - - **CouchDB**: Uses `CouchDBManager` class to spawn Docker containers - - **Studio-UI Server**: Creates HTTP server using Node.js `http` module - - **Process Lifecycle**: Handles graceful shutdown via SIGINT/SIGTERM handlers - -3. **Configuration Injection**: Dynamic config injection for studio-ui - - Injects CouchDB connection details into `window.STUDIO_CONFIG` - - Modifies `index.html` at runtime with database connection info - - Uses SPA fallback routing for client-side routing - -### Express Backend Architecture - -The Express backend (`@vue-skuilder/express`) is: - -1. **Standalone Service**: Designed as independent Node.js/Express application - - Main entry: `src/app.ts` - - Hardcoded port: 3000 - - Manages own CouchDB connections via `nano` client - - Handles authentication, course management, classroom operations - -2. **External Dependencies**: Requires external CouchDB instance - - Connects to CouchDB via environment configuration - - Manages multiple databases (courses, classrooms, users) - - Includes its own initialization and setup logic - -3. **Heavyweight Service**: Full-featured API server - - Authentication middleware - - File upload processing - - Complex business logic for course/classroom management - - Logging and error handling - -## Integration Options Analysis - -### Option A: Bundle Express and Run as Subprocess - -**Approach**: Bundle express into CLI and spawn it as a child process - -**Pros**: -- Clean separation of concerns -- Express runs in its own process space -- Can leverage existing Express configuration -- Easy to manage process lifecycle (start/stop) -- Familiar process management pattern (similar to CouchDB) - -**Cons**: -- Requires bundling entire Express app with CLI -- Multiple Node.js processes running -- More complex communication between CLI and Express -- Harder to pass configuration dynamically -- Potential port conflicts - -**Technical Implementation**: -```typescript -// Similar to how CLI spawns CouchDB -const expressProcess = spawn('node', [expressDistPath], { - env: { ...process.env, COUCHDB_URL: couchUrl } -}); -``` - -### Option B: Import Express Directly (Same Process) - -**Approach**: Import Express app and run it in the same Node.js process as CLI - -**Pros**: -- Single process - more efficient resource usage -- Direct communication between CLI and Express -- Easy to pass configuration objects -- Simpler deployment (single Node.js process) -- Can share CouchDB connection instances - -**Cons**: -- Tight coupling between CLI and Express -- Harder to isolate Express errors from CLI -- Express initialization could block CLI startup -- More complex to handle Express-specific configuration -- Potential conflicts with CLI's HTTP server - -**Technical Implementation**: -```typescript -// Import Express app and configure it -import { createExpressApp } from '@vue-skuilder/express'; -const expressApp = createExpressApp(couchConfig); -expressApp.listen(3000); -``` - -### Option C: Expect Express Running Separately - -**Approach**: CLI expects Express to be running as separate service - -**Pros**: -- Complete separation of concerns -- Express can be managed independently -- No changes needed to CLI architecture -- Easy to scale Express separately -- Clear service boundaries - -**Cons**: -- Additional setup complexity for users -- Need to coordinate between multiple services -- User must manage Express lifecycle manually -- Harder to provide "one-command" studio experience -- Complex error handling when Express is down - -**Technical Implementation**: -```typescript -// CLI just checks if Express is available -const expressHealthCheck = await fetch('http://localhost:3000/health'); -if (!expressHealthCheck.ok) { - throw new Error('Express server not running'); -} -``` - -### Option D: Hybrid Approach - Express Module - -**Approach**: Refactor Express into a configurable module that CLI can import and control - -**Pros**: -- Best of both worlds - modularity with integration -- CLI maintains control over process lifecycle -- Express can be configured per CLI session -- Clean API boundaries -- Reusable Express module - -**Cons**: -- Requires significant refactoring of Express package -- Breaking changes to Express architecture -- More complex implementation -- Need to maintain backward compatibility - -**Technical Implementation**: -```typescript -// Express as configurable module -import { ExpressService } from '@vue-skuilder/express'; -const expressService = new ExpressService({ - port: 3001, - couchdb: couchConfig, - logger: cliLogger -}); -await expressService.start(); -``` - -## Key Considerations - -### 1. **Process Management Consistency** -- CLI already manages CouchDB via subprocess (Docker) -- Studio-UI runs as HTTP server within CLI process -- Express subprocess would follow CouchDB pattern - -### 2. **Configuration Management** -- CLI injects config into Studio-UI at runtime -- Express needs CouchDB connection details -- Studio-UI needs to know Express endpoint - -### 3. **Port Management** -- CLI finds available ports dynamically (Studio-UI: 7174+) -- Express hardcoded to port 3000 -- Need to avoid port conflicts - -### 4. **Error Handling & Lifecycle** -- CLI handles graceful shutdown for all services -- Express needs to integrate with CLI's process management -- Studio-UI depends on both CouchDB and Express - -### 5. **User Experience** -- Current: Single `skuilder studio` command starts everything -- Goal: Maintain single-command simplicity -- Express adds complexity but provides powerful features - -## Recommendation - -**Option A: Bundle Express and Run as Subprocess** is the best approach because: - -1. **Architectural Consistency**: Matches existing CouchDB subprocess pattern -2. **Clean Separation**: Express runs independently but managed by CLI -3. **Minimal Changes**: Can reuse existing Express code with minimal refactoring -4. **Process Management**: Leverages CLI's existing process lifecycle handling -5. **Configuration**: Can pass config via environment variables (established pattern) - -### Implementation Plan - -1. **Express Modifications**: - - Make port configurable via environment variable - - Add health check endpoint - - Ensure clean shutdown on SIGTERM/SIGINT - -2. **CLI Integration**: - - Add Express process management (similar to CouchDB) - - Bundle Express dist in CLI build process - - Dynamic port allocation for Express - - Update Studio-UI config injection to include Express endpoint - -3. **Process Orchestration**: - - Start CouchDB first (as currently done) - - Start Express with CouchDB connection details - - Start Studio-UI with both CouchDB and Express endpoints - - Coordinate shutdown of all services - -This approach maintains the current architecture's clarity while adding the powerful Express backend capabilities that users need for full studio functionality. \ No newline at end of file From 0418f681ec32e12f584d90928289e860ae1f75cf Mon Sep 17 00:00:00 2001 From: NiloCK Date: Mon, 7 Jul 2025 14:17:01 -0300 Subject: [PATCH 11/14] add lookupg for ELO data from staticDB... prior elo display in cardbrowser relied on enriched cardID data [courseid]-[cardid]-[elo]. fetching mechanism is different in scaffolded / statically packed courses, so fallback here to a getter from the datalayerprovider --- .../src/components/CourseCardBrowser.vue | 66 +++++++++---------- 1 file changed, 31 insertions(+), 35 deletions(-) diff --git a/packages/common-ui/src/components/CourseCardBrowser.vue b/packages/common-ui/src/components/CourseCardBrowser.vue index 3268b6fae..d2ffd4aca 100644 --- a/packages/common-ui/src/components/CourseCardBrowser.vue +++ b/packages/common-ui/src/components/CourseCardBrowser.vue @@ -32,7 +32,7 @@ {{ cardPreview[c.id] }} - {{ c.id.split('-').length === 3 ? c.id.split('-')[2] : '' }} + ELO: {{ cardElos[idParse(c.id)]?.global.score || '(unknown)' }} @@ -118,7 +118,7 @@