From 423bd186d17d81babadecbedc853463f5d97d182 Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 11:55:12 +0000 Subject: [PATCH 1/6] docs: add LangChain issue intake enhancement proposal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Explores using LangChain to improve the Agents 63 issue intake pipeline: 1. Human Language → AGENT_ISSUE_TEMPLATE conversion (P1) 2. Contextual data injection for PRs (P2) 3. Agent capability pre-flight check (P0) - validates tasks are agent-actionable 4. Analyze → Approve → Format hybrid optimization (P1) - stateless two-phase flow Key insight: #4 uses label-based approval (agents:optimize → agents:apply-suggestions) instead of stateful multi-turn conversation, reducing complexity from 5-7d to 2-3d while reusing the Formatter (#1) infrastructure. Also identifies additional opportunities: - Task decomposition for large tasks - Duplicate/related issue detection - Post-merge learning feedback --- docs/plans/langchain-issue-intake-proposal.md | 606 ++++++++++++++++++ 1 file changed, 606 insertions(+) create mode 100644 docs/plans/langchain-issue-intake-proposal.md diff --git a/docs/plans/langchain-issue-intake-proposal.md b/docs/plans/langchain-issue-intake-proposal.md new file mode 100644 index 000000000..8c2305d32 --- /dev/null +++ b/docs/plans/langchain-issue-intake-proposal.md @@ -0,0 +1,606 @@ +# LangChain Issue Intake Enhancement Proposal + +> **Status**: Proposal +> **Created**: 2026-01-03 +> **Related**: [langchain-keepalive-integration.md](./langchain-keepalive-integration.md) +> **Target Workflows**: `agents-63-issue-intake.yml`, `reusable-agents-issue-bridge.yml` + +--- + +## Executive Summary + +This proposal explores how LangChain can enhance the Agents 63 issue intake pipeline to improve issue quality, reduce wasted agent iterations, and provide better human-agent collaboration. We focus on **high-probability, scoped improvements** rather than speculative features. + +--- + +## Current Architecture + +``` +Human describes problem → Manual formatting → Issue labeled → agents-63-issue-intake.yml + ↓ + reusable-agents-issue-bridge.yml + ↓ + Creates PR with agent:codex + agents:keepalive + ↓ + Keepalive loop runs until tasks complete (or stuck) +``` + +**Pain Points Identified:** +1. Manual formatting into AGENT_ISSUE_TEMPLATE is tedious and error-prone +2. Tasks/criteria that agents can't address waste iterations +3. No pre-flight validation catches problems before agent engagement +4. Human-agent iteration for issue refinement is clunky + +--- + +## Proposed Enhancements + +### 1. Human Language → AGENT_ISSUE_TEMPLATE Conversion + +**Use Case**: User describes a problem in natural language, LLM structures it into the canonical format. + +**Technical Approach**: +```python +from langchain_openai import ChatOpenAI +from langchain_core.prompts import ChatPromptTemplate + +ISSUE_FORMATTER_PROMPT = """ +You are an expert at formatting GitHub issues for the Codex agent pipeline. + +Convert this human description into the AGENT_ISSUE_TEMPLATE format: + +{human_description} + +Required sections: +- ## Why - Motivation and context +- ## Scope - What this issue covers +- ## Non-Goals - What is explicitly excluded +- ## Tasks - Actionable checklist items (ONLY use bullets for actual work items) +- ## Acceptance Criteria - Verifiable completion conditions +- ## Implementation Notes - Technical guidance (optional) + +CRITICAL RULES: +1. Tasks must be specific, verifiable, and completable by an agent +2. Each task should be small enough for one iteration (~10 minutes) +3. DO NOT include bullets for instructions or notes - only actionable items +4. Acceptance criteria must be objectively verifiable +5. Include relevant file paths if mentioned + +Output the formatted issue in Markdown. +""" + +llm = ChatOpenAI( + model="gpt-4o-mini", + base_url="https://models.inference.ai.azure.com", + api_key=os.environ["GITHUB_TOKEN"], +) + +chain = ChatPromptTemplate.from_template(ISSUE_FORMATTER_PROMPT) | llm +``` + +**Trigger Mechanism**: +- Label `agents:format` on a raw issue +- Workflow parses issue body, calls LLM, updates issue with formatted version +- Adds `agents:formatted` label and removes `agents:format` + +**Plausibility**: ⭐⭐⭐⭐ HIGH +- Simple prompt engineering, proven pattern +- Single LLM call, fast feedback +- Clear input/output contract + +**Scope**: ~2-3 days development +- Add formatting chain to `scripts/` (Python, LangChain) +- Add workflow step to `agents-63-issue-intake.yml` +- Tests for common formatting scenarios + +--- + +### 2. Contextual Data Injection for PRs + +**Use Case**: Add relevant context to PRs that doesn't fit Tasks/Acceptance (e.g., related issues, design decisions, constraints). + +**Technical Approach**: +```python +CONTEXT_EXTRACTOR_PROMPT = """ +From this issue and related discussion, extract: +1. Design constraints or decisions made +2. Related issues/PRs that provide context +3. External references (docs, APIs, specifications) +4. Known blockers or dependencies + +Format as a "## Context for Agent" section that provides helpful background +without creating actionable tasks. +""" +``` + +**Integration Point**: +- Run during PR creation in `reusable-agents-issue-bridge.yml` +- Insert extracted context into PR body after Scope but before Tasks +- Preserves in `...` markers + +**Plausibility**: ⭐⭐⭐ MEDIUM-HIGH +- Straightforward extraction task +- Value depends on issue quality +- May produce low-signal output for sparse issues + +**Scope**: ~1-2 days +- Add context extraction chain +- Modify `agents_pr_meta_update_body.js` to include context section +- Optional: fetch linked issue comments for richer context + +--- + +### 3. Agent Capability Pre-Flight Check (HIGH VALUE) + +**Use Case**: Before engaging the keepalive pipeline, validate that tasks are actionable by the agent. + +**The Problem**: +Agents waste iterations on tasks they fundamentally cannot complete: + +| Task Type | Agent Capability | Example | +|-----------|------------------|---------| +| Code changes | ✅ CAN DO | "Add unit tests for X" | +| CI must pass | ⚠️ PARTIAL | Can fix code, cannot retry CI | +| Workflow files | ❌ CANNOT DO | "Update CI workflow" (protected) | +| Repo settings | ❌ CANNOT DO | "Enable branch protection" | +| External services | ❌ CANNOT DO | "Configure AWS credentials" | +| Human decisions | ❌ CANNOT DO | "Decide on API design" | +| Coverage targets | ⚠️ PARTIAL | Can add tests, cannot guarantee % | + +**Technical Approach**: +```python +AGENT_CAPABILITY_CHECK_PROMPT = """ +Analyze these tasks and acceptance criteria for agent compatibility. + +Tasks: +{tasks} + +Acceptance Criteria: +{acceptance} + +For each item, classify as: +- ACTIONABLE: Agent can directly complete this +- PARTIAL: Agent can contribute but may not fully satisfy +- BLOCKED: Agent cannot complete this (explain why) + +Known agent limitations: +- Cannot modify protected workflow files (.github/workflows/*.yml) +- Cannot change repository settings (branch protection, secrets, etc.) +- Cannot interact with external services requiring credentials +- Cannot make subjective design decisions requiring human input +- Cannot guarantee specific coverage percentages (can add tests, coverage varies) +- Cannot retry CI/CD pipelines - only fix code and push + +Output JSON: +{ + "actionable_tasks": [...], + "partial_tasks": [{"task": "...", "limitation": "..."}], + "blocked_tasks": [{"task": "...", "reason": "...", "suggested_action": "..."}], + "recommendation": "PROCEED|REVIEW_NEEDED|BLOCKED", + "human_actions_needed": [...] +} +""" +``` + +**Integration Points**: +1. **Pre-bridge check** (`agents-63-issue-intake.yml`): + - Before creating PR, run capability check + - If BLOCKED tasks exist, comment on issue with summary + - Add `agents:review-needed` label instead of `agent:codex` + +2. **Task filtering** (`reusable-agents-issue-bridge.yml`): + - Move blocked tasks to "## Deferred Tasks (Requires Human)" section + - Keep only actionable tasks in main Tasks section + - Preserve blocked items for visibility without burning agent iterations + +3. **Summary comment**: + - Post comment explaining: "X tasks are agent-compatible, Y tasks require human action" + - Include specific guidance for human actions + +**Plausibility**: ⭐⭐⭐⭐⭐ VERY HIGH +- Directly addresses #1 pain point (wasted iterations) +- Clear, objective criteria for classification +- High signal-to-noise ratio in output + +**Scope**: ~3-4 days +- Capability classification chain + tests +- Workflow integration with label handling +- Comment formatter for human guidance +- Update `issue_scope_parser.js` to handle deferred section + +--- + +### 4. Analyze → Approve → Format (Hybrid Issue Optimization) + +**Use Case**: Human labels issue with `agents:optimize` to get LLM suggestions, reviews them, then approves application via a second label. + +**Design Philosophy**: Eliminates complex stateful conversation in favor of a simple two-phase flow that reuses the Formatter (#1). + +**Flow**: +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Phase 1: Analysis (label: agents:optimize) │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Human labels issue with `agents:optimize` │ +│ ↓ │ +│ LLM analyzes issue and posts comment: │ +│ ┌────────────────────────────────────────────────────────────┐ │ +│ │ ## Issue Optimization Suggestions │ │ +│ │ │ │ +│ │ **Tasks:** │ │ +│ │ - ⚠️ Task 3 is too broad - split into: X, Y, Z │ │ +│ │ - ❌ Task 5 requires workflow changes (agent can't do) │ │ +│ │ │ │ +│ │ **Acceptance Criteria:** │ │ +│ │ - ⚠️ "Code is clean" is subjective → suggest "ruff passes" │ │ +│ │ │ │ +│ │ **Missing:** │ │ +│ │ - No file paths mentioned │ │ +│ │ - Scope section is empty │ │ +│ │ │ │ +│ │ **To apply:** Add label `agents:apply-suggestions` │ │ +│ │ **To reject:** Remove `agents:optimize` label │ │ +│ │ │ │ +│ └────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────┘ + ↓ + Human reviews suggestions + (can ask questions in comments - LLM responds informatively) + ↓ +┌─────────────────────────────────────────────────────────────────┐ +│ Phase 2: Approval (label: agents:apply-suggestions) │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ System detects approval label │ +│ ↓ │ +│ Extracts suggestions from comment JSON │ +│ ↓ │ +│ Routes to Formatter (#1) with: │ +│ - Original issue body │ +│ - Approved suggestions │ +│ - Instruction: "Apply these improvements" │ +│ ↓ │ +│ Formatter produces optimized issue body │ +│ ↓ │ +│ Updates issue + removes both labels + adds `agents:formatted` │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +**Why This Works Better Than Multi-Turn Conversation**: + +| Aspect | Original Approach | Hybrid Approach | +|--------|-------------------|-----------------| +| State management | LangGraph state machine across runs | Stateless - suggestions in comment JSON | +| Human approval | Implicit in conversation ("apply") | Explicit label = clear signal | +| Implementation | Custom conversation handler | Reuses Formatter (#1) | +| Complexity | 5-7 days | **2-3 days** | +| Human questions | Must track conversation state | Free-form - no state needed | + +**Technical Approach**: +```python +# Phase 1: Analyzer +ANALYZE_ISSUE_PROMPT = """ +Analyze this issue for agent compatibility and formatting quality. + +Issue body: +{issue_body} + +Identify: +1. Tasks that are too broad (should be split) +2. Tasks the agent cannot complete (with reasons from AGENT_LIMITATIONS) +3. Subjective acceptance criteria (suggest objective alternatives) +4. Missing sections (scope, implementation notes, file paths) +5. Formatting issues (bullets used for non-tasks, etc.) + +AGENT_LIMITATIONS: +- Cannot modify .github/workflows/*.yml (protected) +- Cannot change repository settings +- Cannot guarantee specific coverage percentages +- Cannot make subjective design decisions +- Cannot retry CI pipelines + +Output JSON with structured suggestions. +""" + +# Phase 2: Apply - calls Formatter (#1) with suggestions context +APPLY_SUGGESTIONS_PROMPT = """ +Reformat this issue applying the approved suggestions. + +Original issue: +{original_body} + +Approved suggestions: +{suggestions_json} + +Apply ALL suggestions and output the complete reformatted issue +following AGENT_ISSUE_TEMPLATE structure. Move blocked tasks to +a "## Deferred Tasks (Requires Human)" section. +""" +``` + +**Workflow Integration**: +```yaml +# In agents-63-issue-intake.yml + +# Triggered by: labeled with agents:optimize +analyze_for_optimization: + if: github.event.action == 'labeled' && github.event.label.name == 'agents:optimize' + steps: + - name: Analyze issue and post suggestions + uses: actions/github-script@v7 + with: + script: | + const { analyzeIssue } = require('./scripts/langchain/issue_optimizer.py'); + const suggestions = await analyzeIssue({ + issueBody: context.payload.issue.body, + issueNumber: context.payload.issue.number + }); + // Post comment with suggestions + embedded JSON + +# Triggered by: labeled with agents:apply-suggestions +apply_optimization: + if: github.event.action == 'labeled' && github.event.label.name == 'agents:apply-suggestions' + steps: + - name: Extract suggestions from analysis comment + id: extract + # Find comment with marker + + - name: Route to formatter with suggestions + # Calls Formatter (#1) with original body + suggestions + + - name: Update issue body + - name: Clean up labels (remove optimize, apply-suggestions; add formatted) +``` + +**Human Questions Are Free**: +If human comments with a question after Phase 1, an optional responder job can: +- Answer informatively about the suggestions +- NOT update any state +- NOT require approval tracking + +This keeps the complexity low while allowing natural interaction. + +**Plausibility**: ⭐⭐⭐⭐ HIGH +- Stateless design eliminates complexity +- Reuses Formatter (#1) infrastructure +- Clear approval signal via label +- No conversation state to manage + +**Scope**: ~2-3 days +- Issue analyzer chain (new) +- Suggestion comment formatter +- Apply job that extracts JSON and calls Formatter +- Label management + +--- + +## Additional High-Value Enhancements + +### 5. Post-Merge Learning Feedback + +**Use Case**: After PR merges, capture what worked/didn't for future issue formatting. + +**Approach**: +- Track: iterations used, tasks completed vs. stuck, human interventions +- Feed back into formatting prompts: "Issues like X typically need Y structure" +- Build corpus of successful issue patterns + +**Plausibility**: ⭐⭐⭐ MEDIUM-HIGH +**Scope**: ~2-3 days (data collection), ongoing refinement + +### 6. Duplicate/Related Issue Detection + +**Use Case**: Before creating new issue, check if similar work exists. + +**Approach**: +- Embed issue description, compare to existing open issues +- Warn if high similarity detected +- Link related issues for context + +**Plausibility**: ⭐⭐⭐⭐ HIGH (embeddings are well-understood) +**Scope**: ~2 days + +### 7. Automatic Task Decomposition + +**Use Case**: Large tasks get automatically split into smaller, iteration-sized pieces. + +**Approach**: +```python +TASK_DECOMPOSITION_PROMPT = """ +This task is too large for a single agent iteration (~10 minutes): + +{large_task} + +Decompose into smaller, independently verifiable sub-tasks. +Each sub-task should: +- Be completable in one iteration +- Have a clear verification condition +- Not depend on un-merged work from other sub-tasks +""" +``` + +**Plausibility**: ⭐⭐⭐ MEDIUM-HIGH +**Scope**: ~1-2 days + +--- + +## Implementation Priority Matrix + +| Enhancement | Value | Effort | Priority | +|-------------|-------|--------|----------| +| 3. Agent Capability Pre-Flight | ⭐⭐⭐⭐⭐ | 3-4d | **P0 - Do First** | +| 1. Human → Template Conversion | ⭐⭐⭐⭐ | 2-3d | **P1** | +| 4. Analyze → Approve → Format | ⭐⭐⭐⭐ | 2-3d | **P1** (reuses #1) | +| 7. Task Decomposition | ⭐⭐⭐ | 1-2d | **P2** | +| 6. Duplicate Detection | ⭐⭐⭐⭐ | 2d | **P2** | +| 2. Context Injection | ⭐⭐⭐ | 1-2d | **P2** | +| 5. Learning Feedback | ⭐⭐⭐ | 2-3d | **P3** | + +**Note**: #4 moved from P3 to P1 because the hybrid design reuses #1's Formatter infrastructure and eliminates state management complexity. + +--- + +## Technical Architecture + +### Shared Infrastructure + +``` +scripts/ + langchain/ + __init__.py + llm_factory.py # GitHub Models + fallback providers + issue_formatter.py # Human → template conversion (#1) + issue_optimizer.py # Analyze + apply suggestions (#4) + capability_check.py # Agent limitation analysis (#3) + task_decomposer.py # Large task splitting (#7) + prompts/ + format_issue.md + analyze_issue.md + apply_suggestions.md + check_capability.md + decompose_task.md +``` + +### Component Relationships + +``` + ┌─────────────────────┐ + │ llm_factory.py │ + │ (GitHub Models + │ + │ fallback chain) │ + └─────────┬───────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ capability_ │ │ issue_ │ │ issue_ │ +│ check.py (#3) │ │ formatter.py(#1)│◄──│ optimizer.py(#4)│ +│ │ │ │ │ │ +│ Pre-flight │ │ Raw text → │ │ Phase 1: Analyze│ +│ validation │ │ Template format │ │ Phase 2: Apply │ +└───────────────┘ └─────────────────┘ │ (calls #1) │ + └─────────────────┘ +``` + +### Workflow Integration Points + +```yaml +# agents-63-issue-intake.yml additions + +jobs: + # NEW: Optimize flow (Phase 1) + analyze_for_optimization: + if: github.event.action == 'labeled' && github.event.label.name == 'agents:optimize' + steps: + - name: Analyze issue + # Posts suggestion comment with embedded JSON + + # NEW: Optimize flow (Phase 2) + apply_optimization: + if: github.event.action == 'labeled' && github.event.label.name == 'agents:apply-suggestions' + steps: + - name: Extract suggestions from comment + - name: Route to formatter with suggestions + - name: Update issue body + - name: Clean up labels + + # NEW: Format flow + format_issue: + if: github.event.action == 'labeled' && github.event.label.name == 'agents:format' + steps: + - name: Format raw issue into template + - name: Update issue body + - name: Swap labels (format → formatted) + + # EXISTING: Pre-flight check before bridge + preprocess: + name: LLM Pre-processing + steps: + - name: Check agent capability + id: capability + uses: actions/github-script@v7 + with: + script: | + const { checkAgentCapability } = require('./scripts/langchain/capability_check.py'); + const result = await checkAgentCapability({ + tasks: '${{ steps.parse.outputs.tasks }}', + acceptance: '${{ steps.parse.outputs.acceptance }}' + }); + core.setOutput('recommendation', result.recommendation); + core.setOutput('blocked_tasks', JSON.stringify(result.blocked_tasks)); + + - name: Post capability summary + if: steps.capability.outputs.recommendation != 'PROCEED' + uses: actions/github-script@v7 + with: + script: | + // Post comment with blocked task analysis + // Add agents:review-needed label +``` + +--- + +## Risk Assessment + +| Risk | Mitigation | +|------|------------| +| LLM produces poor formatting | Template validation + human review before agent engagement | +| False positives in capability check | Conservative defaults (flag uncertain items for review) | +| API rate limits | Use existing GitHub Models quota, add exponential backoff | +| Increased workflow complexity | Modular design, feature flags for gradual rollout | +| LLM hallucinations in task decomposition | Preserve original task, show decomposition as suggestions | + +--- + +## Success Metrics + +| Metric | Current | Target | How to Measure | +|--------|---------|--------|----------------| +| Agent iterations per PR | ~5 avg | ~3 avg | Keepalive metrics | +| PRs stuck on blocked tasks | ~20% | <5% | Track `agents:review-needed` vs successful merges | +| Issue formatting time | ~15 min | ~2 min | User survey | +| False positive blocked tasks | N/A | <10% | Manual review sample | + +--- + +## Next Steps + +1. **Proof of Concept**: Implement capability check (#3) as standalone script +2. **Validate**: Run against 10 recent issues, measure accuracy +3. **Integrate**: Add to `agents-63-issue-intake.yml` behind feature flag +4. **Iterate**: Refine prompts based on real-world results +5. **Expand**: Add human→template conversion (#1) using same infrastructure + +--- + +## Appendix: Known Agent Limitations Reference + +For inclusion in capability check prompts: + +``` +AGENT CANNOT: +- Modify workflow files in .github/workflows/ (protected by agents-guard) +- Change repository settings (branch protection, secrets, webhooks) +- Access external services requiring credentials not in environment +- Run commands requiring interactive input +- Make subjective design decisions (API design, architecture choices) +- Guarantee specific test coverage percentages +- Retry CI pipelines - can only fix code and push + +AGENT CAN (with limitations): +- Add tests (but coverage depends on test quality) +- Fix lint/format issues (but may introduce new ones) +- Update documentation (but may not match current state) +- Refactor code (but large refactors may exceed iteration time) + +AGENT CAN FULLY: +- Add/modify Python/JS/YAML files (except protected workflows) +- Create new test files +- Update configuration files +- Add type hints and docstrings +- Fix specific identified bugs +``` From bcc57f786a587e8c8c8cd7e9e8ed7e32d260eec9 Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 17:09:21 +0000 Subject: [PATCH 2/6] fix: add PyPI version verification to prevent shipping outdated deps CRITICAL: This fix ensures we NEVER ship outdated versions to consumer repos. Problem: - The sync scripts read from autofix-versions.env which contained static pins - These pins could become stale without any mechanism to detect or update them - Consumer repos received outdated versions, wasting significant time Solution: 1. New script: scripts/update_versions_from_pypi.py - Queries PyPI for latest stable versions - Can check or update autofix-versions.env - Fails if versions are outdated (--fail-on-outdated) 2. New tests: tests/scripts/test_update_versions_from_pypi.py - 31 tests including integration tests that query real PyPI - Consumer repo sampling tests that verify versions are current - Regression prevention tests 3. Modified: maint-52-sync-dev-versions.yml - Added verify-versions-current job that BLOCKS sync if outdated - Syncing now FAILS if autofix-versions.env has stale versions 4. New workflow: maint-auto-update-pypi-versions.yml - Runs daily at 03:00 UTC (before weekly sync at 05:00) - Auto-creates PRs when versions need updating This ensures versions are verified against PyPI before every sync. --- .../workflows/maint-52-sync-dev-versions.yml | 34 +- .../maint-auto-update-pypi-versions.yml | 127 +++++ scripts/update_versions_from_pypi.py | 234 +++++++++ .../scripts/test_update_versions_from_pypi.py | 448 ++++++++++++++++++ 4 files changed, 839 insertions(+), 4 deletions(-) create mode 100644 .github/workflows/maint-auto-update-pypi-versions.yml create mode 100755 scripts/update_versions_from_pypi.py create mode 100755 tests/scripts/test_update_versions_from_pypi.py diff --git a/.github/workflows/maint-52-sync-dev-versions.yml b/.github/workflows/maint-52-sync-dev-versions.yml index 0052b7241..278a4ebc9 100644 --- a/.github/workflows/maint-52-sync-dev-versions.yml +++ b/.github/workflows/maint-52-sync-dev-versions.yml @@ -45,8 +45,34 @@ env: stranske/Trend_Model_Project jobs: + # CRITICAL: Verify versions are current BEFORE syncing to consumer repos + verify-versions-current: + name: Verify versions are current + runs-on: ubuntu-latest + steps: + - name: Checkout Workflows + uses: actions/checkout@v4 + with: + sparse-checkout: | + .github/workflows/autofix-versions.env + scripts/update_versions_from_pypi.py + sparse-checkout-cone-mode: false + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Verify versions against PyPI + run: | + echo "🔍 Checking that all versions in autofix-versions.env are current..." + python scripts/update_versions_from_pypi.py --check --fail-on-outdated + echo "" + echo "✅ All versions are current - safe to sync to consumer repos!" + prepare: name: Prepare version sync + needs: verify-versions-current # Don't sync until we verify versions are current! runs-on: ubuntu-latest outputs: repos: ${{ steps.repos.outputs.matrix }} @@ -122,7 +148,7 @@ jobs: run: | if [ -f "consumer/pyproject.toml" ]; then echo "has_pyproject=true" >> "$GITHUB_OUTPUT" - + # Check if it has dev dependencies if grep -q '\[project.optional-dependencies\]' consumer/pyproject.toml; then if grep -qE '^dev\s*=' consumer/pyproject.toml; then @@ -154,7 +180,7 @@ jobs: echo "has_changes=false" >> "$GITHUB_OUTPUT" else echo "has_changes=true" >> "$GITHUB_OUTPUT" - + # Apply if not dry run if [ "${{ inputs.dry_run }}" != "true" ]; then python ../scripts/sync_dev_dependencies.py --apply --use-minimum-pins @@ -176,7 +202,7 @@ jobs: cd consumer echo "Adding dev dependencies section to pyproject.toml..." - + # Use --create-if-missing to add dev deps if python ../scripts/sync_dev_dependencies.py --apply --use-minimum-pins --create-if-missing 2>&1 | tee /tmp/sync_output.txt; then if grep -q "version updates" /tmp/sync_output.txt; then @@ -223,7 +249,7 @@ jobs: # Add lockfile if it exists and was modified git add pyproject.toml .github/workflows/autofix-versions.env if [ -f requirements.lock ]; then git add requirements.lock; fi - + # Commit with multi-line message commit_msg="deps: sync dev tool versions from Workflows diff --git a/.github/workflows/maint-auto-update-pypi-versions.yml b/.github/workflows/maint-auto-update-pypi-versions.yml new file mode 100644 index 000000000..5689ff943 --- /dev/null +++ b/.github/workflows/maint-auto-update-pypi-versions.yml @@ -0,0 +1,127 @@ +# Auto-update dev tool versions from PyPI +# +# This workflow ensures autofix-versions.env stays current with PyPI releases. +# It runs daily and creates a PR if any versions are outdated. +# +# CRITICAL: This workflow MUST run before maint-52-sync-dev-versions.yml +# to ensure we never ship stale versions to consumer repos. + +name: Maint Auto-Update PyPI Versions + +on: + schedule: + # Daily at 03:00 UTC (before the weekly sync at 05:00) + - cron: '0 3 * * *' + workflow_dispatch: + inputs: + dry_run: + description: 'Preview changes without creating PR' + type: boolean + default: false + +permissions: + contents: write + pull-requests: write + +jobs: + check-and-update: + name: Check PyPI for updates + runs-on: ubuntu-latest + steps: + - name: Checkout Workflows + uses: actions/checkout@v4 + with: + fetch-depth: 1 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Check for outdated versions + id: check + run: | + echo "🔍 Checking PyPI for latest versions..." + if python scripts/update_versions_from_pypi.py --check 2>&1 | tee /tmp/check_output.txt; then + echo "has_updates=false" >> "$GITHUB_OUTPUT" + else + # Check exits with 0 even for outdated (use --fail-on-outdated for non-zero) + if grep -q "outdated" /tmp/check_output.txt; then + echo "has_updates=true" >> "$GITHUB_OUTPUT" + else + echo "has_updates=false" >> "$GITHUB_OUTPUT" + fi + fi + cat /tmp/check_output.txt + + - name: Update versions + id: update + if: steps.check.outputs.has_updates == 'true' && inputs.dry_run != true + run: | + echo "📦 Updating autofix-versions.env with latest PyPI versions..." + python scripts/update_versions_from_pypi.py --apply 2>&1 | tee /tmp/update_output.txt + + # Extract update summary for PR body + { + echo "summary<> "$GITHUB_OUTPUT" + + - name: Create PR + if: steps.check.outputs.has_updates == 'true' && inputs.dry_run != true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + # Configure git + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + + # Check if there are actual changes + if git diff --quiet .github/workflows/autofix-versions.env; then + echo "No changes to commit" + exit 0 + fi + + # Create branch + branch="auto/update-pypi-versions-$(date +%Y%m%d)" + git checkout -b "$branch" + + # Commit changes + git add .github/workflows/autofix-versions.env + git commit -m "chore: update dev tool versions from PyPI + + Auto-generated by maint-auto-update-pypi-versions workflow. + + ${{ steps.update.outputs.summary }}" + + # Push and create PR + git push origin "$branch" + + gh pr create \ + --title "chore: update dev tool versions from PyPI" \ + --body "## Summary + + This PR updates the pinned dev tool versions in \`autofix-versions.env\` to match the latest releases on PyPI. + + ### Changes + \`\`\` + ${{ steps.update.outputs.summary }} + \`\`\` + + ### Why this matters + Keeping dev tool versions current ensures: + - Consumer repos receive the latest bug fixes and features + - We don't ship known-vulnerable or outdated tooling + - Version drift between repos is minimized + + --- + *Auto-generated by the [maint-auto-update-pypi-versions](.github/workflows/maint-auto-update-pypi-versions.yml) workflow*" \ + --label "dependencies" \ + --label "automation" + + - name: Dry run summary + if: inputs.dry_run == true + run: | + echo "🔍 Dry run - would have made these updates:" + python scripts/update_versions_from_pypi.py --check diff --git a/scripts/update_versions_from_pypi.py b/scripts/update_versions_from_pypi.py new file mode 100755 index 000000000..37729b37e --- /dev/null +++ b/scripts/update_versions_from_pypi.py @@ -0,0 +1,234 @@ +#!/usr/bin/env python3 +"""Fetch latest versions from PyPI and update autofix-versions.env. + +This script queries PyPI for the latest stable versions of all dev tools +in the autofix-versions.env file and updates them. + +CRITICAL: This script ensures we never ship outdated versions to consumer repos +by fetching the actual current versions from the authoritative source (PyPI). + +Usage: + python scripts/update_versions_from_pypi.py --check # Show what would be updated + python scripts/update_versions_from_pypi.py --apply # Update autofix-versions.env +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +import urllib.request +from pathlib import Path +from typing import NamedTuple + +# Path to the version pins file +PIN_FILE = Path(".github/workflows/autofix-versions.env") + +# Map env keys to PyPI package names +# This is the authoritative mapping for all synced dev tools +PACKAGE_MAPPING: dict[str, str] = { + "BLACK_VERSION": "black", + "RUFF_VERSION": "ruff", + "ISORT_VERSION": "isort", + "DOCFORMATTER_VERSION": "docformatter", + "MYPY_VERSION": "mypy", + "PYTEST_VERSION": "pytest", + "PYTEST_COV_VERSION": "pytest-cov", + "PYTEST_XDIST_VERSION": "pytest-xdist", + "COVERAGE_VERSION": "coverage", +} + + +class VersionInfo(NamedTuple): + """Information about a package version.""" + + current: str + latest: str + is_outdated: bool + + +def get_latest_pypi_version(package_name: str) -> str | None: + """Fetch the latest stable version from PyPI. + + This queries the PyPI JSON API and returns the latest non-prerelease version. + Falls back to the latest release if all releases are prereleases. + """ + url = f"https://pypi.org/pypi/{package_name}/json" + try: + with urllib.request.urlopen(url, timeout=15) as resp: + data = json.loads(resp.read().decode()) + # Get the latest version (this is the current stable release) + latest: str | None = data.get("info", {}).get("version") + if latest: + return str(latest) + + # Fallback: find the latest from releases + releases: dict[str, list[dict[str, object]]] = data.get("releases", {}) + if releases: + # Filter out prereleases and yanked versions + stable_versions: list[str] = [] + for ver, files in releases.items(): + # Skip if all files are yanked + if files and all(f.get("yanked", False) for f in files): + continue + # Skip prereleases (contains a, b, rc, dev, etc.) + if re.search(r"(a|b|rc|dev|alpha|beta)\d*$", ver, re.IGNORECASE): + continue + stable_versions.append(ver) + + if stable_versions: + # Sort by version tuple + stable_versions.sort(key=_version_tuple, reverse=True) + return stable_versions[0] + + return None + except Exception as e: + print(f" ⚠️ Could not fetch {package_name} from PyPI: {e}", file=sys.stderr) + return None + + +def _version_tuple(version: str) -> tuple[int, ...]: + """Convert version string to tuple for comparison.""" + # Handle versions like "1.2.3rc1" by stripping pre-release suffix + clean = re.match(r"(\d+(?:\.\d+)*)", version) + if clean: + return tuple(int(x) for x in clean.group(1).split(".")) + return (0,) + + +def parse_env_file(path: Path) -> dict[str, str]: + """Parse the autofix-versions.env file into a dict of key=value pairs.""" + if not path.exists(): + return {} + + values: dict[str, str] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + if "=" not in line: + continue + key, value = line.split("=", 1) + values[key.strip()] = value.strip() + + return values + + +def update_env_file(path: Path, updates: dict[str, str]) -> None: + """Update specific values in the env file while preserving comments and order.""" + if not path.exists(): + raise FileNotFoundError(f"Pin file not found: {path}") + + lines = path.read_text(encoding="utf-8").splitlines() + new_lines = [] + + for line in lines: + stripped = line.strip() + if not stripped or stripped.startswith("#"): + new_lines.append(line) + continue + + if "=" in stripped: + key, _ = stripped.split("=", 1) + key = key.strip() + if key in updates: + new_lines.append(f"{key}={updates[key]}") + else: + new_lines.append(line) + else: + new_lines.append(line) + + path.write_text("\n".join(new_lines) + "\n", encoding="utf-8") + + +def check_versions(pin_file: Path) -> dict[str, VersionInfo]: + """Check all versions against PyPI and return comparison info.""" + current_pins = parse_env_file(pin_file) + results: dict[str, VersionInfo] = {} + + for env_key, package_name in PACKAGE_MAPPING.items(): + current_version = current_pins.get(env_key, "") + if not current_version: + print(f" ⚠️ {env_key} not found in pin file") + continue + + print(f" Checking {package_name}...", end=" ", flush=True) + latest_version = get_latest_pypi_version(package_name) + + if latest_version is None: + print("failed to fetch") + continue + + is_outdated = current_version != latest_version + status = "OUTDATED" if is_outdated else "OK" + print(f"{current_version} -> {latest_version} [{status}]") + + results[env_key] = VersionInfo( + current=current_version, + latest=latest_version, + is_outdated=is_outdated, + ) + + return results + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Update autofix-versions.env with latest versions from PyPI" + ) + parser.add_argument( + "--check", + action="store_true", + help="Check for outdated versions without updating", + ) + parser.add_argument( + "--apply", + action="store_true", + help="Update autofix-versions.env with latest versions", + ) + parser.add_argument( + "--pin-file", + type=Path, + default=PIN_FILE, + help=f"Path to pin file (default: {PIN_FILE})", + ) + parser.add_argument( + "--fail-on-outdated", + action="store_true", + help="Exit with code 1 if any version is outdated (useful for CI)", + ) + + args = parser.parse_args(argv) + + if not args.check and not args.apply: + parser.error("Must specify either --check or --apply") + + print(f"Checking versions in {args.pin_file}...") + results = check_versions(args.pin_file) + + outdated = {k: v for k, v in results.items() if v.is_outdated} + + if not outdated: + print("\n✅ All versions are up to date!") + return 0 + + print(f"\n⚠️ Found {len(outdated)} outdated version(s):") + for env_key, info in outdated.items(): + pkg = PACKAGE_MAPPING[env_key] + print(f" {pkg}: {info.current} -> {info.latest}") + + if args.apply: + updates = {k: v.latest for k, v in outdated.items()} + update_env_file(args.pin_file, updates) + print(f"\n✅ Updated {len(updates)} version(s) in {args.pin_file}") + return 0 + + if args.fail_on_outdated: + return 1 + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/scripts/test_update_versions_from_pypi.py b/tests/scripts/test_update_versions_from_pypi.py new file mode 100755 index 000000000..a8f3f9e3a --- /dev/null +++ b/tests/scripts/test_update_versions_from_pypi.py @@ -0,0 +1,448 @@ +#!/usr/bin/env python3 +"""Tests for update_versions_from_pypi.py. + +CRITICAL: These tests ensure we NEVER ship outdated versions to consumer repos. +They include: +1. Unit tests for the script functionality +2. Integration tests that ACTUALLY query PyPI +3. Consumer repo simulation tests that verify versions are current + +The integration tests are marked with @pytest.mark.integration and can be run +separately to validate that our pinned versions are actually current on PyPI. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +from scripts import update_versions_from_pypi +from scripts.update_versions_from_pypi import ( + PACKAGE_MAPPING, + VersionInfo, + _version_tuple, + check_versions, + get_latest_pypi_version, + parse_env_file, + update_env_file, +) + + +class TestVersionTuple: + """Tests for version string to tuple conversion.""" + + def test_simple_version(self) -> None: + assert _version_tuple("1.2.3") == (1, 2, 3) + + def test_major_only(self) -> None: + assert _version_tuple("1") == (1,) + + def test_major_minor(self) -> None: + assert _version_tuple("1.2") == (1, 2) + + def test_four_parts(self) -> None: + assert _version_tuple("1.2.3.4") == (1, 2, 3, 4) + + def test_prerelease_stripped(self) -> None: + assert _version_tuple("1.2.3rc1") == (1, 2, 3) + + def test_invalid_returns_zero(self) -> None: + assert _version_tuple("invalid") == (0,) + + +class TestParseEnvFile: + """Tests for parsing autofix-versions.env files.""" + + def test_parse_simple_file(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.14.10\nMYPY_VERSION=1.19.1\n") + + result = parse_env_file(env_file) + assert result == {"RUFF_VERSION": "0.14.10", "MYPY_VERSION": "1.19.1"} + + def test_skips_comments(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("# Comment\nRUFF_VERSION=0.14.10\n") + + result = parse_env_file(env_file) + assert result == {"RUFF_VERSION": "0.14.10"} + + def test_skips_empty_lines(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.14.10\n\nMYPY_VERSION=1.19.1\n") + + result = parse_env_file(env_file) + assert result == {"RUFF_VERSION": "0.14.10", "MYPY_VERSION": "1.19.1"} + + def test_missing_file_returns_empty(self, tmp_path: Path) -> None: + result = parse_env_file(tmp_path / "nonexistent.env") + assert result == {} + + def test_strips_whitespace(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text(" RUFF_VERSION = 0.14.10 \n") + + result = parse_env_file(env_file) + assert result == {"RUFF_VERSION": "0.14.10"} + + +class TestUpdateEnvFile: + """Tests for updating env file in place.""" + + def test_update_single_value(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.14.0\nMYPY_VERSION=1.19.0\n") + + update_env_file(env_file, {"RUFF_VERSION": "0.14.10"}) + + result = parse_env_file(env_file) + assert result["RUFF_VERSION"] == "0.14.10" + assert result["MYPY_VERSION"] == "1.19.0" + + def test_preserves_comments(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("# This is a comment\nRUFF_VERSION=0.14.0\n") + + update_env_file(env_file, {"RUFF_VERSION": "0.14.10"}) + + content = env_file.read_text() + assert "# This is a comment" in content + assert "RUFF_VERSION=0.14.10" in content + + def test_preserves_order(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("A=1\nB=2\nC=3\n") + + update_env_file(env_file, {"B": "9"}) + + lines = env_file.read_text().strip().split("\n") + assert lines == ["A=1", "B=9", "C=3"] + + def test_missing_file_raises(self, tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + update_env_file(tmp_path / "nonexistent.env", {"X": "1"}) + + +class TestGetLatestPyPIVersion: + """Tests for PyPI API queries.""" + + def test_successful_fetch(self) -> None: + """Mock a successful PyPI response.""" + mock_response = MagicMock() + mock_response.read.return_value = json.dumps( + { + "info": {"version": "1.2.3"}, + "releases": {}, + } + ).encode() + mock_response.__enter__ = MagicMock(return_value=mock_response) + mock_response.__exit__ = MagicMock(return_value=False) + + with patch("urllib.request.urlopen", return_value=mock_response): + result = get_latest_pypi_version("some-package") + + assert result == "1.2.3" + + def test_network_error_returns_none(self) -> None: + """Network errors should return None, not crash.""" + with patch("urllib.request.urlopen", side_effect=TimeoutError("timeout")): + result = get_latest_pypi_version("some-package") + + assert result is None + + +class TestCheckVersions: + """Tests for the check_versions function.""" + + def test_identifies_outdated(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.1.0\n") + + # Mock PyPI to return a newer version + with patch.object( + update_versions_from_pypi, + "get_latest_pypi_version", + return_value="0.14.10", + ): + results = check_versions(env_file) + + assert "RUFF_VERSION" in results + assert results["RUFF_VERSION"].current == "0.1.0" + assert results["RUFF_VERSION"].latest == "0.14.10" + assert results["RUFF_VERSION"].is_outdated is True + + def test_identifies_current(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.14.10\n") + + with patch.object( + update_versions_from_pypi, + "get_latest_pypi_version", + return_value="0.14.10", + ): + results = check_versions(env_file) + + assert results["RUFF_VERSION"].is_outdated is False + + +class TestMain: + """Tests for the main CLI function.""" + + def test_check_mode_no_updates(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.14.10\n") + + with patch.object( + update_versions_from_pypi, + "get_latest_pypi_version", + return_value="0.14.10", + ): + result = update_versions_from_pypi.main( + [ + "--check", + "--pin-file", + str(env_file), + ] + ) + + assert result == 0 + + def test_check_mode_with_outdated_fail_flag(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.1.0\n") + + with patch.object( + update_versions_from_pypi, + "get_latest_pypi_version", + return_value="0.14.10", + ): + result = update_versions_from_pypi.main( + [ + "--check", + "--fail-on-outdated", + "--pin-file", + str(env_file), + ] + ) + + assert result == 1 + + def test_apply_mode_updates_file(self, tmp_path: Path) -> None: + env_file = tmp_path / "test.env" + env_file.write_text("RUFF_VERSION=0.1.0\n") + + with patch.object( + update_versions_from_pypi, + "get_latest_pypi_version", + return_value="0.14.10", + ): + result = update_versions_from_pypi.main( + [ + "--apply", + "--pin-file", + str(env_file), + ] + ) + + assert result == 0 + assert "RUFF_VERSION=0.14.10" in env_file.read_text() + + +# ============================================================================ +# INTEGRATION TESTS - Actually query PyPI +# These tests ensure our pinned versions are not outdated +# ============================================================================ + + +@pytest.mark.integration +class TestPyPIIntegration: + """Integration tests that actually query PyPI. + + Run with: pytest -m integration tests/scripts/test_update_versions_from_pypi.py + """ + + def test_can_fetch_real_ruff_version(self) -> None: + """Verify we can fetch the real ruff version from PyPI.""" + version = get_latest_pypi_version("ruff") + assert version is not None + assert len(version) > 0 + # Version should be a valid semver-ish format + parts = version.split(".") + assert len(parts) >= 2 + assert all(p.isdigit() or p[0].isdigit() for p in parts) + + def test_can_fetch_real_mypy_version(self) -> None: + """Verify we can fetch the real mypy version from PyPI.""" + version = get_latest_pypi_version("mypy") + assert version is not None + assert len(version) > 0 + + def test_can_fetch_all_mapped_packages(self) -> None: + """Verify we can fetch versions for ALL packages in our mapping.""" + for env_key, package_name in PACKAGE_MAPPING.items(): + version = get_latest_pypi_version(package_name) + assert version is not None, f"Failed to fetch {package_name} for {env_key}" + + +# ============================================================================ +# CONSUMER REPO SAMPLING TESTS +# These tests simulate what happens when we sync to consumer repos +# ============================================================================ + + +@pytest.mark.integration +class TestConsumerRepoSampling: + """Tests that sample consumer repo dependencies to ensure we're shipping current versions. + + CRITICAL: These tests catch the exact problem of shipping outdated versions. + They verify that the versions in autofix-versions.env are actually current on PyPI. + """ + + def test_autofix_versions_env_not_stale(self) -> None: + """CRITICAL: Verify autofix-versions.env has current PyPI versions. + + This test reads the actual autofix-versions.env file and checks EACH + package against PyPI to ensure we're not shipping outdated versions. + """ + pin_file = Path(".github/workflows/autofix-versions.env") + if not pin_file.exists(): + pytest.skip("autofix-versions.env not found (not in Workflows repo)") + + current_pins = parse_env_file(pin_file) + stale_packages: list[str] = [] + + for env_key, package_name in PACKAGE_MAPPING.items(): + if env_key not in current_pins: + continue + + current_version = current_pins[env_key] + latest_version = get_latest_pypi_version(package_name) + + if latest_version is None: + continue # Skip if we can't reach PyPI + + if current_version != latest_version: + stale_packages.append( + f"{package_name}: pinned={current_version}, latest={latest_version}" + ) + + if stale_packages: + pytest.fail( + "STALE VERSIONS IN autofix-versions.env! " + "These packages are outdated:\n " + + "\n ".join(stale_packages) + + "\n\nRun: python scripts/update_versions_from_pypi.py --apply" + ) + + def test_template_sync_script_has_all_packages(self) -> None: + """Verify the template sync script maps all the same packages.""" + template_script = Path("templates/consumer-repo/scripts/sync_dev_dependencies.py") + if not template_script.exists(): + pytest.skip("Template sync script not found") + + content = template_script.read_text() + + # Check that all our package mappings exist in the template + for env_key in PACKAGE_MAPPING: + assert env_key in content, ( + f"Template sync script missing {env_key}. " + f"Consumer repos won't sync this package!" + ) + + def test_simulated_consumer_repo_sync(self, tmp_path: Path) -> None: + """Simulate what a consumer repo would receive. + + This test: + 1. Creates a fake consumer repo pyproject.toml with older versions + 2. Runs the sync process with current autofix-versions.env + 3. Verifies the resulting versions are what PyPI has + """ + # Read actual autofix-versions.env + pin_file = Path(".github/workflows/autofix-versions.env") + if not pin_file.exists(): + pytest.skip("autofix-versions.env not found") + + current_pins = parse_env_file(pin_file) + + # For each pinned package, verify it matches PyPI + # This catches the case where autofix-versions.env itself is stale + mismatches: list[str] = [] + + for env_key, package_name in PACKAGE_MAPPING.items(): + if env_key not in current_pins: + continue + + our_version = current_pins[env_key] + pypi_version = get_latest_pypi_version(package_name) + + if pypi_version and our_version != pypi_version: + mismatches.append(f"{package_name}: we have {our_version}, PyPI has {pypi_version}") + + if mismatches: + pytest.fail( + "Consumer repos would receive STALE versions!\n" + "Mismatches:\n " + "\n ".join(mismatches) + ) + + +# ============================================================================ +# REGRESSION TESTS +# Specific tests to prevent past failures from recurring +# ============================================================================ + + +class TestRegressionPrevention: + """Tests specifically designed to prevent past failures.""" + + def test_version_comparison_is_exact(self) -> None: + """Ensure version comparison doesn't use >= or fuzzy matching. + + Past issue: Versions were compared loosely, allowing older versions to pass. + """ + info = VersionInfo(current="1.0.0", latest="1.0.1", is_outdated=True) + # Even minor version differences should be flagged + assert info.is_outdated is True + + info2 = VersionInfo(current="1.0.1", latest="1.0.1", is_outdated=False) + assert info2.is_outdated is False + + def test_package_mapping_completeness(self) -> None: + """Ensure PACKAGE_MAPPING covers all expected dev tools.""" + expected_tools = { + "ruff", + "black", + "mypy", + "pytest", + "pytest-cov", + "coverage", + } + + mapped_packages = set(PACKAGE_MAPPING.values()) + + missing = expected_tools - mapped_packages + assert not missing, f"Missing critical tools in PACKAGE_MAPPING: {missing}" + + def test_no_hardcoded_fallback_versions(self) -> None: + """Ensure there are no hardcoded fallback versions that could be stale. + + Past issue: Scripts had DEFAULT_VERSION constants that became stale. + """ + import ast + + script_path = Path("scripts/update_versions_from_pypi.py") + content = script_path.read_text() + tree = ast.parse(content) + + for node in ast.walk(tree): + if isinstance(node, ast.Assign): + for target in node.targets: + if isinstance(target, ast.Name): + name = target.id.upper() + if "VERSION" in name and "FALLBACK" in name: + pytest.fail( + f"Found potential hardcoded fallback: {target.id}. " + f"Remove it - we must always query PyPI!" + ) From 30dca198ce44161892372d0cfd58f1195dc53ddc Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 17:33:10 +0000 Subject: [PATCH 3/6] docs: expand semantic dedup section in LangChain proposal - Add detailed comparison of Levenshtein vs embeddings-based similarity - Include code example using LangChain + FAISS vector store - Document advantages: catches 'same idea, different phrasing' duplicates - Clarify integration point in agents-63-issue-intake.yml Addresses concern about upgrading Agents 63 issue reuse/dedup from Levenshtein to semantic matching. --- docs/plans/langchain-issue-intake-proposal.md | 49 +++++++++++++++++-- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/docs/plans/langchain-issue-intake-proposal.md b/docs/plans/langchain-issue-intake-proposal.md index 8c2305d32..47c2c29fe 100644 --- a/docs/plans/langchain-issue-intake-proposal.md +++ b/docs/plans/langchain-issue-intake-proposal.md @@ -391,13 +391,54 @@ This keeps the complexity low while allowing natural interaction. **Plausibility**: ⭐⭐⭐ MEDIUM-HIGH **Scope**: ~2-3 days (data collection), ongoing refinement -### 6. Duplicate/Related Issue Detection +### 6. Duplicate/Related Issue Detection (Semantic Matching Upgrade) **Use Case**: Before creating new issue, check if similar work exists. -**Approach**: -- Embed issue description, compare to existing open issues -- Warn if high similarity detected +**The Problem (Current State)**: +- Existing dedup logic uses exact title matching or Levenshtein distance +- Levenshtein is good for typos ("fix bug" vs "fxi bug") but bad at semantic similarity +- "Add unit tests for portfolio module" and "Write test coverage for portfolio.py" are the same intent but have low Levenshtein similarity +- Result: False negatives (duplicate issues created) and false positives (unrelated issues flagged) + +**LangChain Solution**: +- **Embeddings-based similarity** catches "same idea, different phrasing" +- Uses vector stores (FAISS, Chroma) for efficient similarity search +- Semantic distance measures conceptual similarity, not character edits + +**Technical Approach**: +```python +from langchain_openai import OpenAIEmbeddings +from langchain_community.vectorstores import FAISS + +# Generate embeddings for issue description +embeddings = OpenAIEmbeddings( + model="text-embedding-3-small", + base_url="https://models.inference.ai.azure.com", + api_key=os.environ["GITHUB_TOKEN"], +) + +# Build vector store from existing open issues +issue_texts = [f"{issue.title}\n{issue.body}" for issue in open_issues] +vector_store = FAISS.from_texts(issue_texts, embeddings, metadatas=[{"number": i.number} for i in open_issues]) + +# Search for similar issues +similar = vector_store.similarity_search_with_score(new_issue_text, k=5) +duplicates = [(doc.metadata["number"], score) for doc, score in similar if score > THRESHOLD] +``` + +**Advantages over Levenshtein**: +| Aspect | Levenshtein | Semantic Embeddings | +|--------|-------------|---------------------| +| "Same typo" detection | ✅ Excellent | ✅ Good | +| "Same idea, different words" | ❌ Poor | ✅ Excellent | +| Performance at scale | ⚠️ O(n*m) per comparison | ✅ O(log n) with vector index | +| False positives | High (similar chars ≠ similar meaning) | Low | +| False negatives | High (different chars = missed duplicates) | Low | + +**Integration Point**: +- Run during `agents-63-issue-intake.yml` before bridge creation +- Post advisory comment with similar issues (doesn't block creation) - Link related issues for context **Plausibility**: ⭐⭐⭐⭐ HIGH (embeddings are well-understood) From 0ca1ca4c68c41ee3d389c87266d525bee58df616 Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 17:41:10 +0000 Subject: [PATCH 4/6] docs: expand semantic matching to cover both issues AND labels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhancement #6 now covers: 1. Issue deduplication - semantic similarity for duplicate detection 2. Label matching - replace Levenshtein in findMatchingLabel() with embeddings Both use cases share the same embeddings infrastructure (FAISS + GitHub Models). Examples of label matching improvements: - 'defect' → 'bug' (synonyms) - 'improvement' → 'enhancement' (synonyms) - 'testing' → 'tests' (related concepts) Updated issue #481 with expanded scope and tasks. --- docs/plans/langchain-issue-intake-proposal.md | 46 ++++++++++++++++--- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/docs/plans/langchain-issue-intake-proposal.md b/docs/plans/langchain-issue-intake-proposal.md index 47c2c29fe..09b5d0209 100644 --- a/docs/plans/langchain-issue-intake-proposal.md +++ b/docs/plans/langchain-issue-intake-proposal.md @@ -393,20 +393,29 @@ This keeps the complexity low while allowing natural interaction. ### 6. Duplicate/Related Issue Detection (Semantic Matching Upgrade) -**Use Case**: Before creating new issue, check if similar work exists. +**Use Case**: Before creating new issue, check if similar work exists. Also improve label matching from Levenshtein to semantic similarity. **The Problem (Current State)**: + +*Issue Deduplication:* - Existing dedup logic uses exact title matching or Levenshtein distance - Levenshtein is good for typos ("fix bug" vs "fxi bug") but bad at semantic similarity - "Add unit tests for portfolio module" and "Write test coverage for portfolio.py" are the same intent but have low Levenshtein similarity - Result: False negatives (duplicate issues created) and false positives (unrelated issues flagged) +*Label Matching (in `agents-63-issue-intake.yml` lines 601-634):* +- Current implementation uses Levenshtein distance to find similar labels +- Works for typos: `bugfix` → matches `bug` ✅ +- Fails for synonyms: `defect` → doesn't match `bug` ❌ +- "enhancement", "feature", "improvement" are semantically equivalent but have no character similarity + **LangChain Solution**: - **Embeddings-based similarity** catches "same idea, different phrasing" - Uses vector stores (FAISS, Chroma) for efficient similarity search - Semantic distance measures conceptual similarity, not character edits +- **Same infrastructure serves both use cases** (issues AND labels) -**Technical Approach**: +**Technical Approach - Issue Deduplication**: ```python from langchain_openai import OpenAIEmbeddings from langchain_community.vectorstores import FAISS @@ -427,6 +436,26 @@ similar = vector_store.similarity_search_with_score(new_issue_text, k=5) duplicates = [(doc.metadata["number"], score) for doc, score in similar if score > THRESHOLD] ``` +**Technical Approach - Label Matching**: +```python +# Build vector store from existing repo labels +label_names = [label.name for label in repo_labels] +label_store = FAISS.from_texts(label_names, embeddings, metadatas=[{"name": l.name} for l in repo_labels]) + +# Match user-specified label to existing labels +def find_semantic_label_match(user_label: str, threshold: float = 0.8) -> str | None: + """Find semantically similar existing label.""" + results = label_store.similarity_search_with_score(user_label, k=1) + if results and results[0][1] >= threshold: + return results[0][0].metadata["name"] + return None + +# Examples: +# find_semantic_label_match("defect") → "bug" +# find_semantic_label_match("improvement") → "enhancement" +# find_semantic_label_match("testing") → "tests" +``` + **Advantages over Levenshtein**: | Aspect | Levenshtein | Semantic Embeddings | |--------|-------------|---------------------| @@ -436,13 +465,16 @@ duplicates = [(doc.metadata["number"], score) for doc, score in similar if score | False positives | High (similar chars ≠ similar meaning) | Low | | False negatives | High (different chars = missed duplicates) | Low | -**Integration Point**: -- Run during `agents-63-issue-intake.yml` before bridge creation -- Post advisory comment with similar issues (doesn't block creation) -- Link related issues for context +**Integration Points**: +1. **Issue deduplication**: Run during `agents-63-issue-intake.yml` before bridge creation + - Post advisory comment with similar issues (doesn't block creation) + - Link related issues for context +2. **Label matching**: Replace Levenshtein in `findMatchingLabel()` function + - Same embeddings model, different vector store + - Cache label embeddings (labels change rarely) **Plausibility**: ⭐⭐⭐⭐ HIGH (embeddings are well-understood) -**Scope**: ~2 days +**Scope**: ~2-3 days (expanded to include label matching) ### 7. Automatic Task Decomposition From 50305db288d7830b719b4f98fe29ef1ae6c0db37 Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 17:52:45 +0000 Subject: [PATCH 5/6] fix: address PR review feedback from bot comments - Remove unnecessary str() cast in get_latest_pypi_version (Copilot) - Fix update detection logic that would never find outdated versions (Copilot + Codex P1) - Improve test to catch more fallback version naming patterns (Copilot) The workflow check step was incorrectly relying on exit codes when the script always exits 0 for --check mode. Now directly greps output for 'outdated' to properly detect when updates are needed. --- .../maint-auto-update-pypi-versions.yml | 14 ++++++-------- scripts/update_versions_from_pypi.py | 2 +- .../scripts/test_update_versions_from_pypi.py | 18 +++++++++++++----- 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/.github/workflows/maint-auto-update-pypi-versions.yml b/.github/workflows/maint-auto-update-pypi-versions.yml index 5689ff943..eb5a321e0 100644 --- a/.github/workflows/maint-auto-update-pypi-versions.yml +++ b/.github/workflows/maint-auto-update-pypi-versions.yml @@ -42,15 +42,13 @@ jobs: id: check run: | echo "🔍 Checking PyPI for latest versions..." - if python scripts/update_versions_from_pypi.py --check 2>&1 | tee /tmp/check_output.txt; then - echo "has_updates=false" >> "$GITHUB_OUTPUT" + python scripts/update_versions_from_pypi.py --check 2>&1 | tee /tmp/check_output.txt + # Script exits 0 even for outdated (use --fail-on-outdated for non-zero) + # Check output directly for "outdated" to determine if updates are needed + if grep -q "outdated" /tmp/check_output.txt; then + echo "has_updates=true" >> "$GITHUB_OUTPUT" else - # Check exits with 0 even for outdated (use --fail-on-outdated for non-zero) - if grep -q "outdated" /tmp/check_output.txt; then - echo "has_updates=true" >> "$GITHUB_OUTPUT" - else - echo "has_updates=false" >> "$GITHUB_OUTPUT" - fi + echo "has_updates=false" >> "$GITHUB_OUTPUT" fi cat /tmp/check_output.txt diff --git a/scripts/update_versions_from_pypi.py b/scripts/update_versions_from_pypi.py index 37729b37e..674773e7b 100755 --- a/scripts/update_versions_from_pypi.py +++ b/scripts/update_versions_from_pypi.py @@ -61,7 +61,7 @@ def get_latest_pypi_version(package_name: str) -> str | None: # Get the latest version (this is the current stable release) latest: str | None = data.get("info", {}).get("version") if latest: - return str(latest) + return latest # Fallback: find the latest from releases releases: dict[str, list[dict[str, object]]] = data.get("releases", {}) diff --git a/tests/scripts/test_update_versions_from_pypi.py b/tests/scripts/test_update_versions_from_pypi.py index a8f3f9e3a..71ba912ab 100755 --- a/tests/scripts/test_update_versions_from_pypi.py +++ b/tests/scripts/test_update_versions_from_pypi.py @@ -436,13 +436,21 @@ def test_no_hardcoded_fallback_versions(self) -> None: content = script_path.read_text() tree = ast.parse(content) + # Check for various fallback naming patterns that could contain stale versions + fallback_patterns = [ + ("VERSION", "FALLBACK"), # VERSION_FALLBACK, FALLBACK_VERSION + ("VERSION", "DEFAULT"), # DEFAULT_VERSION, VERSION_DEFAULT + ("DEFAULT", "VER"), # DEFAULT_VER + ] + for node in ast.walk(tree): if isinstance(node, ast.Assign): for target in node.targets: if isinstance(target, ast.Name): name = target.id.upper() - if "VERSION" in name and "FALLBACK" in name: - pytest.fail( - f"Found potential hardcoded fallback: {target.id}. " - f"Remove it - we must always query PyPI!" - ) + for pattern1, pattern2 in fallback_patterns: + if pattern1 in name and pattern2 in name: + pytest.fail( + f"Found potential hardcoded fallback: {target.id}. " + f"Remove it - we must always query PyPI!" + ) From dc56436a2e05e9a1e71e22ab00ed62f259320bc0 Mon Sep 17 00:00:00 2001 From: stranske Date: Sat, 3 Jan 2026 17:56:12 +0000 Subject: [PATCH 6/6] fix: add maint-auto-update-pypi-versions.yml to workflow inventory Add the new workflow to: - test_workflow_naming.py EXPECTED_NAMES mapping - docs/ci/WORKFLOWS.md workflow list - docs/ci/WORKFLOW_SYSTEM.md description and reference table This fixes the failing workflow inventory tests. --- docs/ci/WORKFLOWS.md | 1 + docs/ci/WORKFLOW_SYSTEM.md | 5 +++++ tests/workflows/test_workflow_naming.py | 1 + 3 files changed, 7 insertions(+) diff --git a/docs/ci/WORKFLOWS.md b/docs/ci/WORKFLOWS.md index e8a64e881..7ef75b409 100644 --- a/docs/ci/WORKFLOWS.md +++ b/docs/ci/WORKFLOWS.md @@ -105,6 +105,7 @@ The gate uses the shared `.github/scripts/detect-changes.js` helper to decide wh * [`maint-sync-env-from-pyproject.yml`](../../.github/workflows/maint-sync-env-from-pyproject.yml) syncs dev tool version pins from `pyproject.toml` to `autofix-versions.env` after Dependabot updates land. * [`maint-52-validate-workflows.yml`](../../.github/workflows/maint-52-validate-workflows.yml) dry-parses every workflow with `yq`, runs `actionlint` with the repository allowlist, and fails fast when malformed YAML or unapproved actionlint findings slip in. * [`maint-52-sync-dev-versions.yml`](../../.github/workflows/maint-52-sync-dev-versions.yml) syncs dev tool versions (ruff, mypy, black, isort, pytest) from `autofix-versions.env` to consumer repository `pyproject.toml` files weekly or on version changes. +* [`maint-auto-update-pypi-versions.yml`](../../.github/workflows/maint-auto-update-pypi-versions.yml) checks PyPI daily for latest dev tool versions and creates a PR to update `autofix-versions.env` when versions are outdated. * [`maint-62-integration-consumer.yml`](../../.github/workflows/maint-62-integration-consumer.yml) runs daily at 05:05 UTC, on release publication, or by manual dispatch to execute the integration-repo scenarios via the reusable Python CI template and keep the integration failure issue updated. * [`maint-63-ensure-environments.yml`](../../.github/workflows/maint-63-ensure-environments.yml) ensures agent environments (`agent-standard`, `agent-high-privilege`) exist with appropriate protection rules for environment-gated workflows. * [`maint-65-sync-label-docs.yml`](../../.github/workflows/maint-65-sync-label-docs.yml) synchronizes `docs/LABELS.md` to consumer repositories weekly (Sundays 00:00 UTC) or via manual dispatch. diff --git a/docs/ci/WORKFLOW_SYSTEM.md b/docs/ci/WORKFLOW_SYSTEM.md index 829fe9018..2b2217b4b 100644 --- a/docs/ci/WORKFLOW_SYSTEM.md +++ b/docs/ci/WORKFLOW_SYSTEM.md @@ -537,6 +537,10 @@ Keep this table handy when you are triaging automation: it confirms which workfl syncs dev tool versions (ruff, mypy, black, isort, pytest) from `autofix-versions.env` to consumer repository `pyproject.toml` files weekly or when version changes are detected. +- **Maint Auto-Update PyPI Versions** – `.github/workflows/maint-auto-update-pypi-versions.yml` + checks PyPI daily (03:00 UTC) for latest dev tool versions and creates a PR + to update `autofix-versions.env` when versions are outdated, ensuring the + sync workflow never ships stale versions to consumer repos. - **Maint 62 Integration Consumer** – `.github/workflows/maint-62-integration-consumer.yml` exercises the reusable Python CI template against the `templates/integration-repo` scenarios on a daily schedule (05:05 UTC), on release publication, or via @@ -666,6 +670,7 @@ Keep this table handy when you are triaging automation: it confirms which workfl | **Maint Sync versions.env from pyproject.toml** (`maint-sync-env-from-pyproject.yml`, maintenance bucket) | `push` (`main`, `pyproject.toml`), `workflow_dispatch` | Sync dev tool version pins from `pyproject.toml` into `autofix-versions.env` after changes land. | ⚪ Automatic on main | [Maint sync env runs](https://github.com/stranske/Workflows/actions/workflows/maint-sync-env-from-pyproject.yml) | | **Maint 52 Validate Workflows** (`maint-52-validate-workflows.yml`, maintenance bucket) | `pull_request`, `push` (`main`) | Parse every workflow file with `yq`, honour the Actionlint allowlist, and fail fast when syntax errors or lint violations appear. | ⚪ Automatic on PR/main | [Maint 52 workflow validations](https://github.com/stranske/Trend_Model_Project/actions/workflows/maint-52-validate-workflows.yml) | | **Maint 52 Sync Dev Versions** (`maint-52-sync-dev-versions.yml`, maintenance bucket) | `schedule` (Sundays 01:00 UTC), `push` (`autofix-versions.env`), `workflow_dispatch` | Sync dev tool versions from `autofix-versions.env` to consumer repository `pyproject.toml` files. | ⚪ Scheduled/manual | [Sync dev versions runs](https://github.com/stranske/Workflows/actions/workflows/maint-52-sync-dev-versions.yml) | +| **Maint Auto-Update PyPI Versions** (`maint-auto-update-pypi-versions.yml`, maintenance bucket) | `schedule` (daily 03:00 UTC), `workflow_dispatch` | Check PyPI for latest dev tool versions and create a PR to update `autofix-versions.env` when versions are outdated. | ⚪ Scheduled | [Auto-update PyPI versions runs](https://github.com/stranske/Workflows/actions/workflows/maint-auto-update-pypi-versions.yml) | | **Maint Coverage Guard** (`maint-coverage-guard.yml`, maintenance bucket) | `schedule` (`45 6 * * *`), `workflow_dispatch` | Audit the latest Gate coverage trend artifact and compare it against the baseline, failing when coverage regresses beyond the guard thresholds. | ⚪ Scheduled | [Maint Coverage Guard runs](https://github.com/stranske/Trend_Model_Project/actions/workflows/maint-coverage-guard.yml) | | **Maint 46 Post CI** (`maint-46-post-ci.yml`, maintenance bucket) | `workflow_run` (Gate, `completed`) | Recovery-only: inspect the Gate run for a missing or failed `summary` job; when recovery is needed, collect the Gate artifacts, render the consolidated CI summary with coverage deltas, publish a markdown preview, and refresh the Gate commit status. Otherwise exit immediately. | ⚪ Automatic follow-up | [Maint 46 runs](https://github.com/stranske/Trend_Model_Project/actions/workflows/maint-46-post-ci.yml) | | **Maint 45 Cosmetic Repair** (`maint-45-cosmetic-repair.yml`, maintenance bucket) | `workflow_dispatch` | Run pytest + fixers manually and open a labelled PR when changes are required. | ⚪ Manual | [Maint 45 manual entry](https://github.com/stranske/Trend_Model_Project/actions/workflows/maint-45-cosmetic-repair.yml) | diff --git a/tests/workflows/test_workflow_naming.py b/tests/workflows/test_workflow_naming.py index e1cb739c6..9ee0b25a5 100644 --- a/tests/workflows/test_workflow_naming.py +++ b/tests/workflows/test_workflow_naming.py @@ -197,6 +197,7 @@ def test_workflow_display_names_are_unique(): "maint-sync-env-from-pyproject.yml": "Maint - Sync versions.env from pyproject.toml", "maint-52-validate-workflows.yml": "Maint 52 Validate Workflows", "maint-52-sync-dev-versions.yml": "Maint 52 Sync Dev Versions", + "maint-auto-update-pypi-versions.yml": "Maint Auto-Update PyPI Versions", "maint-62-integration-consumer.yml": "Maint 62 Integration Consumer", "maint-65-sync-label-docs.yml": "Maint 65 Sync Label Docs", "maint-66-monthly-audit.yml": "Maint 66 Monthly Audit",