Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .githooks/prepare-commit-msg
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash
# AI Adoption Signal Detection Hook
# Auto-tags commits assisted by GitHub Copilot or other AI tools
# Idempotent: safe to run on every commit

COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2

# Skip if commit is from merge or squash (auto-generated commit messages)
if [ "$COMMIT_SOURCE" = "merge" ] || [ "$COMMIT_SOURCE" = "squash" ]; then
exit 0
fi
Comment thread
gundersen-lumn marked this conversation as resolved.

# Read current commit message
COMMIT_MSG=$(cat "$COMMIT_MSG_FILE")

# Check for AI signals in multiple places
# Signal 1: GitHub Copilot co-author metadata
HAS_COPILOT_COAUTHOR=$(echo "$COMMIT_MSG" | grep -qi "co-authored-by.*github copilot" && echo 1 || echo 0)

# Signal 2: Explicit AI markers at start of message
HAS_COPILOT_MARKER=$(echo "$COMMIT_MSG" | grep -qiE "^(copilot|ai-assisted|@copilot):" && echo 1 || echo 0)

# Signal 3: AI tool mentions in commit body
HAS_AI_PATTERN=$(echo "$COMMIT_MSG" | grep -qiE "(copilot|claude|gpt|cursor|ai-assisted)" && echo 1 || echo 0)
Comment thread
gundersen-lumn marked this conversation as resolved.
Comment thread
gundersen-lumn marked this conversation as resolved.

# Idempotent check: don't re-tag if already tagged
if echo "$COMMIT_MSG" | head -1 | grep -q "^\[AI-ASSISTED\]"; then
exit 0
fi

# Auto-tag if any AI signal detected
if [ "$HAS_COPILOT_COAUTHOR" -gt 0 ] || [ "$HAS_COPILOT_MARKER" -gt 0 ] || [ "$HAS_AI_PATTERN" -gt 0 ]; then
FIRST_LINE=$(echo "$COMMIT_MSG" | head -1)
REST=$(echo "$COMMIT_MSG" | tail -n +2)

# Prepend tag to first line (idempotent: already checked above)
if [ -z "$REST" ]; then
echo "[AI-ASSISTED] $FIRST_LINE" > "$COMMIT_MSG_FILE"
else
echo "[AI-ASSISTED] $FIRST_LINE" > "$COMMIT_MSG_FILE"
echo "$REST" >> "$COMMIT_MSG_FILE"
fi
fi

exit 0
35 changes: 28 additions & 7 deletions .github/ISSUE_TEMPLATE/comprehensive-codebase-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,8 @@ body:
- scoring.md — tabular scoring with 1-10 ratings and justifications.
- architecture.mmd — Mermaid diagram source for the architecture.
- improvement-issues.md — synopsis of open improvement-focused GitHub issues with solution steps.
- ai-adoption.md — explicit AI adoption analysis for the last 6 months with an executive judgment (low/moderate/high), score, shipped-to-master section, contributor/subsystem adoption analysis, non-adopter table, next steps, and caveats.
- ai-adoption.csv — commit-level explicit AI-related commits in the last 6 months with columns: sha, date, author, normalized_author, subject, on_master, files_changed, top_level_area, dominant_subsystem, dominant_file.
- ai-adoption.md — explicit AI adoption analysis for the last 6 months with an executive judgment (low/moderate/high), score, shipped-to-master section, contributor/subsystem adoption analysis, **squashed commits analysis**, non-adopter table, next steps, and caveats.
- ai-adoption.csv — commit-level explicit AI-related commits in the last 6 months with columns: sha, date, author, normalized_author, subject, on_master, files_changed, top_level_area, dominant_subsystem, dominant_file, squash_merge_indicator, signal_type (explicit_keyword | github_metadata | git_hook | implicit_pattern).
- ai-adoption-by-developer.csv — developer aggregation from ai-adoption.csv with columns: developer, total_ai_commits, ai_commits_on_master, ai_commits_off_master, master_commit_share_pct.
- chi-compliance.md — generated only when the repository includes a Vue-based UI;
* FIRST, check index.html for Chi CDN references (CSS/JS URLs) - this is the authoritative Chi version
Expand All @@ -177,21 +177,42 @@ body:
- AI Adoption Report Requirements (last 6 months, output to copilot-eval):
* Analyze commits from now minus 6 months through today.
* Include all refs for contribution analysis, and separately analyze what reached master.
* Treat explicit AI-related commits as commit subjects containing (case-insensitive): copilot, @Copilot, ai-assisted, gpt, claude, cursor.
* **Signal Detection Strategy** (multiple layers to reduce developer friction):
1. Explicit AI Keywords (optional, zero friction if auto-tagged): Treat commit subjects containing (case-insensitive): copilot, @Copilot, ai-assisted, gpt, claude, cursor.
2. GitHub Copilot Attribution Metadata: If available, detect commits tagged with GitHub's AI-generated indicator (via GitHub API or webhook metadata).
3. Git Hooks (recommended for ease-of-use): If the repo includes a `.githooks/prepare-commit-msg` hook or similar, document its presence and how it auto-tags AI commits without developer intervention.
4. IDE/CLI Integration: Note if the codebase uses Copilot CLI or IDE extensions that auto-append AI indicators at commit time (no developer action required).
5. Implicit Patterns (fallback): Scan commit message structure for AI-generated patterns (e.g., structured formatting, typical Copilot phrasing, co-authored metadata like "Co-authored-by: GitHub Copilot").
6. Squashed Commit Recovery: Extract AI signals from individual commits inside squashed merges using all methods above.
* Recommended Setup for Minimal Developer Friction:
- Use a git hook (prepare-commit-msg or commit-msg) to auto-detect and auto-tag AI commits at commit time.
- Provide a `.githooks/prepare-commit-msg` template in the repo README for teams to adopt.
* If no automation in place, fall back to explicit keyword detection; but acknowledge this undercounts adoption.
* Treat on_master as commit is an ancestor of master.
* Treat content commit as files_changed > 0 and merge-only commit as files_changed = 0.
* Gather baseline counts: total commits in 6 months (all refs) and total commits in 6 months on master.
* Build ai-adoption.csv with one row per explicit AI-related commit and columns exactly: sha, date, author, normalized_author, subject, on_master, files_changed, top_level_area, dominant_subsystem, dominant_file.
* **Squashed Commit Inspection** (critical for accurate adoption metrics):
- For each *squash-merged PR commit* on master (typically a single-parent commit with subject like `... (#123)`), extract the PR number from the commit subject.
- If PR metadata is available (GitHub API), list the PR's commits and scan each commit subject/body for AI signals (explicit keywords, metadata, patterns).
- If PR metadata is not available locally, document that individual pre-squash commits cannot be reliably recovered from git history alone.
- Attribute detected AI signals to the original PR commit authors (not the squash-merge author) when PR commit data is available.
- Document the observed merge strategy (squash vs. merge vs. rebase) and any recovery limitations in the Caveats section.
- For squashed commits with AI signals, preserve the original author/date and create a logical "unmerged" entry in ai-adoption.csv with the source commits listed in a supplementary column.
- If PR metadata is available (GitHub API), cross-reference the PR number from the merge commit message to retrieve the full commit history before squashing.
- Document merge strategy (squash vs. fast-forward vs. three-way merge) in the Caveats section and note any adoption metrics recovered from squashed commits.
* Build ai-adoption.csv with one row per explicit AI-related commit (including unmerged squashed commits) and columns exactly: sha, date, author, normalized_author, subject, on_master, files_changed, top_level_area, dominant_subsystem, dominant_file, squash_merge_indicator (true/false), signal_type (explicit_keyword | github_metadata | git_hook | implicit_pattern).
* Compute normalized_author by merging obvious aliases via shared email identity and clear name/username variants (e.g., corporate name vs GitHub handle), and document normalization caveat in markdown.
* Build ai-adoption-by-developer.csv from ai-adoption.csv with columns exactly: developer, total_ai_commits, ai_commits_on_master, ai_commits_off_master, master_commit_share_pct.
* Detect active contributors with zero explicit AI signal: all active contributors in 6 months minus normalized authors in ai-adoption.csv, and include commit counts for this group.
* Avoid brittle one-liner heredocs in terminal; if scripting is needed, create small temporary scripts/files, run them, and clean up helper artifacts afterward.
* ai-adoption.md must include: Executive Judgment (moderate/low/high) with rationale; Method with exact filter terms, timeframe, and lower-bound caveat; section #2 What Actually Shipped To master with counts and table (Date, Author, SHA short, Subject, Dominant subsystem, Files changed); section #3 Developer And Subsystem Adoption with normalized contributor counts and subsystem distribution; section All AI Contributions Per Developer sourced from ai-adoption-by-developer.csv (Developer, Total AI commits, AI commits on master, AI commits off master, master share); section Developers With 0 Explicit AI Assistance (Developer, six-month commit count, recommendation priority); section Next Steps; and section Caveats.
* Quality bar: markdown numbers must reconcile with both CSVs; use normalized identities for developer reporting; distinguish all AI contributions vs merged-to-master AI contributions; remove temporary helper artifacts; validate outputs are readable and error-free.
* ai-adoption.md must include: Executive Judgment (moderate/low/high) with rationale; Method with exact filter terms, timeframe, lower-bound caveat, **merge strategy discovery notes**, and **detection strategy used** (explicit keywords only vs. multi-signal); section #2 What Actually Shipped To master with counts and table (Date, Author, SHA short, Subject, Dominant subsystem, Files changed, Squash Indicator, Signal Type); section #3 Developer And Subsystem Adoption with normalized contributor counts and subsystem distribution; section All AI Contributions Per Developer sourced from ai-adoption-by-developer.csv (Developer, Total AI commits, AI commits on master, AI commits off master, master share); section Developers With 0 Explicit AI Assistance (Developer, six-month commit count, recommendation priority); **section Squashed Commits Analysis** with summary of recovered AI signals, original vs. merge authors, and files impacted; section Setup Recommendations (e.g., "Add git hooks for zero-friction auto-tagging"; section Next Steps; and section Caveats.
* Quality bar: markdown numbers must reconcile with both CSVs and squashed commit inspection logs; use normalized identities for developer reporting; distinguish all AI contributions vs merged-to-master AI contributions; account for squashed commits when computing adoption rate; remove temporary helper artifacts; validate outputs are readable and error-free.
- For Vue-based UI repositories, expand the analysis phase to review template markup across all views, pages, and components against Chi component guidance (web components or HTML boilerplate) and describe non-compliant patterns in chi-compliance.md.
- When reporting Chi compliance, treat the version declared by the CDN-style script or stylesheet reference in the primary HTML entrypoint (for example, `<script src="https://lib.lumen.com/chi/5.78.0/js/chi.js" ...>`) as the authoritative design system version even if package.json or chi-vue dependencies specify different numbers.
- For Java repositories, audit logging frameworks and statements to highlight excessive verbosity, missing stack traces, or swallowed exceptions and document findings in logging-review.md.
- Ensure the scores calculated per category in the table match exactly with the scores reported per associated category detail deliverable.
- Ensure the scores calculated per category in the table match exactly with the scores reported per associated category detail deliverable.
- **Squashed Commits & Merge Strategy Handling**: When ai-adoption.csv includes squashed commits, clearly document the merge strategy observed in the repository (e.g., "squash-and-merge", "conventional merge", "rebase-and-merge"), the proportion of commits squashed vs. merged linearly, and any assumptions made during individual commit extraction. Update adoption rates in ai-adoption.md to reflect both direct signals and recovered signals from squashed commits, with separate line items for each category.
- **Zero-Friction AI Adoption Tracking**: To avoid burdening developers with manual tagging, prefer automated detection methods over explicit keywords. Order of detection preference: (1) GitHub Copilot metadata / API attribution, (2) Git hooks that auto-tag at commit time, (3) IDE/CLI integration auto-detection, (4) Implicit patterns in commit structure, (5) Explicit keywords as fallback. When setting up the analysis, check for existence of `.githooks/` or `.git/hooks/` directory and document any auto-tagging infrastructure. If no automation exists, recommend adding a `prepare-commit-msg` hook template to the repo README with instructions for developers to opt-in passively (hook runs automatically after commit, no manual action required). Report which detection methods were used and their coverage in the ai-adoption.md Method section.
validations:
required: true

Expand Down
144 changes: 144 additions & 0 deletions .github/ISSUE_TEMPLATE/create-copilot-instructions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,150 @@ body:
validations:
required: false

- type: textarea
id: ai-adoption-tracking
attributes:
label: AI Adoption Tracking Setup
description: Auto-tag AI-assisted commits for adoption metrics
value: |
Step 3 (Always Required) - Create AI Adoption Git Hook and Setup

Create the git hook file:
- **Location:** `.githooks/prepare-commit-msg`
- **Make executable:** `chmod +x .githooks/prepare-commit-msg`
- **Idempotent:** If the file already exists with correct content, skip creation. If outdated, replace it.

Hook Implementation
Create the file with the following content:

```bash
#!/bin/bash
# AI Adoption Signal Detection Hook
# Auto-tags commits assisted by GitHub Copilot or other AI tools
# Idempotent: safe to run on every commit

COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2

# Skip if commit is from merge, squash, rebase, or other special cases
if [ "$COMMIT_SOURCE" = "merge" ] || [ "$COMMIT_SOURCE" = "squash" ] || [ "$COMMIT_SOURCE" = "rebase" ]; then
exit 0
fi
Comment thread
gundersen-lumn marked this conversation as resolved.

# Read current commit message
COMMIT_MSG=$(cat "$COMMIT_MSG_FILE")

# Check for AI signals in multiple places
HAS_COPILOT_COAUTHOR=$(echo "$COMMIT_MSG" | grep -i "co-authored-by.*github copilot" | wc -l)
HAS_COPILOT_MARKER=$(echo "$COMMIT_MSG" | grep -iE "^(copilot|ai-assisted|@copilot):" | wc -l)
HAS_AI_PATTERN=$(echo "$COMMIT_MSG" | grep -iE "(copilot|claude|gpt|cursor|ai-assisted)" | wc -l)

# Idempotent check: don't re-tag if already tagged
if echo "$COMMIT_MSG" | head -1 | grep -q "^\[AI-ASSISTED\]"; then
exit 0
fi

# Auto-tag if any AI signal detected
if [ "$HAS_COPILOT_COAUTHOR" -gt 0 ] || [ "$HAS_COPILOT_MARKER" -gt 0 ] || [ "$HAS_AI_PATTERN" -gt 0 ]; then
FIRST_LINE=$(echo "$COMMIT_MSG" | head -1)
REST=$(echo "$COMMIT_MSG" | tail -n +2)

if [ -z "$REST" ]; then
echo "[AI-ASSISTED] $FIRST_LINE" > "$COMMIT_MSG_FILE"
else
echo "[AI-ASSISTED] $FIRST_LINE" > "$COMMIT_MSG_FILE"
echo "$REST" >> "$COMMIT_MSG_FILE"
fi
fi

exit 0
```

Step 3A - For Node.js Projects
If package.json exists, add these scripts to enable automatic hook setup on npm install:

```json
"scripts": {
"postinstall": "git config core.hooksPath .githooks",
"prepare": "git config core.hooksPath .githooks"
}
```

- `postinstall`: Runs after `npm install` installs dependencies
- `prepare`: Runs before `npm install` and also on `npm ci`
- Result: Developers get hook setup automatically with zero extra steps

Add to copilot-instructions.md for Node projects:
```markdown
## Git Hook Setup (Automatic)

When you run `npm install` or `npm ci`, the git hook path is automatically configured.

**Verify setup:**
```bash
git config core.hooksPath
```
Should output: `.githooks`

**What happens:** Commits using Copilot are automatically tagged with `[AI-ASSISTED]`.
```

Step 3B - For Non-Node Projects (Java, Python, etc.)
If package.json does NOT exist, create `.githooks/setup-dev-env.sh` as follows:

Comment thread
gundersen-lumn marked this conversation as resolved.
```bash
#!/bin/bash
set -e

echo "🔧 Setting up development environment..."

# Configure git hooks
git config core.hooksPath .githooks
echo "✓ Git hooks configured"

# Add project-specific setup here (database, environment variables, etc.)
# Example:
# if [ -f ".env.example" ]; then
# cp .env.example .env
# echo "✓ Environment file created"
# fi

echo "✅ Setup complete!"
echo ""
echo "To verify hook setup:"
echo " git config core.hooksPath"
```

Add to copilot-instructions.md for non-Node projects:
```markdown
## Development Setup

Before starting development, run the setup script once:

```bash
./setup-dev-env.sh
```

This script:
- Configures git hooks for AI adoption tracking
- [Add other setup steps specific to your project]

**What happens:** Commits using Copilot are automatically tagged with `[AI-ASSISTED]`.

**Verify setup:**
```bash
git config core.hooksPath
```
Should output: `.githooks`
```

Integration with Comprehensive Review
- The `comprehensive-codebase-review.yml` will detect `[AI-ASSISTED]` tags and other AI signals
- Tagged commits are counted in `copilot-eval/ai-adoption.md` with `signal_type = git_hook`
- Zero developer friction: hook auto-tags, no manual tagging required
validations:
required: true

- type: textarea
id: ai-attribution
attributes:
Expand Down
Loading