Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion docs/src/content/docs/guides/campaigns.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ When the spec includes orchestration, the tooling generates an orchestrator work

**Note:** During compilation, a `.campaign.g.md` file is generated locally as a debug artifact to help developers understand the orchestrator structure, but this file is not committed to git—only the source `.campaign.md` and compiled `.campaign.lock.yml` are tracked.

### Orchestrator architecture

The generated orchestrator workflow consists of two key components:

1. **Discovery precomputation step**: Runs before the agent to gather candidate items from GitHub (issues, PRs, discussions) and writes a normalized manifest file (`.gh-aw/campaign.discovery.json`). This step uses GitHub Script to efficiently query the GitHub API with pagination budgets and cursor-based incremental discovery.

2. **Agent coordination job**: Reads the discovery manifest, makes deterministic decisions about project updates, and executes writes in a separate phase. The agent never performs GitHub-wide searches directly—it only processes the precomputed manifest.

This architecture reduces API traffic, enables deterministic processing with rate limit controls, and ensures consistent campaign coordination across runs.

### Mental model

```mermaid
Expand Down Expand Up @@ -66,17 +76,38 @@ flowchart TB

## How it works

Most campaigns follow the same shape. The GitHub Project is the human-facing status view and the canonical source of campaign membership. The orchestrator workflow discovers tracked items from the workers and updates the Project. Worker workflows do the real work, such as opening pull requests or applying fixes but they stay campaign-agnostic. If you want cross-run discovery of worker-created assets, workers can include a `tracker-id` marker which the orchestrator can search for. Optionally, you can configure a tracker label (e.g., `campaign:<id>`) as an ingestion hint to help discover issues and PRs created by workers.
Most campaigns follow the same shape. The GitHub Project is the human-facing status view and the canonical source of campaign membership. The orchestrator workflow has two phases:

1. **Discovery precomputation**: A dedicated GitHub Actions step queries GitHub for candidate items (using tracker label searches and workflow run queries), applies pagination budgets, and writes a normalized manifest (`.gh-aw/campaign.discovery.json`). This manifest contains all discovered items with metadata like URLs, content types, timestamps, and state.

2. **Agent coordination**: The agent reads the discovery manifest, makes deterministic decisions about which items need to be added or updated on the project board, and executes writes using the `update-project` safe output. The agent follows strict phases (read, plan, write, report) to ensure idempotent operation.

Worker workflows do the real work, such as opening pull requests or applying fixes, but they stay campaign-agnostic. Workers can include a `tracker-id` marker in their outputs, and you can configure a tracker label (e.g., `campaign:<id>`) as an ingestion hint to help discover issues and PRs created by workers.

## Memory

Campaigns become repeatable when they also write durable state to repo-memory (a git branch used for snapshots). The recommended layout is `memory/campaigns/<campaign-id>/cursor.json` for the checkpoint (treated as an opaque JSON object) and `memory/campaigns/<campaign-id>/metrics/<date>.json` for append-only metrics snapshots.

Campaign tooling enforces this durability contract at push time: when a campaign writes repo-memory, it must include a cursor and at least one metrics snapshot.

### Cursor-based incremental discovery

The orchestrator uses cursor-based incremental discovery to avoid rescanning the same items repeatedly. The cursor file (`cursor.json`) stores the last processed item boundary (typically an `updatedAt` timestamp and item ID). On subsequent runs, the discovery precomputation step reads this cursor and continues from where it left off, using deterministic ordering (e.g., oldest `updatedAt` first, tie-break by ID).

### Metrics snapshots

Each orchestrator run writes a new metrics snapshot file with the UTC date in the filename (e.g., `metrics/2025-01-05.json`). These snapshots are append-only and must include required fields:
- `campaign_id`: The campaign identifier
- `date`: UTC date in YYYY-MM-DD format
- `tasks_total`: Total number of tasks (>= 0)
- `tasks_completed`: Completed task count (>= 0)

Optional fields include `tasks_in_progress`, `tasks_blocked`, `velocity_per_day`, and `estimated_completion`.

## Next steps

- [Getting started](/gh-aw/guides/campaigns/getting-started/) – create a campaign quickly
- [Campaign specs](/gh-aw/guides/campaigns/specs/) – spec fields (objective/KPIs, governance, memory)
- [Project management](/gh-aw/guides/campaigns/project-management/) – project board setup tips
- [CLI commands](/gh-aw/guides/campaigns/cli-commands/) – CLI reference
- [Improvements & future directions](/gh-aw/guides/campaigns/improvements/) – enhancement opportunities for reporting and learning systems
10 changes: 9 additions & 1 deletion docs/src/content/docs/guides/campaigns/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,19 @@ gh aw compile

This validates the spec. When the spec has meaningful details (tracker label, workflows, memory paths, or a metrics glob), `compile` also generates an orchestrator and compiles it to `.campaign.lock.yml`.

The orchestrator workflow consists of:
1. **Discovery precomputation step**: Queries GitHub for candidate items and writes a normalized manifest
2. **Agent coordination job**: Reads the manifest and updates the project board

**Note:** A `.campaign.g.md` file is generated locally as a debug artifact to help you understand the orchestrator structure, but this file is not committed to git—only the compiled `.campaign.lock.yml` is tracked.

## 4) Run the orchestrator

Trigger the orchestrator workflow from GitHub Actions to keep the dashboard in sync. It finds tracker-labeled issues/PRs, adds them to the Project, updates fields/status, and posts a report.
Trigger the orchestrator workflow from GitHub Actions. Its job is to keep the dashboard in sync:

1. **Discovery precomputation**: Queries GitHub for items with the tracker label and writes a manifest
2. **Agent coordination**: Reads the manifest, determines what needs updating, and updates the project board
3. **Reporting**: Reports counts of items discovered, processed, and deferred

## 5) Add work items

Expand Down
252 changes: 252 additions & 0 deletions docs/src/content/docs/guides/campaigns/improvements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
---
title: "Campaign Improvements & Future Directions"
description: "Recommendations for enhancing campaign reporting, learning systems, and orchestrator capabilities"
---

This document outlines opportunities to improve campaign functionality, particularly around summarized reporting, learning from campaign outcomes, and better integration between orchestrators and metrics.

## Current State

The campaign system provides:
- Discovery precomputation for efficient item tracking
- Cursor-based incremental processing
- Metrics snapshots written to repo-memory
- Project board synchronization
- Rate limit controls and governance policies

## Improvement Opportunities

### 1. Summarized Campaign Reports

**Current Limitation**: Campaign orchestrators write individual metrics snapshots but don't generate human-readable progress summaries.

**Proposed Enhancement**: Add a summary report generation capability to orchestrators that:

- **Aggregates metrics across runs**: Read multiple metrics snapshots from repo-memory and calculate trends
- **Generates markdown reports**: Create formatted markdown summaries with:
- Current campaign status (tasks completed, in progress, blocked)
- Velocity trends (tasks per day over time)
- KPI progress toward targets
- Top contributors (workflows with most completed items)
- Blockers and risks (items stuck in certain states)
- **Posts to Epic issue**: Automatically update the campaign Epic issue with the latest summary as a comment
- **Configurable frequency**: Allow campaigns to specify report frequency (e.g., weekly, every 10 runs)

**Example Report Format**:

```markdown
## Campaign Progress Report (2025-01-05)

**Overall Status**: 🟢 On track

### Metrics Summary
- **Tasks Completed**: 45 / 200 (22.5%)
- **Current Velocity**: 7.5 tasks/day
- **Estimated Completion**: 2025-02-12 (38 days remaining)

### This Week's Progress
- ✅ 52 new tasks completed (+15%)
- 🚧 8 items in progress
- 🚫 3 items blocked (down from 5)

### Worker Activity
- `migration-worker`: 28 completed (top contributor)
- `daily-doc-updater`: 12 completed
- `unbloat-docs`: 7 completed

### KPI Progress
- **Services upgraded**: 45 / 50 target (90%) ⬆️
- **Incidents caused**: 0 / 0 target (✅ met)

### Blockers Resolved This Week
- Fixed API rate limit issue in discovery
- Unblocked 2 items waiting for external reviews
```

### 2. Campaign Learning System

**Current Limitation**: Campaigns don't capture or share learnings across runs or between campaigns.

**Proposed Enhancement**: Implement a structured learning system that:

- **Captures common patterns**: Identify recurring issues, successful strategies, and anti-patterns
- **Stores learnings in repo-memory**: Add `memory/campaigns/<id>/learnings.json` with structured insights
- **Shares learnings across campaigns**: Enable campaigns with similar objectives to reference learnings from completed campaigns
- **Surfaces recommendations**: Orchestrators can suggest improvements based on historical data

**Learning Categories**:

1. **Discovery Efficiency**
- Optimal pagination budgets for different campaign scales
- Most effective tracker label patterns
- API rate limit patterns and mitigation strategies

2. **Worker Performance**
- Average completion time per workflow
- Success rates and common failure modes
- Optimal scheduling for different workflows

3. **Project Management**
- Field usage patterns (which fields are most valuable)
- View configurations that work best
- Status transition patterns (typical item lifecycle)

4. **Governance Tuning**
- Effective rate limit configurations
- Optimal max-items-per-run values
- Successful opt-out label strategies

**Example Learning Entry**:

```json
{
"campaign_id": "docs-quality-maintenance-project73",
"date": "2025-01-05",
"category": "discovery_efficiency",
"insight": "Increased max-discovery-pages-per-run from 5 to 10",
"impact": "Reduced average discovery time by 40%, improved cursor freshness",
"recommendation": "For campaigns with >100 tracked items, start with 10 pages minimum"
}
```

### 3. Enhanced Metrics Integration

**Current Limitation**: Metrics are written but not actively used by orchestrators for decision-making.

**Proposed Enhancement**: Enable orchestrators to read and act on historical metrics:

- **Adaptive rate limiting**: Adjust discovery budgets based on recent velocity trends
- **Dynamic prioritization**: Focus on blocked items when velocity drops
- **Anomaly detection**: Alert when completion rate deviates significantly from trends
- **Capacity planning**: Estimate required orchestrator run frequency to meet targets

**Example Decision Logic**:

```yaml
# If velocity drops below 50% of average, increase discovery budget
if current_velocity < (avg_velocity * 0.5):
max_discovery_items = max_discovery_items * 1.5

# If >20% of items are blocked, prioritize unblocking
if blocked_percentage > 0.2:
focus_on_blocked = true
```

### 4. Campaign Retrospectives

**Current Limitation**: No structured retrospective process when campaigns complete.

**Proposed Enhancement**: Add campaign completion workflow that:

- **Analyzes final metrics**: Calculate success rate against KPIs
- **Generates retrospective report**: Document what worked, what didn't, and recommendations
- **Archives learnings**: Store insights in `memory/campaigns/<id>/retrospective.json`
- **Updates campaign state**: Mark campaign as `completed` with final outcomes

**Retrospective Template**:

```markdown
# Campaign Retrospective: {{ campaign_name }}

## Objectives vs Outcomes
- **Target**: {{ target_description }}
- **Achieved**: {{ actual_outcome }}
- **Success Rate**: {{ percentage }}%

## Timeline
- **Planned Duration**: {{ planned_days }} days
- **Actual Duration**: {{ actual_days }} days
- **Variance**: {{ variance }}% {{ ahead/behind }}

## What Worked Well
- {{ success_factor_1 }}
- {{ success_factor_2 }}

## Challenges Encountered
- {{ challenge_1 }}: {{ resolution }}
- {{ challenge_2 }}: {{ resolution }}

## Recommendations for Future Campaigns
1. {{ recommendation_1 }}
2. {{ recommendation_2 }}

## Worker Performance Summary
| Worker | Items Completed | Avg Time | Success Rate |
|--------|----------------|----------|--------------|
| {{ worker_1 }} | {{ count }} | {{ time }} | {{ rate }}% |
```

### 5. Cross-Campaign Analytics

**Current Limitation**: No visibility across multiple campaigns for portfolio management.

**Proposed Enhancement**: Add a campaign analytics dashboard that:

- **Aggregates metrics across campaigns**: Show portfolio-level health
- **Identifies patterns**: Highlight common blockers, top-performing workflows
- **Enables comparison**: Compare similar campaigns' performance
- **Supports resource allocation**: Help prioritize which campaigns need attention

**Dashboard Metrics**:

- Total active campaigns
- Overall completion rate
- Average velocity across campaigns
- Top blockers affecting multiple campaigns
- Worker utilization across campaigns

## Implementation Priority

**High Priority** (Immediate Value):
1. Summarized campaign reports (Epic issue updates)
2. Enhanced metrics integration (adaptive rate limiting)

**Medium Priority** (Near-term):
3. Campaign learning system (structured insights)
4. Campaign retrospectives (completion workflow)

**Low Priority** (Future):
5. Cross-campaign analytics (portfolio dashboard)

## Configuration Examples

### Enable Summarized Reporting

```yaml
# .github/workflows/my-campaign.campaign.md
governance:
# ... existing governance ...
reporting:
enabled: true
frequency: 10 # Generate report every 10 runs
format: "markdown"
post-to-epic: true
```

### Enable Learning Capture

```yaml
# .github/workflows/my-campaign.campaign.md
learning:
enabled: true
categories:
- discovery_efficiency
- worker_performance
- governance_tuning
share-with-campaigns:
- "similar-campaign-*" # Share learnings with similar campaigns
```

## Next Steps

To implement these improvements:

1. **Start with metrics aggregation**: Build utility functions to read and analyze historical metrics snapshots
2. **Add report generation**: Create markdown report templates and integrate with Epic issue comments
3. **Implement learning capture**: Define learning schema and storage format
4. **Build retrospective workflow**: Create workflow that triggers on campaign completion
5. **Design analytics dashboard**: Plan portfolio-level metrics and visualization

## Feedback Welcome

These are proposed enhancements based on analysis of current campaign architecture. Feedback and additional ideas are welcome—please open an issue or discussion to share your thoughts.
22 changes: 19 additions & 3 deletions docs/src/content/docs/guides/campaigns/specs.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,14 @@ description: "Move services to Framework vNext"
project-url: "https://github.com/orgs/ORG/projects/1"
tracker-label: "campaign:framework-upgrade"

# Optional: Custom GitHub token for Projects v2 operations
# project-github-token: "${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}"

objective: "Upgrade all services to Framework vNext with zero downtime."
kpis:
- id: services_upgraded
name: "Services upgraded"
primary: true
priority: primary
direction: "increase"
target: 50
- id: incidents
Expand All @@ -53,9 +56,10 @@ owners:

- `id`: stable identifier used for file naming, reporting, and (if used) repo-memory paths.
- `project-url`: the GitHub Project that acts as the campaign dashboard and canonical source of campaign membership.
- `tracker-label` (optional): an ingestion hint label that helps discover issues and pull requests created by workers (commonly `campaign:<id>`). When provided, the orchestrator can discover work across runs. The project board remains the canonical source of truth.
- `project-github-token` (optional): a GitHub token expression (e.g., `${{ secrets.GH_AW_PROJECT_GITHUB_TOKEN }}`) used for GitHub Projects v2 operations. When specified, this token is passed to the `update-project` safe output configuration in the generated orchestrator workflow. Use this when the default `GITHUB_TOKEN` doesn't have sufficient permissions for project board operations.
- `tracker-label` (optional): an ingestion hint label that helps discover issues and pull requests created by workers (commonly `campaign:<id>`). When provided, the orchestrator's discovery precomputation step can discover work across runs. The project board remains the canonical source of truth.
- `objective`: a single sentence describing what “done” means.
- `kpis`: the measures you use to report progress (exactly one should be marked `primary`).
- `kpis`: the measures you use to report progress. Use `priority: primary` to mark exactly one KPI as the primary measure (not `primary: true`).
- `workflows`: the participating workflow IDs. These refer to workflows in the repo (commonly `.github/workflows/<workflow-id>.md`), and they can be scheduled, event-driven, or long-running.

## KPIs (recommended shape)
Expand Down Expand Up @@ -100,6 +104,18 @@ governance:
max-comments-per-run: 10
```

### Governance fields

- `max-new-items-per-run`: Maximum number of new items to add to the project board per run (applies to agent write phase)
- `max-discovery-items-per-run`: Maximum number of candidate items the discovery precomputation step will scan per run (default: 100)
- `max-discovery-pages-per-run`: Maximum number of API result pages the discovery step will fetch per run (default: 10)
- `opt-out-labels`: Labels that exclude an item from campaign tracking
- `do-not-downgrade-done-items`: Prevent moving items backwards from "Done" status
- `max-project-updates-per-run`: Maximum number of project board updates per run (default: 10)
- `max-comments-per-run`: Maximum number of comments the orchestrator can post per run (default: 10)

These governance policies are enforced during the discovery precomputation step (for read budgets) and during the agent coordination phase (for write budgets), ensuring sustainable API usage and manageable workload.

## Compilation and orchestrators

`gh aw compile` validates campaign specs. When the spec has meaningful details (tracker label, workflows, memory paths, or a metrics glob), it also generates an orchestrator and compiles it to `.campaign.lock.yml`.
Expand Down
Loading