Document custom job dependencies (needs) for agentic workflows

[Copilot]

Investigation Complete: Matrix Jobs and Job Dependencies for Project Boards and Orchestration

Summary

Finding: The needs field for job dependencies is already fully supported. Matrix jobs are not beneficial for project boards or orchestration patterns due to their dynamic, agentic nature.

Key Discoveries

• ✅ needs field fully functional - custom jobs can define complex dependency chains
• ⚠️ Matrix strategy in schema but unused - not processed by compiler
• ❌ Matrix jobs not suitable - projectops/orchestration require runtime decisions
• 📝 Documentation gap identified - users unaware of needs support
• ✅ Test validation complete - dependencies compile and execute correctly
• ✅ All documentation updated - including clear guidance on when to use needs vs orchestration

Changes Made

1. Documentation Enhancements ✅

• docs/src/content/docs/reference/frontmatter.md

  • Added "Job Dependencies (needs)" section with examples
  • Documented multi-dependency syntax: needs: [job1, job2]
  • Showed job output references: ${{ needs.job.outputs.value }}
  • Added best practices and cycle detection notes

• docs/src/content/docs/patterns/orchestration.md

  • Added "Dynamic vs. Static Parallelization" section
  • Explained why matrix jobs don't fit agentic workflows
  • Clarified when to use custom jobs with dependencies

• .github/aw/orchestration.md

  • Added "When Execution Order Matters" section
  • Clarified when to use needs (deterministic single-run order) vs dispatch-workflow (dynamic multi-run orchestration)
  • Simplified examples to show pre/post-processing pattern
  • Contrasts fixed-order dependencies with dynamic orchestration decisions

2. Test Validation ✅

• .github/workflows/test-custom-jobs-needs.md
  • Demonstrates complex job dependency chains
  • Shows multiple dependencies: needs: [agent, collect_data]
  • Tests job output passing between jobs
  • Verified compilation and correct dependency ordering

3. Investigation Report ✅

• MATRIX_JOBS_INVESTIGATION.md
  • Complete analysis of matrix vs needs capabilities
  • Use case assessment for projectops and orchestration
  • Recommendations with rationale
  • Documentation of changes made

When to Use Each Pattern

Use needs for deterministic execution order (single workflow run):

• Pre-processing steps must complete before agent (data fetch, linting, static analysis)
• Post-processing requires outputs from agent and other jobs (reporting, cleanup)
• Execution order is fixed and known at workflow design time

Use dispatch-workflow for dynamic orchestration (multiple workflow runs):

• Orchestrator makes runtime decisions about which workers to dispatch
• Workers can update Projects to coordinate state across runs
• Each workflow run is independent and can be retried
• Number and type of workers depends on runtime analysis

Dependency Graph from Test Workflow

pre_activation
    ↓
activation
    ↓
    ├─→ collect_data (outputs: item_count)
    │       ↓
    │   process_data (uses collect_data output)
    │       ↓
    └─→ agent
            ↓
        report (needs: [agent, collect_data], if: always())

Why Matrix Jobs Are Not Needed

ProjectOps: Content-driven decisions require dynamic routing

• Agent analyzes issue/PR content → decides which projects to update
• Matrix requires predefined static combinations
• Current pattern (update_project with max) is correct

Orchestration: Intelligent worker dispatch based on analysis

• Orchestrator discovers work items at runtime → dispatches appropriate workers
• Matrix would force premature static decisions
• Current pattern (dispatch-workflow with max) is correct

Bottom Line

✅ needs already works - now documented with clear guidance on when to use it
❌ Matrix not needed - incompatible with agentic patterns
✅ Current patterns correct - dynamic decision-making preserved
✅ No code changes required - feature already exists and functions properly

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[mnkiefer]

@copilot update aw/orchestration.md to be aware that needs syntax for dependency chains can be used.

[Copilot]

@copilot update aw/orchestration.md to be aware that needs syntax for dependency chains can be used.

Updated .github/aw/orchestration.md with a new "Custom Job Dependencies" section that documents needs syntax for dependency chains. Includes examples of single dependencies, multiple dependencies, job outputs, and key capabilities. Commit: a9e5817

[mnkiefer]

@copilot Maybe this should be rephrased as for orchestration where order matters? Or should be solved more naturally by orchestration cycles/runs?

[Copilot]

@copilot Maybe this should be rephrased as for orchestration where order matters? Or should be solved more naturally by orchestration cycles/runs?

Rephrased the section to focus on "When Execution Order Matters" - clarified that needs is for deterministic single-run ordering (pre/post-processing), while dispatch-workflow is for dynamic multi-run orchestration. The section now contrasts when to use each pattern. Commit: a2ea4c9