Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,9 @@ index to jump to the right page.
- [`docs/safe-outputs.md`](docs/safe-outputs.md) — full reference for every
safe-output tool agents can use to propose actions (PRs, work items, wiki
pages, comments, etc.) plus their per-agent configuration.
- [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) — debug-only `ado-aw-debug:`
front-matter section (`skip-integrity`, `create-issue` for filing GitHub
issues from dogfood pipelines). NOT a regular safe-output.

### Compiler internals & operations

Expand Down
195 changes: 195 additions & 0 deletions docs/ado-aw-debug.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
# `ado-aw-debug:` — Debug-only front-matter section

_Part of the [ado-aw documentation](../AGENTS.md)._

> ⚠️ **This section is for dogfood pipelines only.**
> Anything declared under `ado-aw-debug:` is **not** part of the regular
> agent surface and is not recommended for general use. Knobs here exist so
> the team can validate `githubnext/ado-aw` changes against real Azure
> DevOps pipelines and file failures back to GitHub for triage. Each knob
> bypasses or weakens a normal safety control.

The compiler accepts a top-level `ado-aw-debug:` block in agent front
matter. Currently exposed knobs:

| Knob | Purpose | Default |
| --- | --- | --- |
| `skip-integrity` | Omit the "Verify pipeline integrity" step from the generated YAML. OR-ed with the `--skip-integrity` CLI flag. | `false` |
| `create-issue` | Enable the [debug-only `create-issue`](#create-issue) safe output. | absent (disabled) |

Unrecognised keys under `ado-aw-debug:` cause a compile-time error
(`#[serde(deny_unknown_fields)]`).

## `create-issue`

Files a GitHub issue against an operator-configured target repository.
Used to surface failures from ADO-hosted dogfood pipelines back to
`githubnext/ado-aw` for triage.

### Why it's gated

`create-issue` is **default-deny** at three layers:

1. **MCP layer.** The SafeOutputs MCP server lists `create-issue` in
[`DEBUG_ONLY_TOOLS`][debug-only-tools], so the route is removed from
the tool router unless the compiler explicitly opts in via
`--enabled-tools`.
2. **Compiler layer.** `--enabled-tools create-issue` is only emitted
when `ado-aw-debug.create-issue:` is present in front matter. The
compiler also rejects `safe-outputs.create-issue:` outright, so the
tool can't be smuggled in via the regular safe-outputs surface.
3. **Executor layer.** Stage 3 maintains a separate
`ExecutionContext.debug_enabled_tools` set populated only from
`ado-aw-debug:`. The executor refuses any NDJSON `create-issue`
entry that isn't in that set, so a forged or smuggled NDJSON entry
fails closed before any token is read.

[debug-only-tools]: ../src/safeoutputs/mod.rs

### Front-matter schema

```yaml
ado-aw-debug:
create-issue:
target-repo: githubnext/ado-aw # REQUIRED. Operator-only; agent has no override.
title-prefix: "[pipeline-failure] " # Optional; prepended to every agent title.
labels: # Optional; static labels always applied.
- pipeline-failure
- automated
allowed-labels: # Optional; default-deny — see below.
- "agent-*"
- "pipeline-failure"
assignees: # Optional; static assignees always applied.
- "jamesdevine"
max: 3 # Optional; per-run budget. Default 1.
```

* **`target-repo`** is required. Format `owner/repo`. The agent has no
parameter to override it; you cannot redirect issues to a different
repository at runtime.
* **`title-prefix`** is appended at execution time. The final title length
(prefix + agent title) must be ≤ 256 characters; longer titles fail at
Stage 3.
* **`labels`** are applied unconditionally to every issue, on top of any
agent-supplied labels that pass `allowed-labels`.
* **`allowed-labels`** is **default-deny**: an empty or absent list means
**no agent-supplied labels are accepted**. To accept any agent label,
set `allowed-labels: ["*"]` explicitly. Patterns may include `*`
wildcards (e.g. `"agent-*"`).
* **Allowed-label matching is case-insensitive.** It uses the same
`tag_matches_pattern` helper as ADO tag allow-lists. GitHub labels are
case-sensitive, so `allowed-labels: ["safe"]` will also admit
`SAFE` and `Safe` — keep that in mind when modelling policy.
* **`assignees`** are merged with agent-supplied assignees. There is
intentionally no `allowed-assignees` allowlist in v1; if you need
one, configure assignees only via the static `assignees:` list and
skip the agent parameter.
* **`max`** controls per-run budget the same way it does for other
safe-output tools.

### Agent-supplied parameters

The agent calls the `create-issue` MCP tool with:

```jsonc
{
"title": "Pipeline failure on main",
"body": "<markdown body, ≥ 30 chars>",
"labels": ["pipeline-failure"], // optional
"assignees": ["copilot"] // optional
}
```

The MCP-side `Validate` impl rejects ADO pipeline-command sequences in
labels and assignees (see [src/safeoutputs/create_issue.rs](../src/safeoutputs/create_issue.rs)).
Stage 3 also neutralises `##vso[…]` in any error messages it produces, so
agent-supplied content cannot escape the executor's stdout.

### Pipeline variable: `ADO_AW_DEBUG_GITHUB_TOKEN`

Stage 3 authenticates against GitHub using the
**`ADO_AW_DEBUG_GITHUB_TOKEN`** ADO pipeline variable. The compiler emits

```yaml
env:
SYSTEM_ACCESSTOKEN: $(SC_WRITE_TOKEN) # if write permissions: are set
ADO_AW_DEBUG_GITHUB_TOKEN: $(ADO_AW_DEBUG_GITHUB_TOKEN) # only when ado-aw-debug.create-issue is set
```

into the executor step's `env:` block. The token is **not** exposed to
the agent in Stage 1 — the read-only `GITHUB_TOKEN` the agent sees is a
separate variable wired through `engine.env` and used only for GitHub
MCP read access.

### Setting up the PAT

1. **Generate a fine-grained PAT** scoped to **only** `target-repo` (e.g.
`githubnext/ado-aw`). Required permissions:
* Repository access: only the target repo.
* Permissions: **Issues** = Read and write. Nothing else.
2. **Store as a secret pipeline variable** named exactly
`ADO_AW_DEBUG_GITHUB_TOKEN`. Mark it secret. Do **not** copy it into
`engine.env` or any non-secret variable.
3. **Confirm the operator-configured target-repo matches the PAT scope.**
The compiler validator only checks shape (`owner/repo`); it cannot
verify the PAT has access. If the PAT lacks Issues:write, the Stage 3
call fails with the GitHub API error and Stage 3 reports
`succeeded with issues`.
4. `ado-aw configure` does **not** automate this variable today — set it
manually in the ADO pipeline definition.

### Auto-footer

Every issue gets an auto-appended traceability footer that looks like:

```markdown
<!-- ado-aw -->
---
Pipeline: `dogfood-failure-reporter`
Run: <https://dev.azure.com/myorg/MyProject/_build/results?buildId=42>
Trigger: `Manual`
```

The `<!-- ado-aw -->` marker is stable so that future tooling can locate
the generated content without parsing prose. The footer is built from
`BUILD_BUILDID`, `BUILD_DEFINITIONNAME`, `BUILD_REASON`,
`SYSTEM_TEAMFOUNDATIONCOLLECTIONURI` and `SYSTEM_TEAMPROJECT` — these are
present whenever Stage 3 runs inside an ADO pipeline.

If your pipeline / org / project names are sensitive, do not enable
`create-issue` against a public repo.

### Security checklist

- [ ] Target repo's GitHub PAT is scoped to that repo only and only has
Issues:write.
- [ ] `ADO_AW_DEBUG_GITHUB_TOKEN` is stored as a secret pipeline
variable, never hard-coded or printed.
- [ ] `allowed-labels` is set explicitly. Empty means default-deny;
`["*"]` accepts any agent label — pick deliberately.
- [ ] `target-repo` is private if the agent's prompts or pipeline
metadata are sensitive (the auto-footer publishes ADO run URLs and
pipeline names).
- [ ] `skip-integrity` is **not** enabled in pipelines triggered by
untrusted PRs.

## `skip-integrity`

Equivalent to passing `--skip-integrity` on the `ado-aw compile` CLI.
Setting either OR setting both omits the `Verify pipeline integrity`
step from the generated YAML.

The integrity step downloads the same `ado-aw` binary the pipeline was
compiled with and runs `ado-aw check` against the committed pipeline
file. Without it, a tampered `*.yml` won't be caught at run time.

Use this only for short-lived dogfood pipelines where you're iterating
on the compiler and re-compiling frequently.

## See also

- [`docs/safe-outputs.md`](safe-outputs.md) — regular safe-outputs
surface (`create-issue` is **not** in it).
- [`docs/cli.md`](cli.md) — `--skip-integrity` CLI flag.
- [`docs/template-markers.md`](template-markers.md) — `{{ executor_ado_env }}`
and `{{ integrity_check }}` markers and their conditional behaviour.
2 changes: 1 addition & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Global flags (apply to all subcommands): `--verbose, -v` (enable info-level logg
- The agent auto-downloads the ado-aw compiler and handles the full lifecycle (create → compile → check)
- `compile [<path>]` - Compile a markdown file to Azure DevOps pipeline YAML. If no path is given, auto-discovers and recompiles all detected agentic pipelines in the current directory.
- `--output, -o <path>` - Optional output path for the generated YAML (only valid when a path is provided). If the path is an existing directory, the compiled YAML is written inside that directory using the default filename derived from the markdown source (e.g. `foo.md` → `<dir>/foo.lock.yml`).
- `--skip-integrity` - *(debug builds only)* Omit the "Verify pipeline integrity" step from the generated pipeline. Useful during local development when the compiled output won't match a released compiler version. This flag is not available in release builds.
- `--skip-integrity` - *(debug builds only)* Omit the "Verify pipeline integrity" step from the generated pipeline. Useful during local development when the compiled output won't match a released compiler version. This flag is not available in release builds. OR-ed with `ado-aw-debug.skip-integrity:` in front matter — either is sufficient. See [`docs/ado-aw-debug.md`](ado-aw-debug.md).
- `--debug-pipeline` - *(debug builds only)* Include MCPG debug diagnostics in the generated pipeline: `DEBUG=*` environment variable for verbose MCPG logging, stderr streaming to log files, and a "Verify MCP backends" step that probes each backend with MCP initialize + tools/list before the agent runs. This flag is not available in release builds.
- `check <pipeline>` - Verify that a compiled pipeline matches its source markdown
- `<pipeline>` - Path to the pipeline YAML file to verify
Expand Down
6 changes: 6 additions & 0 deletions docs/safe-outputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

_Part of the [ado-aw documentation](../AGENTS.md)._

> ℹ️ The debug-only `create-issue` tool (used by dogfood pipelines to file
> failure reports back to GitHub) is **not** a safe output and is not
> configurable here. It is gated by a separate `ado-aw-debug:` front-matter
> section and stripped from the SafeOutputs MCP server unless explicitly
> enabled. See [`docs/ado-aw-debug.md`](ado-aw-debug.md).

## Safe Outputs Configuration

The front matter supports a `safe-outputs:` field for configuring specific tool behaviors:
Expand Down
9 changes: 6 additions & 3 deletions docs/template-markers.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ Generates the "Verify pipeline integrity" pipeline step that downloads the relea

The step sets `workingDirectory: {{ trigger_repo_directory }}` so that the relative `{{ pipeline_path }}` argument resolves correctly when `checkout:` produces a multi-repo `$(Build.SourcesDirectory)` layout, and so `ado-aw check`'s internal recompile can infer the ADO org from the trigger repo's git remote.

When the compiler is built with `--skip-integrity` (debug builds only), this placeholder is replaced with an empty string and the integrity step is omitted from the generated pipeline.
When the compiler is built with `--skip-integrity` (debug builds only) **OR** when the agent's front matter sets `ado-aw-debug.skip-integrity: true`, this placeholder is replaced with an empty string and the integrity step is omitted from the generated pipeline. The two flags are OR-ed — either is sufficient. See [`docs/ado-aw-debug.md`](ado-aw-debug.md).

## {{ mcpg_debug_flags }}

Expand Down Expand Up @@ -444,9 +444,12 @@ If `permissions.write` is not configured, this marker is replaced with an empty

## {{ executor_ado_env }}

Generates the complete `env:` block (including the `env:` key) for the Stage 3 executor step when `permissions.write` is configured. Sets `SYSTEM_ACCESSTOKEN` to the write service connection token (`SC_WRITE_TOKEN`).
Generates the complete `env:` block (including the `env:` key) for the Stage 3 executor step. The block contains zero, one, or two lines depending on which features are configured:

If `permissions.write` is not configured, this marker is replaced with an empty string so that no `env:` block is emitted at all. Note: `System.AccessToken` is never used directly — all ADO tokens come from explicitly configured service connections.
* `SYSTEM_ACCESSTOKEN: $(SC_WRITE_TOKEN)` — emitted when `permissions.write` is configured. Provides the write-capable ADO token to the executor.
* `ADO_AW_DEBUG_GITHUB_TOKEN: $(ADO_AW_DEBUG_GITHUB_TOKEN)` — emitted when `ado-aw-debug.create-issue` is configured. Provides the GitHub PAT used by the debug-only `create-issue` safe output. See [`docs/ado-aw-debug.md`](ado-aw-debug.md).

If neither feature is configured, this marker is replaced with an empty string so that no `env:` block is emitted at all. Note: `System.AccessToken` is never used directly — all ADO tokens come from explicitly configured service connections, and the GitHub PAT is sourced from a dedicated pipeline variable separate from the read-only `GITHUB_TOKEN` the agent sees in Stage 1.

## {{ compiler_version }}

Expand Down
51 changes: 51 additions & 0 deletions examples/dogfood-failure-reporter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
name: "Dogfood Failure Reporter"
description: "Files a GitHub issue on githubnext/ado-aw when a dogfood pipeline run fails"
on:
schedule: daily
permissions:
read: my-read-arm-connection
ado-aw-debug:
create-issue:
target-repo: githubnext/ado-aw
title-prefix: "[pipeline-failure] "
labels:
- pipeline-failure
- automated
allowed-labels:
- "agent-*"
- "pipeline-failure"
assignees:
- jamesdevine
max: 3
---

## Dogfood Failure Reporter

You are a dogfood failure-reporting agent for `githubnext/ado-aw`. You run
in Azure DevOps inside an AWF-isolated sandbox.

### Tasks

1. Read the pipeline run logs available under `$BUILD_SOURCESDIRECTORY`
for any signs of recent failures.
2. For each distinct failure, file **one** GitHub issue using the
`create-issue` MCP tool with:
- A concise `title` describing the failure.
- A markdown `body` with reproduction steps, log excerpts, and links
to relevant ADO build URLs.
- `labels: ["pipeline-failure"]` (must match the `allowed-labels` allowlist
configured by the operator).
3. Limit yourself to **at most 3** issues per run (the `max` budget).
4. If you cannot file an issue (e.g., the failure isn't reproducible),
call `report-incomplete` instead — do **not** invent details.

### Important

- Do not attempt to redirect issues to a different repository — the agent
has no `target_repo` parameter and the target is fixed by the operator.
- The `ADO_AW_DEBUG_GITHUB_TOKEN` PAT is **not** visible to you; it is
used only by Stage 3 to authenticate against GitHub.
- Issues are reviewed for prompt injection by Stage 2 before they are
filed, so do not include text that looks like ADO pipeline commands
(`##vso[...]`) — they will be flagged and the run rejected.
Loading
Loading