Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/aw/syntax-agentic.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,7 @@ description: Agentic workflow specific frontmatter fields for GitHub Agentic Wor
```

- `setup-steps`/`pre-steps` also apply to built-in jobs (e.g. `activation`): use `setup-steps` for OIDC/secret bootstrap that must run before framework token minting, then verify the result in `pre-steps`.
- **`needs`/`if` on built-in jobs** — targeting a compiler-generated job (`agent`, `activation`, `safe_outputs`, etc.) under `jobs:` also accepts additive `needs` and `if`: `jobs.agent.needs` merges with compiler-generated dependencies, and `jobs.agent.if` combines with compiler-generated conditions using `&&`. Use this to gate the agent job on a custom setup job's outcome.

- **`engine:`** - AI processor configuration (string or object: `id`, `model`, `permission-mode`, `agent`, `max-continuations`, `driver`, `copilot-sdk`, `auth`, and more). See [syntax-engine.md](syntax-engine.md) for the full field reference, per-engine support notes, and inline driver examples.

Expand Down
50 changes: 50 additions & 0 deletions .github/aw/token-optimization-observability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
description: OpenTelemetry export and harness-execution-experience learning loops for token optimization in GitHub Agentic Workflows.
---

# Token Optimization — Observability and Harness Learning

See [token-optimization.md](token-optimization.md) for the full technique index and quick-reference checklist.

## Technique 7 — Measure Continuously with OpenTelemetry and AgenticOps

Export telemetry automatically and add workflows that keep finding token waste over time.

### Enable OTLP export

Add workflow-level OpenTelemetry export so each run emits token and phase data to your observability backend:

```yaml
observability:
otlp:
endpoint: ${{ secrets.GH_AW_OTEL_ENDPOINT }}
headers: ${{ secrets.GH_AW_OTEL_HEADERS }}
```

Setup, agent, and conclusion spans carry token usage attributes. See [Frontmatter syntax](syntax-agentic.md#agentic-workflow-specific-fields).

### Add AgenticOps token workflows

- `copilot-token-audit` — scheduled audit of token usage across workflows
- `copilot-token-optimizer` — scheduled follow-up that identifies one expensive workflow and proposes concrete savings

Loop: export OTEL → summarize usage → open optimization issues → re-measure. See `.github/workflows/` for examples.

---

## Technique 8 — Learn from Harness Execution Experience

Treat the agent harness as six separate control surfaces rather than one prompt:

| Dimension | gh-aw control surface |
|---|---|
| Context assembly | Prompt structure, imports, DataOps, and context compression |
| Tool interaction | Tool selection, `gh-proxy`, `cli-proxy`, permissions, and result filtering |
| Generation control | Engine and model selection, `max-turns`, and `timeout-minutes` |
| Orchestration | Deterministic steps, sub-agents, planning, execution, and refinement |
| Memory management | `cache-memory`, `repo-memory`, summaries, and stale-context removal |
| Output processing | Safe outputs, schema validation, fallbacks, and `noop` behavior |

Start with the smallest known-good harness. Per experiment, record a compact entry (task features, config change, outcome quality, AIC/token cost, diagnosed failure dimension), distill repeated diagnoses into reusable patterns, and retrieve only relevant cases later instead of re-searching broadly. Select changes **correctness first**: maximize the quality metric, then minimize AIC among equivalent-quality variants so a cheap but degraded result cannot win.

Prioritize this for long-horizon, tool-heavy workflows with measurable headroom; keep retrieved experience compact so prompt caching offsets its input-token overhead. Based on [MemoHarness](https://arxiv.org/pdf/2607.14159) — treat its gains as directional (small held-out set, unablated components, cache-dependent cost advantage).
45 changes: 3 additions & 42 deletions .github/aw/token-optimization.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,48 +358,9 @@ Keep each batch idempotent, skip items already fixed, and report the processed s

---

## Technique 7Measure Continuously with OpenTelemetry and AgenticOps
## Techniques 7–8Observability and Harness Learning

Export telemetry automatically and add workflows that keep finding token waste over time.

### Enable OTLP export

Add workflow-level OpenTelemetry export so each run emits token and phase data to your observability backend:

```yaml
observability:
otlp:
endpoint: ${{ secrets.GH_AW_OTEL_ENDPOINT }}
headers: ${{ secrets.GH_AW_OTEL_HEADERS }}
```

Setup, agent, and conclusion spans carry token usage attributes. See [Frontmatter syntax](syntax-agentic.md#agentic-workflow-specific-fields).

### Add AgenticOps token workflows

- `copilot-token-audit` — scheduled audit of token usage across workflows
- `copilot-token-optimizer` — scheduled follow-up that identifies one expensive workflow and proposes concrete savings

Loop: export OTEL → summarize usage → open optimization issues → re-measure. See `.github/workflows/` for examples.

---

## Technique 8 — Learn from Harness Execution Experience

Treat the agent harness as six separate control surfaces rather than one prompt:

| Dimension | gh-aw control surface |
|---|---|
| Context assembly | Prompt structure, imports, DataOps, and context compression |
| Tool interaction | Tool selection, `gh-proxy`, `cli-proxy`, permissions, and result filtering |
| Generation control | Engine and model selection, `max-turns`, and `timeout-minutes` |
| Orchestration | Deterministic steps, sub-agents, planning, execution, and refinement |
| Memory management | `cache-memory`, `repo-memory`, summaries, and stale-context removal |
| Output processing | Safe outputs, schema validation, fallbacks, and `noop` behavior |

Start with the smallest known-good harness. Per experiment, record a compact entry (task features, config change, outcome quality, AIC/token cost, diagnosed failure dimension), distill repeated diagnoses into reusable patterns, and retrieve only relevant cases later instead of re-searching broadly. Select changes **correctness first**: maximize the quality metric, then minimize AIC among equivalent-quality variants so a cheap but degraded result cannot win.

Prioritize this for long-horizon, tool-heavy workflows with measurable headroom; keep retrieved experience compact so prompt caching offsets its input-token overhead. Based on [MemoHarness](https://arxiv.org/pdf/2607.14159) — treat its gains as directional (small held-out set, unablated components, cache-dependent cost advantage).
See [token-optimization-observability.md](token-optimization-observability.md) for OpenTelemetry export, AgenticOps token workflows, and learning from harness execution experience.

---

Expand All @@ -413,11 +374,11 @@ See [token-optimization-caching-budgets.md](token-optimization-caching-budgets.m

| Topic | File |
|---|---|
| OpenTelemetry export, AgenticOps, harness-experience learning | [token-optimization-observability.md](token-optimization-observability.md) |
| Prompt caching, AI-credit guardrails, bounded file reads | [token-optimization-caching-budgets.md](token-optimization-caching-budgets.md) |
| Inline sub-agents syntax | [subagents.md](subagents.md) |
| A/B experiments | [experiments.md](experiments.md) |
| Persistent memory | [memory.md](memory.md) |
| Experience-guided harness optimization | [MemoHarness](https://arxiv.org/pdf/2607.14159) |
| DataOps pattern | [DataOps guide](https://github.com/github/gh-aw/blob/main/docs/src/content/docs/patterns/data-ops.md) |
| Audit command reference | [cli-commands.md](cli-commands.md) |
| Frontmatter syntax | [syntax.md](syntax.md) |
Loading