Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions actions/setup/js/codex_harness.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -858,6 +858,10 @@ process.exit(1);`,
});

describe("resolvePostResultWatchdogIdleTimeoutMs", () => {
it("uses a 2-minute shared default", () => {
expect(DEFAULT_POST_RESULT_WATCHDOG_IDLE_TIMEOUT_MS).toBe(120000);
});

it("returns the default when no env var is set", () => {
expect(resolvePostResultWatchdogIdleTimeoutMs({})).toBe(DEFAULT_POST_RESULT_WATCHDOG_IDLE_TIMEOUT_MS);
});
Expand Down
2 changes: 1 addition & 1 deletion actions/setup/js/process_runner.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ function runProcess({ command, args, attempt, log, logArgs, env, postResultWatch
// Post-result watchdog: shared constants and timeout resolver used by all harnesses.
// These are kept here so both copilot_harness and codex_harness stay in sync.
const MIN_POST_RESULT_WATCHDOG_TIMEOUT_MS = 50;
const DEFAULT_POST_RESULT_WATCHDOG_IDLE_TIMEOUT_MS = 20 * 1000;
const DEFAULT_POST_RESULT_WATCHDOG_IDLE_TIMEOUT_MS = 2 * 60 * 1000;
/** Maximum allowed value for GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS to prevent the watchdog from being
* effectively disabled by an excessively large override (e.g. a stray zero). */
const MAX_POST_RESULT_WATCHDOG_TIMEOUT_MS = 10 * 60 * 1000;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# ADR-51292: Increase Harness Watchdog Default and Add Frontmatter Timeout Control

**Date**: 2026-08-08
**Status**: Draft
**Deciders**: pelikhan

---

### Context

The post-result harness watchdog kills any engine process that is silent (no stdio) for longer than a configured idle timeout after it has emitted a terminal safe output. The previous default was 20 seconds. In practice, large-repo workflows commonly pass through quiet shell phases (package installs, builds, index refreshes) that comfortably exceed 20 seconds without producing output, causing the watchdog to send a premature SIGTERM and leave runs incomplete.

The only existing override surface was `GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` set via `engine.env`, which requires the caller to know the env var name, know the unit is milliseconds, and accept that the override is indistinguishable from other env-var customizations. This surface is not surfaced in schema validation or reference docs in a discoverable way.

### Decision

We will raise `DEFAULT_POST_RESULT_WATCHDOG_IDLE_TIMEOUT_MS` from 20 seconds (20 000 ms) to 2 minutes (120 000 ms) in `process_runner.cjs` — the shared location read by all harnesses — and we will add `engine.harness.watchdog-timeout` as a first-class frontmatter integer field (unit: seconds) that compiles to `GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` (unit: milliseconds). The new field sits alongside the existing retry-policy fields under `engine.harness`, follows the same literal-or-expression pattern, and is validated by the main workflow JSON schema.

### Alternatives Considered

#### Alternative 1: Raise the default only, no new frontmatter field

Increasing the default to 120 s without adding a frontmatter override surface would fix the most common regression at minimal API cost. Workflows with genuine requirements for a shorter or longer timeout would still have to fall back to the raw `engine.env.GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` override, with its undiscoverable name and millisecond unit. This option trades surface simplicity for discoverability and per-workflow configurability.

#### Alternative 2: Keep the 20 s default, improve env-var documentation only

Documenting `GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` more prominently in reference docs and adding schema validation for it as an `engine.env` value would let advanced users tune the timeout without changing the default. This avoids the risk that a longer default masks genuinely stuck processes, but does not address the root cause: the 20 s window is too short for routine large-repo operations and results in hard-to-diagnose incomplete runs.

### Consequences

#### Positive
- Premature watchdog kills during legitimate quiet phases (installs, builds, index refreshes) are eliminated for typical large-repo workflows under the new 2-minute default.
- Workflow authors can tune per-workflow watchdog behavior via a discoverable, schema-validated frontmatter field without memorising env var names or unit conversions.
- Explicit `engine.env.GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` overrides continue to take precedence, preserving backward compatibility for callers already using the env-var surface.

#### Negative
- A genuinely stuck post-result process now takes up to 2 minutes (instead of 20 seconds) to be forcibly terminated under the new default, increasing the worst-case cost of a hung run.
- Adding `watchdog-timeout` to `engine.harness` extends the frontmatter API surface; future breaking changes to this field require a deprecation path.

#### Neutral
- The `watchdog-timeout` frontmatter field accepts seconds; the runtime converts to milliseconds before injecting `GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS`. This unit difference is intentional (seconds are more human-readable in config) and is documented in schema and reference docs.
- The existing `MIN_POST_RESULT_WATCHDOG_TIMEOUT_MS` (50 ms) and `MAX_POST_RESULT_WATCHDOG_TIMEOUT_MS` (10 min) guards are unchanged and continue to prevent obviously invalid overrides.

---

*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*
10 changes: 7 additions & 3 deletions docs/src/content/docs/reference/engines.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ The `use` value must be a bare filename — no directory separators, no `..`, an
| Must start with `[A-Za-z0-9_]` | `harness.js` | `-harness.cjs` |
| Must end with `.js`, `.cjs`, or `.mjs` | `wrapper.cjs` | `harness.sh` |

### Harness Retry Policy
### Harness Retry and Post-result Watchdog Policy

The built-in Copilot, Claude, and Codex harnesses default to **3 retries** after the initial run (4 total attempts), with exponential backoff starting at 5 s (capped at 60 s). Use sub-keys under `engine.harness` to widen the retry window without replacing the harness:

Expand All @@ -355,18 +355,22 @@ engine:
initial-delay-ms: 10000
backoff-multiplier: 2
max-delay-ms: 180000
watchdog-timeout: 120
```

All four fields accept a literal integer or a GitHub Actions expression (e.g. `${{ vars.MY_RETRIES }}`):
All five fields accept a literal integer or a GitHub Actions expression (e.g. `${{ vars.MY_RETRIES }}`).
For `watchdog-timeout`, the value is treated as seconds when it is a literal integer.
When an expression is used, it must already be in milliseconds (GitHub Actions expressions do not support arithmetic operators):

| Sub-key | Default | Description |
|---|---|---|
| `max-retries` | `3` | Maximum retry attempts after the initial run (0 = no retries) |
| `initial-delay-ms` | `5000` | Delay in ms before the first retry |
| `backoff-multiplier` | `2` | Multiplier applied to the delay after each retry |
| `max-delay-ms` | `60000` | Maximum delay cap in ms |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] watchdog-timeout uses seconds while every other engine.harness field uses milliseconds (initial-delay-ms, max-delay-ms, backoff-multiplier). This unit inconsistency is a user-facing footgun.

💡 Discussion

All sibling fields in the same engine.harness block use milliseconds. A user copying the retry-policy pattern might write watchdog-timeout: 120000 expecting 120 s, but would get 33 hours.

Consider:

  1. Rename to watchdog-timeout-ms (aligns with the injected env var GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS) — no s→ms conversion needed, no unit mismatch.
  2. Or keep seconds but rename to watchdog-timeout-seconds or watchdog-timeout-s to make the unit explicit.

If the seconds choice is intentional (human-friendly), the docs and schema description should explicitly warn about the unit difference from the other fields.

@copilot please address this.

| `watchdog-timeout` | `120` | Post-result idle watchdog timeout in seconds before terminating a quiet process |

You can also set the underlying `GH_AW_HARNESS_*` env vars directly via `engine.env` when you need expression-level control. Explicit `engine.env` values take precedence over `engine.harness` sub-key values.
You can also set the underlying `GH_AW_HARNESS_*` env vars directly via `engine.env` when you need expression-level control, including `GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS` for the post-result watchdog. Explicit `engine.env` values take precedence over `engine.harness` sub-key values.

### Copilot SDK Support

Expand Down
11 changes: 11 additions & 0 deletions docs/src/content/docs/reference/frontmatter-full.md
Original file line number Diff line number Diff line change
Expand Up @@ -2495,6 +2495,17 @@ engine:
# Format 2: string
max-delay-ms: "example-value"

# Post-result idle watchdog timeout in seconds. Accepts a literal integer or a
# GitHub Actions expression.
# (optional)
# Accepted formats:

# Format 1: integer
watchdog-timeout: 1

# Format 2: string
watchdog-timeout: "example-value"

# Custom environment variables to pass to the AI engine, including secret
# overrides (e.g., OPENAI_API_KEY: ${{ secrets.CUSTOM_KEY }})
# (optional)
Expand Down
34 changes: 34 additions & 0 deletions pkg/parser/schema_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,40 @@ func TestValidateMainWorkflowFrontmatterWithSchemaAndLocation_EngineHarnessPatte
}
}

func TestValidateMainWorkflowFrontmatterWithSchemaAndLocation_EngineHarnessWatchdogTimeout(t *testing.T) {
t.Parallel()

validFrontmatter := map[string]any{
"on": "push",
"engine": map[string]any{
"id": "copilot",
"harness": map[string]any{
"watchdog-timeout": 120,
},
},
}

err := ValidateMainWorkflowFrontmatterWithSchemaAndLocation(validFrontmatter, "/tmp/gh-aw/engine-harness-watchdog-timeout-valid-test.md")
if err != nil {
t.Fatalf("expected valid engine.harness.watchdog-timeout to pass schema validation, got: %v", err)
}

invalidFrontmatter := map[string]any{
"on": "push",
"engine": map[string]any{
"id": "copilot",
"harness": map[string]any{
"watchdog-timeout": 0,
},
},
}

err = ValidateMainWorkflowFrontmatterWithSchemaAndLocation(invalidFrontmatter, "/tmp/gh-aw/engine-harness-watchdog-timeout-invalid-test.md")
if err == nil {
t.Fatal("expected non-positive engine.harness.watchdog-timeout to fail schema validation")
}
}

func TestValidateMainWorkflowFrontmatterWithSchemaAndLocation_EngineDriverPattern(t *testing.T) {
t.Parallel()

Expand Down
12 changes: 12 additions & 0 deletions pkg/parser/schemas/main_workflow_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -12797,6 +12797,18 @@
}
],
"description": "Maximum delay cap in ms. Accepts a literal integer or a GitHub Actions expression."
},
"watchdog-timeout": {
"oneOf": [
{
"type": "integer",
"minimum": 1
},
{
"type": "string"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Schema enforces minimum: 1 but has no maximum, so values above the JS-side MAX_POST_RESULT_WATCHDOG_TIMEOUT_MS (600 s / 10 min) are silently clamped by the harness with no feedback to the user.

💡 Suggestion

Add "maximum": 600 to the schema integer branch to match MAX_POST_RESULT_WATCHDOG_TIMEOUT_MS:

"watchdog-timeout": {
  "oneOf": [
    {
      "type": "integer",
      "minimum": 1,
      "maximum": 600
    },
    { "type": "string" }
  ]
}

And add a negative test in schema_test.go for a value of 601 to document and protect this bound.

@copilot please address this.

}
],
"description": "Post-result idle watchdog timeout in seconds. Accepts a literal integer or a GitHub Actions expression."
}
},
"additionalProperties": false
Expand Down
6 changes: 5 additions & 1 deletion pkg/workflow/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ type EngineConfig struct {
// Defaults to the repository workspace (GITHUB_WORKSPACE) when empty.
Cwd string

// Harness retry policy fields — templatable integers (literal value or ${{ expr }}).
// Harness policy fields — templatable integers (literal value or ${{ expr }}).
// When set, the value is injected as the corresponding GH_AW_HARNESS_* env var so
// that all harness scripts (copilot, claude, codex) can read it from the environment.
// The harness falls back to its built-in default when the env var is absent.
Expand All @@ -106,6 +106,7 @@ type EngineConfig struct {
HarnessInitialDelayMs string // engine.harness.initial-delay-ms → GH_AW_HARNESS_INITIAL_DELAY_MS
HarnessBackoffMultiplier string // engine.harness.backoff-multiplier → GH_AW_HARNESS_BACKOFF_MULTIPLIER
HarnessMaxDelayMs string // engine.harness.max-delay-ms → GH_AW_HARNESS_MAX_DELAY_MS
HarnessWatchdogTimeoutMs string // engine.harness.watchdog-timeout (seconds) → GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS
}

// InlineEngineDriver represents an inline engine.driver source block that gh-aw materializes
Expand Down Expand Up @@ -527,6 +528,9 @@ func applyEngineHarnessField(config *EngineConfig, engineObj map[string]any) {
if v, ok := h["max-delay-ms"]; ok {
config.HarnessMaxDelayMs = parseMaxTurnsValue(v)
}
if v, ok := h["watchdog-timeout"]; ok {
config.HarnessWatchdogTimeoutMs = parseHarnessWatchdogTimeoutValue(v)
Comment on lines +531 to +532
}
}
}

Expand Down
27 changes: 27 additions & 0 deletions pkg/workflow/engine_config_parser.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,33 @@ func parseHarnessMaxRetriesValue(raw any) string {
return parseIntOrExpressionValue(raw, 0, "harness.max-retries")
}

// parseHarnessWatchdogTimeoutValue parses harness.watchdog-timeout (seconds)
// and converts it to milliseconds for GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS.
// Accepts a positive integer (converted seconds→ms) or a GitHub Actions expression
// template (${{ ... }}), which is passed through unchanged and must already be in ms.
func parseHarnessWatchdogTimeoutValue(raw any) string {
seconds := parseIntOrExpressionValue(raw, 1, "harness.watchdog-timeout")
if seconds == "" {
return ""
}
// GitHub Actions expressions do not support arithmetic operators; pass through
// unchanged. Callers using an expression must supply a value already in ms.
if _, ok := extractWrappedGitHubExpression(seconds); ok {
return seconds
}
parsedSeconds, err := strconv.ParseInt(seconds, 10, 64)
if err != nil {
engineLog.Printf("Ignoring invalid harness.watchdog-timeout value: %q", seconds)
return ""
}
const maxInt64Div1000 = int64((1<<63)-1) / 1000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The maxInt64Div1000 overflow guard is unreachable — ParseIntValue returns int, which overflows long before reaching int64 max ÷ 1000 on any supported platform.

💡 Details

parseIntOrExpressionValue calls typeutil.ParseIntValue which returns (int, bool). On 64-bit targets int tops out at ~9.2 × 1018, so parsedSeconds will never exceed maxInt64Div1000 (~9.2 × 1015). The guard was copied defensively but adds noise without protection.

The real upper bound enforced by the JS runtime is MAX_POST_RESULT_WATCHDOG_TIMEOUT_MS = 10 * 60 * 1000 (600 s). Consider validating against that instead:

const maxWatchdogSeconds = 600
if parsedSeconds > maxWatchdogSeconds {
    engineLog.Printf("Ignoring out-of-range harness.watchdog-timeout value (max 600 s): %q", seconds)
    return ""
}

This gives users a meaningful error when they exceed the JS-side clamp, keeping Go and JS bounds in sync.

@copilot please address this.

if parsedSeconds > maxInt64Div1000 {
engineLog.Printf("Ignoring out-of-range harness.watchdog-timeout value: %q", seconds)
return ""
}
return strconv.FormatInt(parsedSeconds*1000, 10)
}

func parseIntOrExpressionValue(raw any, minValue int, fieldName string) string {
if val, ok := typeutil.ParseIntValue(raw); ok && val >= minValue {
return strconv.Itoa(val)
Expand Down
7 changes: 7 additions & 0 deletions pkg/workflow/engine_config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,7 @@ func TestExtractEngineConfig(t *testing.T) {
"initial-delay-ms": 10000,
"backoff-multiplier": 2,
"max-delay-ms": 180000,
"watchdog-timeout": 120,
},
},
},
Expand All @@ -552,6 +553,7 @@ func TestExtractEngineConfig(t *testing.T) {
HarnessInitialDelayMs: "10000",
HarnessBackoffMultiplier: "2",
HarnessMaxDelayMs: "180000",
HarnessWatchdogTimeoutMs: "120000",
},
},
{
Expand All @@ -564,6 +566,7 @@ func TestExtractEngineConfig(t *testing.T) {
"initial-delay-ms": "${{ vars.RETRY_DELAY }}",
"backoff-multiplier": "${{ vars.BACKOFF }}",
"max-delay-ms": "${{ vars.MAX_DELAY }}",
"watchdog-timeout": "${{ vars.WATCHDOG_TIMEOUT_SEC }}",
},
},
},
Expand All @@ -574,6 +577,7 @@ func TestExtractEngineConfig(t *testing.T) {
HarnessInitialDelayMs: "${{ vars.RETRY_DELAY }}",
HarnessBackoffMultiplier: "${{ vars.BACKOFF }}",
HarnessMaxDelayMs: "${{ vars.MAX_DELAY }}",
HarnessWatchdogTimeoutMs: "${{ vars.WATCHDOG_TIMEOUT_SEC }}",
},
},
{
Expand Down Expand Up @@ -665,6 +669,9 @@ func TestExtractEngineConfig(t *testing.T) {
if config.HarnessMaxDelayMs != test.expectedConfig.HarnessMaxDelayMs {
t.Errorf("Expected config.HarnessMaxDelayMs '%s', got '%s'", test.expectedConfig.HarnessMaxDelayMs, config.HarnessMaxDelayMs)
}
if config.HarnessWatchdogTimeoutMs != test.expectedConfig.HarnessWatchdogTimeoutMs {
t.Errorf("Expected config.HarnessWatchdogTimeoutMs '%s', got '%s'", test.expectedConfig.HarnessWatchdogTimeoutMs, config.HarnessWatchdogTimeoutMs)
}

if len(config.Env) != len(test.expectedConfig.Env) {
t.Errorf("Expected config.Env length %d, got %d", len(test.expectedConfig.Env), len(config.Env))
Expand Down
5 changes: 4 additions & 1 deletion pkg/workflow/engine_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ func applyEngineMaxTurnsEnv(env map[string]string, workflowData *WorkflowData) {
}

// applyEngineHarnessRetryEnv injects GH_AW_HARNESS_* environment variables from
// the engine frontmatter retry policy fields (engine.harness.max-retries, etc.).
// the engine frontmatter harness policy fields (engine.harness.max-retries, etc.).
// Only fields that are explicitly set are injected; absent fields let the harness
// fall back to its built-in defaults. Must be called before applyEngineAndAgentEnv
// so that explicit engine.env overrides take precedence.
Expand All @@ -168,6 +168,9 @@ func applyEngineHarnessRetryEnv(env map[string]string, workflowData *WorkflowDat
if cfg.HarnessMaxDelayMs != "" {
env["GH_AW_HARNESS_MAX_DELAY_MS"] = cfg.HarnessMaxDelayMs
}
if cfg.HarnessWatchdogTimeoutMs != "" {
env["GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS"] = cfg.HarnessWatchdogTimeoutMs
}
}

// applyEngineAndAgentEnv merges custom environment variables from engine and agent configs.
Expand Down
40 changes: 40 additions & 0 deletions pkg/workflow/engine_helpers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,46 @@ func TestResolveEngineID(t *testing.T) {
}
}

func TestApplyEngineHarnessRetryEnv(t *testing.T) {
t.Run("injects harness policy env vars including watchdog timeout", func(t *testing.T) {
env := map[string]string{}
workflowData := &WorkflowData{
EngineConfig: &EngineConfig{
HarnessMaxRetries: "6",
HarnessInitialDelayMs: "10000",
HarnessBackoffMultiplier: "2",
HarnessMaxDelayMs: "180000",
HarnessWatchdogTimeoutMs: "120000",
},
}

applyEngineHarnessRetryEnv(env, workflowData)

assert.Equal(t, "6", env["GH_AW_HARNESS_MAX_RETRIES"])
assert.Equal(t, "10000", env["GH_AW_HARNESS_INITIAL_DELAY_MS"])
assert.Equal(t, "2", env["GH_AW_HARNESS_BACKOFF_MULTIPLIER"])
assert.Equal(t, "180000", env["GH_AW_HARNESS_MAX_DELAY_MS"])
assert.Equal(t, "120000", env["GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS"])
})

t.Run("engine.env override still wins after harness policy env injection", func(t *testing.T) {
env := map[string]string{}
workflowData := &WorkflowData{
EngineConfig: &EngineConfig{
HarnessWatchdogTimeoutMs: "120000",
Env: map[string]string{
"GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS": "90000",
},
},
}

applyEngineHarnessRetryEnv(env, workflowData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The precedence test (engine.env override wins) calls applyEngineHarnessRetryEnv and applyEngineAndAgentEnv sequentially, but does not assert that the harness-injected value "120000" was first written and then overwritten — it only checks the final value. A race or bug that skips the harness injection entirely would still pass.

💡 Suggestion

Capture an intermediate snapshot to make the test a real precedence proof:

applyEngineHarnessRetryEnv(env, workflowData)
assert.Equal(t, "120000", env["GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS"], "harness injection should write 120000 first")

applyEngineAndAgentEnv(env, workflowData, nil)
assert.Equal(t, "90000", env["GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS"], "engine.env override should win")

@copilot please address this.

applyEngineAndAgentEnv(env, workflowData, nil)

assert.Equal(t, "90000", env["GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS"])
})
}

func TestBuildStandardNpmEngineInstallStepsNoCooldown(t *testing.T) {
steps := BuildStandardNpmEngineInstallStepsNoCooldown(
"@github/copilot",
Expand Down