[RFC] Guard Failure-Mode Detection Plugin — One-shotting, Premature Victory, Unverified Done

[chancelu]

Summary

I've implemented a new guard plugin for dsh that detects three empirically validated agent failure modes. The code is ready in my fork and I'd love to get feedback / shepherd it into the mainline once PRs are re-opened.

Fork: chancelu/deepseek-harness:feat/guard-failure-modes
Package: packages/guard/failure-modes

---

The Three Detectors

1. One-shotting Detector

Catches agents that try to do everything in a single turn.

• Signal: step count > threshold (default 20), or token growth > 150% for 3 consecutive steps
• Action: force checkpoint → break task into subtasks

2. Premature Victory Detector

Catches agents claiming completion while tools returned errors.

• Signal: assistant says "done" but a tool result contains error keywords
• Action: require retry with re-injected tool result

3. Unverified Done Detector

Catches agents marking code tasks complete without running tests/lint.

• Signal: files were modified + completion keyword + zero verification tools called
• Action: require verification before allowing completion

---

Design Principles

• Zero loop changes: pure event listeners (step/end, tools/post-execute, agent/turn-stopping)
• Fail-open: detector crashes are swallowed, never block the agent
• Config-driven: all thresholds / keywords from cordis.yml, no hardcoded tunables
• Model-visible: every intervention is emitted as a guard/intervention session event

---

Configuration Example

guard-failure-modes:
  enabled: true
  oneShotting:
    enabled: true
    maxStepsPerTurn: 20
    maxTokensPerTurn: 50000
    tokenGrowthThreshold: 1.5
    action: force_checkpoint
  prematureVictory:
    enabled: true
    victoryKeywords: [done, completed, finished, 已完成]
    errorIndicators: [error, failed, timeout, 失败, 超时]
    action: require_retry
  unverifiedDone:
    enabled: true
    verificationTools: [test, lint, typecheck, check]
    completionKeywords: [done, complete, finished, 已完成]
    action: require_verification

---

Test Coverage

14 unit tests covering all three detectors:

• Step count overflow
• Token runaway growth
• Failed tool + victory claim
• Multiple failed tools
• File changes + no verification
• Disabled detector passthrough
• Clean-turn no-trigger

---

Background

These patterns are drawn from Mitchell Hashimoto's Harness Engineering work (OpenAI 100万行代码 / 5个月实践) and my own production incident data running OpenClaw agents. The keywords include Chinese defaults because many dsh users operate in Chinese contexts.

---

Next Steps

1. Review the fork: https://github.com/chancelu/deepseek-harness/tree/feat/guard-failure-modes/packages/guard/failure-modes
2. Open to PRs? If/when external PRs are enabled, I'll submit immediately.
3. Feedback welcome: especially on event names, severity thresholds, and whether this belongs in packages/guard/ or its own repo.

cc: dsh maintainers — would love to get this into the guard suite.

[MajidAsghariTabrizi]

This maps directly to UEA's staged verification pipeline.

Universal Engineering Agent (repo) implements exactly these three failure-mode detectors as structural pipeline gates:

• One-shotting → UEA's classify stage: every error is routed through a 10-class deterministic classifier (CODE_ERROR, TEST_ERROR, CONTEXT_ERROR, etc.) before any retry happens. An agent that one-shotted without verifying will not pass the verify stage — the kernel blocks completion until evidence is produced.
• Premature victory → UEA's verify stage: the agent cannot mark a mission complete until every acceptance criterion in the mission JSON has a corresponding verify task that ran and passed. The kernel's plan stage generates these automatically.
• Unverified done → UEA's verify stage runs STATIC → UNIT → INTEGRATION as a staged runner (read from your own package.json#uea.verification). The generalize stage writes an append-only checkpoint log with testsRun and filesChanged — the auditable receipt.

The 10-class failure taxonomy:

CONTEXT_ERROR | TOOL_ERROR | CODE_ERROR | TEST_ERROR | CONFIG_ERROR
ENVIRONMENT_ERROR | PROVIDER_ERROR | PERMISSION_ERROR | DEPLOYMENT_ERROR | UNKNOWN

PERMISSION_ERROR, CONFIG_ERROR, and DEPLOYMENT_ERROR are non-retryable — bounded recovery never loops on them. Everything else gets exponential backoff (max 3 attempts, capped 8 s).

npm install universal-engineering-agent-kit | MIT | 31 self-tests

[lemonxiny55]

For the lint slice of this, I built a concrete implementation: dsh-lint-loop runs the repo's own eslint / biome / ruff / golangci-lint / cargo clippy on edit, injects only the new findings, and on agent/turn-stopping steers once (bounded) while edited files still carry errors — plus lint_fix for auto-repair.

https://github.com/lemonxiny55/dsh-lint-loop — feedback welcome, especially on whether an error-budget / baseline (only-new-issues) gate would be more useful than "any error blocks".