A curated list of articles, playbooks, benchmarks, specifications, and open-source projects for harness engineering: the practice of shaping the environment around AI agents so they can work reliably.
Harness engineering sits at the intersection of context engineering, evaluation, observability, orchestration, safe autonomy, and software architecture. This list focuses on resources that make agents more dependable in real workflows, especially long-running coding and research tasks.
Generic agent tooling is out of scope unless the page directly covers harness design, context management, evaluation, runtime control, or other reliability-critical harness primitives.
- Courses & Learning Resources
- Foundations
- Context, Memory & Working State
- Constraints, Guardrails & Safe Autonomy
- Specs, Agent Files & Workflow Design
- Evals & Observability
- Benchmarks
- Runtimes, Harnesses & Reference Implementations
- Contributing
- License
- walkinglabs/learn-harness-engineering - A project-based course repository on making Codex and Claude Code more reliable, centered on an Electron personal knowledge base app with lecture handouts, example artifacts, and practical harness projects.
- Phelan164/codex-howto - A Codex-focused engineering curriculum with installable skills, repository instructions, scoped permissions, testing, review, orchestration, and reproducible token measurements for building an inspectable coding-agent harness.
- hardness1020/awesome-agent-architecture - Trilingual architecture notes and runnable demos covering agent loops, tool execution, memory, permissions, context delivery, and orchestration.
- Harness engineering: leveraging Codex in an agent-first world - OpenAI's flagship field report on building a large application with Codex using architectural constraints, repo-local instructions, browser validation, and telemetry.
- Effective harnesses for long-running agents - Anthropic's core article on initializer agents, feature lists,
init.sh, self-verification, and handoff artifacts across many context windows. - Harness design for long-running application development - Anthropic follow-up focused on improving long-running app generation with better task state and evaluator design.
- The Anatomy of an Agent Harness - LangChain's concise framing of an agent as model plus harness, with prompts, tools, middleware, orchestration, and runtime infrastructure.
- Harness Engineering - Thoughtworks' framing of harness work into context engineering, architectural constraints, and "garbage collection" against entropy.
- Building effective agents - Anthropic's broader guide to workflows, agents, tools, and when structured systems outperform raw prompting.
- Skill Issue: Harness Engineering for Coding Agents - A practical argument that weak results from coding agents are often harness problems rather than model problems.
- Your Agent Needs a Harness, Not a Framework - Inngest's case for treating state, retries, traces, and concurrency as first-class infrastructure.
- Greenfield AI, Brownfield AI, and the Vibecode You Just Inherited - A three-way taxonomy of codebases agents encounter โ agent-native greenfield, true legacy brownfield, and recently-vibecoded inheritance โ with playbooks for installing layered
CLAUDE.mdrules, ratcheted pre-commit hooks, baselined lint violations, and feature-folder refactors so the codebase itself stops being the harness bottleneck. - Harness Engineering for Language Agents: The Harness Layer as Control, Agency, and Runtime - A position paper that treats the harness layer as a first-class research object, proposes the controlโagencyโruntime (CAR) decomposition, and introduces HarnessCard for structured reporting of harness design and evaluation.
- Many Hands Engineering - A handbook framing the layer above the per-agent harness: how multiple harnessed agents share a commons, where decisions belong on a planned / emergent spectrum, and how human stewardship operates at a different cadence than agent execution. Treats harness engineering as a critical layer of "terrain" the framework sits on top of.
- Agent Frameworks, Runtimes, and Harnesses, Oh My! - LangChain's decomposition of what belongs in a framework, a runtime, and a harness.
- Effective context engineering for AI agents - Anthropic's guidance on managing the context window as a working memory budget rather than a dumping ground.
- Context Engineering for AI Agents: Lessons from Building Manus - Manus' detailed playbook on KV-cache locality, tool masking, filesystem memory, and keeping useful failures in-context.
- Context Engineering for Coding Agents - Thoughtworks guidance on shaping the task environment so coding agents can stay grounded and productive.
- Advanced Context Engineering for Coding Agents - HumanLayer patterns for reducing context drift and making coding sessions easier to resume.
- Context-Efficient Backpressure for Coding Agents - HumanLayer's ideas for preventing agents from burning context on noisy or low-value work.
- OpenHands Context Condensensation for More Efficient AI Agents - OpenHands' design for bounded conversation memory that preserves goals, progress, critical files, and failing tests while keeping long-running coding sessions efficient.
- Writing a good CLAUDE.md - A practical guide to creating durable, repo-local instructions that agents can repeatedly follow.
- Deterministic Context Routing - A context-management methodology that routes only the necessary-and-sufficient context to an agent through a deterministic chain (module registry โ wiki โ session state), cutting overread and context loss on large, under-documented multi-module codebases.
- DevProjex - GUI, TUI, and CLI tooling for selecting and exporting structured codebase context with folder trees, token estimates, ignore rules, and previews.
- wiki - Indexed Markdown knowledge bases that give agents incremental project context through deterministic indexes, scoped CLI access, and merge handling for parallel edits.
- Data Olympus - Git-native project knowledge base and MCP server with governed proposal-to-acceptance workflows, validity windows, supersession chains, and retrieval of in-force engineering guidance.
- OpenViking - Context database that unifies agent memory, knowledge retrieval, and skills behind an MCP-accessible storage layer.
- Code execution with MCP: building more efficient agents - Anthropic's approach to giving agents controlled execution power through explicit, inspectable tool boundaries.
- Writing effective tools for agents - Anthropic's guidance on tool interfaces that are easier for models to call correctly and safely.
- mcp-guardian - MCP proxy for progressive tool discovery, scope-based allow/block lists, and audit logging, reducing startup context while preventing blocked tools from being discovered or called.
- Beyond permission prompts: making Claude Code more secure and autonomous - Anthropic on reducing approval friction without losing control through better sandboxing and policy design.
- Mitigating Prompt Injection Attacks in Software Agents - OpenHands' practical guide to confirmation mode, analyzers, sandboxing, and hard policies for reducing prompt-injection risk in autonomous coding agents.
- APort Agent Guardrails - Deterministic pre-action authorization hooks for AI-agent tool calls, with adapters for Claude Code, Cursor, OpenClaw, LangChain, CrewAI, and related runtimes.
- Lurkr - Static scanner that runs in CI before deploy to surface AI-agent capability risks, including shadow capabilities, credentials into LLM context, eval/subprocess in
@tool, direct prompt interpolation, and unverified MCP endpoints. - HEAAL - Grammar-enforced safety constraints for AI agents via AIL (AI-Intent Language).
- Spend rails for autonomous agents: a number, not a vibe - A concrete pattern for giving a scheduled agent real purchasing authority without real risk: a per-transaction autonomy line, an escalation threshold, a hard lifetime ceiling, and a transaction ledger as the actual enforcement mechanism instead of relying on the model's judgment.
- Distributed retry patterns: bounding blast radius across a fleet - Practitioner guidance on concurrency bounds, decorrelated backoff, circuit breakers, and idempotency for preventing agent retry storms.
- AGENTS.md - A lightweight open format for repo-local instructions that tell agents how to work inside a codebase.
- agent.md - A related standardization effort for machine-readable agent instructions across projects and tools.
- GitHub Spec Kit - GitHub's toolkit for spec-driven development, useful when you want agents to execute against explicit product and engineering specs.
- Context Repository-Driven Development (CRDD) - A repository-centered methodology for preserving product intent, decisions, specifications, evidence, and traceability as durable context while keeping approval authority with humans.
- Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl - Thoughtworks on why strong specs make AI-assisted software delivery more dependable.
- 12 Factor Agents - HumanLayer's operating principles for production agents, including explicit prompts, state ownership, and clean pause-resume behavior.
- 12-Factor AgentOps - An operations-oriented companion focused on context discipline, validation, and reproducible agent workflows.
- Anchoring AI to a reference application - Thoughtworks on constraining agents with concrete exemplars so they produce more consistent output.
- Humans and Agents in Software Engineering Loops - A clear mental model for where humans should strengthen the harness instead of micromanaging every artifact.
- Claude Code: Best practices for agentic coding - Anthropic's practical recommendations for repo structure, checkpoints, validation, and delegation in agentic coding workflows.
- Testing Agent Skills Systematically with Evals - OpenAI's concrete guide to turning agent traces into repeatable evals with JSONL logs and deterministic checks.
- How to Evaluate Agent Skills (And Why You Should) - OpenHands' hands-on playbook for measuring whether a skill actually helps using bounded tasks, deterministic verifiers, no-skill baselines, and trace review.
- Agent evals - OpenAI's product guide for measuring agent quality with reproducible task-level and workflow-level evaluations.
- Evaluation best practices - OpenAI's general guide to building eval suites that match real-world distributions and catch regressions early.
- Trace grading - OpenAI documentation on grading agent traces directly, which is especially helpful for long multi-step tasks.
- Inspect AI - UK AISI's open-source evaluation framework with solver, scorer, sandboxing, tool-use, MCP, and log-viewer primitives for building reproducible agent eval harnesses.
- Demystifying Evals for AI Agents - Anthropic's guidance on what to measure when agents have many possible trajectories to success or failure.
- Evaluating Deep Agents: Our Learnings - LangChain's practical breakdown of single-step, full-run, and multi-turn eval design for stateful agents.
- Assessing internal quality while coding with an agent - Thoughtworks on moving quality checks into the loop instead of relying on after-the-fact manual review.
- Better Harness - Reviewer for coding-agent workflows that turns repository and session evidence into prioritized, verifiable harness improvements while keeping unobserved behavior explicit.
- Learning to Verify AI-Generated Code - OpenHands' overview of a layered verification stack using trajectory critics trained on production traces for reranking, early stopping, and review-time quality control.
- Improving Deep Agents with harness engineering - LangChain's evidence that harness changes alone can significantly improve benchmark performance.
- OpenTelemetry Semantic Conventions for Generative AI Systems - Standard span, metric, event, and attribute conventions for instrumenting LLM and agent workflows so harness traces stay portable across observability backends.
- AgentOps - Open-source Python SDK for agent monitoring, session replay, cost tracking, benchmarking, and tracing across common LLM and agent frameworks.
- agenttrace - Local-first TUI/CLI for auditing AI coding-agent session traces, health gates, cost spikes, tool failures, latency gaps, and attempt-to-attempt diffs.
- flameox - Profiling and optimization toolkit for agents: bounded CLI and MCP workflows capture traces, preserve native evidence, and compare experiments behind runtime conclusions.
- ax - Local-first telemetry and memory graph for auditing coding-agent sessions, costs, skills, tool usage, and OTLP events across multiple agent runtimes.
- Quantifying infrastructure noise in agentic coding evals - Anthropic on how runtime configuration can move coding benchmark scores by more than many leaderboard gaps.
These benchmarks are especially useful when you want to compare harness quality, not just model quality. They stress context handling, tool calling, environment control, verification logic, and the runtime scaffolding around the model.
- EvoClaw: Evaluating AI Agents on Continuous Software Evolution - A benchmark write-up on evaluating agents across dependent milestone sequences from real repository history, surfacing regression accumulation and long-horizon precision loss.
- Introducing Terminal-Bench 2.0 and Harbor - The Terminal-Bench 2.0 announcement, useful for understanding the harder tasks and generalized evaluation harness behind Harbor.
- LeetCode-Hard Gym - An RL environment interface to LeetCode's submission server for evaluating codegen agents, giving harnesses direct access to execution-based feedback on hard algorithmic problems.
- SEC-bench - A benchmark for evaluating LLM agents on real-world software security tasks including vulnerability reproduction and patching, stressing harness design around code execution, containerized environments, and security-aware tooling.
- SWE-bench Verified - A strong benchmark for software engineering agents working against real GitHub issues and tests, which makes harness choices around retrieval, patching, and validation highly visible.
- Terminal-Bench - A benchmark suite for terminal-native agents operating in shells, filesystems, and verification-heavy environments, which is especially useful for comparing coding-agent harnesses.
- AgentStudio - An integrated benchmark suite with realistic environments and comprehensive toolkits for evaluating virtual agents on real computer software, useful for measuring harness depth against a broad task surface.
- AssistantBench - A benchmark that evaluates web agents on realistic, time-consuming research tasks requiring multi-step tool use and information synthesis, making it a good proxy for harness quality in long-horizon web scenarios.
- BrowseComp - A benchmark that evaluates AI agents on locating hard-to-find information, stressing search strategy, context management, and retrieval harness design under difficult conditions.
- BrowserGym Leaderboard - A gym environment and leaderboard for evaluating LLMs, VLMs, and agents on web navigation tasks, offering a reproducible framework for comparing harnesses across multiple web benchmarks in one place.
- ClawBench: Can AI Agents Complete Everyday Online Tasks? - A browser-agent benchmark of 153 everyday web tasks across 144 live production sites in 15 categories, using a lightweight interception layer that captures and blocks only the final submission request so agents can be scored end-to-end on real websites without real-world side effects.
- Computer Agent Arena - An open evaluation platform where users compare LLM/VLM-based agents on real-world computer tasks ranging from general computer use to coding, data analysis, and video editing, surfacing harness differences across a wide task surface.
- OSWorld - A real computer-use benchmark with 369 tasks across Ubuntu, Windows, and macOS, complete with initial-state setup and execution-based evaluators, making it excellent for testing desktop and multimodal harnesses.
- OSWorld-MCP - An extension of OSWorld that evaluates AI agents on real-world computer tasks using the Model Context Protocol, making it useful for comparing MCP-enabled harnesses on a realistic desktop task suite.
- VAB - VisualAgentBench evaluates large multimodal models as visual foundation agents across embodied, GUI, and visual design tasks, useful for comparing harnesses on visually grounded, multi-step agent workflows.
- VisualWebArena - A benchmark for multimodal web agents on realistic visually grounded tasks, extending WebArena with image and screenshot inputs that stress harness support for visual context in browser environments.
- WebArena - A standalone, self-hostable web environment for evaluating autonomous agents on realistic tasks, making it a reproducible baseline for comparing web-facing harness designs.
- WebArena-Verified - A verified web-agent benchmark with curated tasks and deterministic evaluators over agent responses and captured network traces, making it a good fit for measuring web-facing harnesses.
- WorkArena - A benchmark for browser agents on common knowledge-work tasks, useful for comparing harnesses on realistic enterprise-style web workflows instead of toy browser tasks.
- AgentBench - A cross-environment benchmark spanning OS, databases, knowledge graphs, web browsing, and more, useful for seeing whether a harness generalizes beyond one narrow task loop.
- AppWorld - A controllable world of apps and people for benchmarking interactive coding agents, with state-based and execution-based unit tests that surface harness quality around planning, code generation, and collateral-damage control.
- GTA - A benchmark that evaluates the tool-use capability of LLM-based agents using human-written queries, real deployed tools, and authentic multimodal inputs, exposing harness gaps between isolated testing and real deployment.
- MCP Bench - A benchmark for evaluating AI models on MCP server interactions, measuring tool accuracy, latency, and token use across server types, which directly reflects harness design choices around MCP integration.
- MCP Universe - A leaderboard comparing AI model performance on MCP tasks, tracking how different models and harness configurations handle tool-augmented agent workflows.
- MCPMark - A stress-testing benchmark for model and agent capabilities in real-world MCP tasks across tools like Notion, GitHub, and Postgres, making harness MCP integration quality directly measurable.
- ฯ-Bench - A benchmark that emulates dynamic conversations between a simulated user and a language agent equipped with domain-specific API tools and policy guidelines, making it useful for evaluating harnesses built around structured tool use and policy enforcement.
- tau2-bench - A benchmark for realistic, multi-step agent tasks where success depends on tool use and execution quality rather than a single-shot answer.
- TravelPlanner - A benchmark for evaluating LLM agents on tool use and complex planning within multiple constraints, revealing how harness design handles multi-constraint satisfaction and long-horizon planning.
- Agent Arena - A leaderboard that ranks AI agents, models, tools, and frameworks using ELO-style ratings from head-to-head battles, providing a structured way to compare harness-level choices across categories.
- AgentBoard - A benchmark for multi-turn LLM agents complemented by an analytical evaluation board for assessing model performance beyond final success rates, making partial-progress and trajectory quality visible.
- CharacterEval - A benchmark for evaluating role-playing conversational agents using multi-turn dialogues and character profiles, with metrics across four dimensions including character fidelity and conversational coherence.
- ClawBench - A benchmark that evaluates AI agents across search, reasoning, coding, safety, and multi-turn conversation tasks, covering the breadth of harness demands in a single suite.
- GAIA - A benchmark for general AI assistants that is often used to compare harness-level choices around tools, planning, verification, and long-horizon autonomy.
- Galileo Agent Leaderboard - An open evaluation platform tracking LLM agents on task completion and tool calling across business domains, useful for comparing harness quality in enterprise-grade agentic scenarios.
- HAL: Holistic Agent Leaderboard - A benchmark and leaderboard for agent systems with attention to reliability, cost, and broad task coverage, making it useful for comparing end-to-end harness behavior.
- LLM Colosseum Leaderboard - A platform that evaluates LLMs by having them fight in Street Fighter III, testing speed, adaptability, and real-time decision-making as proxies for harness responsiveness under tight latency constraints.
- MAgIC - A benchmark measuring cognition, adaptability, rationality, and collaboration of LLMs in multi-agent systems, useful for evaluating how harnesses coordinate agent interactions and shared state.
- WildClawBench - An in-the-wild benchmark running agents inside a live OpenClaw environment on 60 original tasks including multimodal, long-horizon, and safety-critical scenarios, making harness robustness under real-world conditions directly visible.
- ClawWork - A real-world economic benchmark where AI agents complete professional tasks spanning 44 occupations, earning income while managing token costs and economic solvency, making it a direct test of harness efficiency under resource constraints.
- Olas Predict Benchmark - A benchmark for evaluating agents on historical prediction market data, testing harness design for research, retrieval, and forecasting in long-horizon reasoning tasks.
- Building agents with the Claude Agent SDK - Anthropic's guide to a production-oriented agent SDK with sessions, tools, and orchestration support.
- AgentKit - Inngest's TypeScript toolkit for building durable, workflow-aware agents on top of event-driven infrastructure.
- SandBase Harness - Apache-2.0 agent runtime with persistent sessions, governed MCP tools, credential handling, audit and replay, and interchangeable local or sandboxed execution backends.
- DSH Studio - Cross-platform desktop host for DeepSeek Harness with health probes, restart backoff, collision-free ports, and whole-process-tree cleanup.
- BitRouter - Apache-2.0 model router with cross-protocol routing, MCP gateway, guardrails, observability, virtual keys, and multi-account failover.
- rust-norion - GPL-3.0 Rust inference-control prototype exploring runtime boundaries, governed memory and replay, evidence-based writer gates, audit traces, and rollback for self-evolving agent systems.
- SWE-ReX - Sandboxed code execution infrastructure for AI agents, useful when harness work starts to merge into execution runtime design.
- Harbor - A generalized harness for evaluating and improving agents at scale, released alongside Terminal-Bench 2.0.
- Mitos - Snapshot-fork microVM sandboxes that give agent sessions clean, isolated starting states with declarative lifecycle control and parallel execution.
- deepagents - LangChain's open-source project for building deeper, longer-running agents with middleware and harness patterns.
- SWE-agent - A mature research coding agent that makes the harness, prompt, tools, and environment design directly inspectable.
- Citadel - A harness for Claude Code and OpenAI Codex with isolated worktrees, multi-agent coordination, and persisted memory and campaign state.
- Harness Evolver - Claude Code plugin that autonomously evolves LLM agent harnesses using multi-agent proposers, LangSmith-backed evaluation, and git worktree isolation. Based on Meta-Harness (Lee et al., 2026).
- Ralph Wiggum as a Software Engineer - Geoffrey Huntley's write-up of "Ralph," a minimalist
while :; do cat PROMPT.md | claude-code; doneharness pattern that uses single-task loops, deterministic prompt stacking, and bounded subagent parallelism to drive long-running autonomous coding. - RailWarden - Deterministic control plane for multi-agent software work with dependency-aware packages, isolated worktrees, durable validation evidence, recovery checkpoints, and integration gates.
- OpenCode Agent Orchestration Kit - Reproducible OpenCode harness with role-based agents, explicit handoffs, repo-local skills, safe installation, and mechanical contract validation.
- LoopTroop - Local-first GUI harness for long-running coding work with multi-model planning, isolated worktrees, and fresh-context recovery loops.
- Agent AFK - Headless coding-agent harness for asynchronous runs with explicit terminal states, editable lifecycle hooks, permission gates, model routing, and append-only traces.
- completely - Claude Code plugin harness with a default-fail evaluator, deterministic write and close gates, orphan recovery, and parallel-worker integration checks.
- forge-harness - Claude Code plugin for adversarial validation, source-grounding audits, session-learning capture, and pre-deployment transfer simulation.
- How we built our multi-agent research system - Anthropic's architecture write-up for a multi-agent system with separation of roles and structured coordination.
- Orkas - Local-first desktop harness for coordinating multiple agents with shared files, independent work contexts, human approval gates, and resumable execution.
- Agentlas OS - Local-first agent operation environment that composes specialist teams while retaining host-local tools, permissions, memory boundaries, and verification rules.
- Squadron - MIT-licensed declarative runtime for multi-agent workflows defined in HCL, including orchestration, state, dependency resolution, routing, persistence, and resume.
- Cowork Forge - MIT-licensed multi-agent software-development workflow with specialized roles and a staged pipeline from requirements through delivery.
- browser-use/browser-harness - A thin CDP-based browser harness that lets agents extend helper functions during execution, useful for inspecting self-healing web-task workflows.
- Uni-CLI - Universal CLI hub connecting agents to 134 sites and desktop apps via 711 declarative YAML pipelines. Ships an 8-phase Karpathy-style self-repair loop, eval harness with a starter catalog, per-call cost ledger, hardcoded sensitive-path deny list, and
unicli mcp servethat auto-registers one MCP tool per adapter. ~80 tokens per invocation. - OpenAgentRelay - Inspectable runtime boundary for exposing a local agent or automation as a keyed LAN capability with target verification, bounded conversations, JSON output, and explicit exit codes.
- BrowserAct - Open-source browser automation layer for agents with isolated parallel sessions, multi-account operation, and human handoff when automation is blocked.
- Bring Your AI MCP - Public harness-migration reference for Claude Code to Codex moves, with installable auditor artifacts and explicit validation notes for hooks, MCP config, and instruction-file differences.
- skills.sh - A community marketplace for discovering, sharing, and installing reusable AI agent skills across runtimes like Claude Code and OpenClaw, making harness capabilities portable and composable.
- stelow - Agentic product-workflow harness with Shape Up boundaries, adversarial plan review, acceptance-based execution contracts, and audit loops.
- Build A Harness - Apache-2.0 visual canvas for agent harnesses that compiles a runtime-neutral FlowSpec to several orchestration frameworks.
- agent-harness - Reproducible lifecycle for coding-agent assets with authority-ranked discovery, pinned mirrors, quarantine routing, staged activation, and host-specific wiring.
- codex-profiles - Codex CLI and Desktop profile launcher that isolates authentication, configuration, sessions, connectors, plugins, and logs by
CODEX_HOME. - AgentPlane - Git-native workflow-control harness that stores task records, policy, verification evidence, and closure state as reviewable repository artifacts.
Contributions are welcome. Please prefer resources that are:
- Specific about how agents are constrained, evaluated, resumed, observed, or orchestrated
- Original implementations, primary-source articles, or high-signal technical write-ups
- Useful to practitioners building real harnesses instead of generic AI commentary
If two links say the same thing, prefer the more primary, practical, and implementation-oriented one.
See CONTRIBUTING.md for contribution guidelines and the preferred entry format.