Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Changelog

## 0.131.0

- Consume agent-profile-materialize 0.13.1 and Sandbox 0.19.4 with Interface 0.46.1, Eval 0.144.6, and Knowledge 7.1.2.
- Bind candidate system-prompt launch controls through the native harness plan, including OpenCode's generated primary agent.
- Keep Codex and Gemini prompt delivery fail-closed, and preserve separate replacement and additive controls for Claude Code, Pi, and Prime.

## 0.130.0

### Stability contract + first graduation
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ There is no local fallback.
Install its optional Python process before using it:

```bash
python -m pip install "agent-eval-rpc==0.144.4"
python -m pip install "agent-eval-rpc==0.144.6"
python -m pip install "gepa[full]==0.1.4"
```

Expand All @@ -256,7 +256,7 @@ python -m pip install "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919
Use `officialSkillOpt(...)` for Microsoft's SkillOpt:

```bash
python -m pip install "agent-eval-rpc==0.144.4"
python -m pip install "agent-eval-rpc==0.144.6"
python -m pip install "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90"
```

Expand Down
4 changes: 4 additions & 0 deletions bench/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog

## 0.8.0

- Consume Runtime 0.131.0, Eval 0.144.6, Interface 0.46.1, Knowledge 7.1.2, and Sandbox 0.19.4 as one compatible dependency set.

## 0.7.2

- Consume Runtime 0.129.0, Eval 0.144.4, Interface 0.43.1, Knowledge 7.0.11, and Sandbox 0.19.1 so benchmark model calls use the exact-profile execution boundary and the released optimizer callback contract without loading duplicate agent contracts.
Expand Down
2 changes: 1 addition & 1 deletion bench/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-bench",
"version": "0.7.2",
"version": "0.8.0",
"type": "module",
"description": "Benchmark adapters and execution for agent-runtime across coding, tool-use, RAG, memory, browser, and terminal tasks.",
"repository": {
Expand Down
2 changes: 1 addition & 1 deletion bench/src/swe-arena/gepa-seat.mts
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ export function innerSmokeJudge(): JudgeConfig<SmokeVerdict, GepaSeatScenario> {
// ---------------------------------------------------------------------------

export const GEPA_PYTHON_INSTALL_HINT =
'install `agent-eval-rpc==0.144.4`, then install ' +
'install `agent-eval-rpc==0.144.6`, then install ' +
'`gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f`'

export type GepaMethodFactory = (
Expand Down
2 changes: 1 addition & 1 deletion docs/api/primitive-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# Primitive catalog — the never-stale anti-reinvention inventory

> **GENERATED** from `@tangle-network/agent-runtime@0.130.0` and `@tangle-network/agent-eval@0.144.6` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`.
> **GENERATED** from `@tangle-network/agent-runtime@0.131.0` and `@tangle-network/agent-eval@0.144.6` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`.

## 1. agent-runtime — own public surface

Expand Down
8 changes: 4 additions & 4 deletions docs/canonical-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
Generated signatures and the complete export list live in docs/api/.
Run pnpm docs:freshness after editing this file. -->

> **Version 0.130.0.**
> **Version 0.131.0.**
> [`docs/api/primitive-catalog.md`](./api/primitive-catalog.md) lists every export and import path.
> `agent-eval` must satisfy `>=0.144.4 <0.145.0`.
> `sandbox` must satisfy `>=0.19.1 <0.20.0`.
> Portable profile and tool-part types come from `@tangle-network/agent-interface` `>=0.43.1 <0.44.0`.
> `agent-eval` must satisfy `>=0.144.6 <0.145.0`.
> `sandbox` must satisfy `>=0.19.4 <0.20.0`.
> Portable profile and tool-part types come from `@tangle-network/agent-interface` `>=0.46.1 <0.47.0`.
>
> **`./kernel` is the execution kernel**: `package.json` maps it to `src/runtime/index.ts`. Everything below labelled `/kernel` lives there — the recursive atom (`Scope`/`Supervisor`), the executor registry, budget conservation, the finalizer seam, analyst wiring, and the round-synchronous loop.
>
Expand Down
7 changes: 6 additions & 1 deletion examples/coding-benchmark/coding-benchmark.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import { leaderboard } from '@tangle-network/agent-runtime/kernel'
import { describe, expect, it } from 'vitest'
import { main, offlineAgentScripts } from './benchmark'
import { type CheckBox, composeScore, runChecks, runHeldout } from './eval'
import { harnessProfiles } from './profiles'
import { harnesses, harnessOf, harnessProfiles } from './profiles'
import { type CodingScenario, checkCmds, routeCodingFields, scenarios } from './scenarios'

const execAsync = promisify(execCb)
Expand Down Expand Up @@ -95,6 +95,11 @@ async function gradeSolution(
}

describe('coding-benchmark (offline)', () => {
it('uses the canonical AgentProfile harness field', () => {
expect(harnessProfiles.map((profile) => profile.harness)).toEqual(harnesses)
expect(harnessProfiles.map(harnessOf)).toEqual(harnesses)
})

// Integration smoke: runs the real matrix end-to-end (real box.exec on the offline
// toolchain, all refine rounds since the dev checks can't pass without tsc).
it('runs the full matrix and returns a defined leaderboard', async () => {
Expand Down
20 changes: 9 additions & 11 deletions examples/coding-benchmark/profiles.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,13 @@
* Each profile is deliberately bare (name + model, no skills, no injected prompt) so we
* measure the HARNESS, not our scaffolding; the tool surface is a separate orthogonal knob
* (`withTools`), making harness × tool a clean cartesian. Two non-obvious facts about the
* shape: `AgentProfile` (`@tangle-network/agent-interface`) has no `harness` field (harness
* is a SANDBOX concept), so the harness selector rides `metadata.harness` (`harnessOf()` is
* the one reader); and `runProfileMatrix` REQUIRES a snapshot-dated `model.default` — see
* `harnessModel` below.
* shape: `AgentProfile.harness` is the canonical selector, while Sandbox owns validation
* that the selected harness is an executable backend; and `runProfileMatrix` REQUIRES a
* snapshot-dated `model.default` — see `harnessModel` below.
*/

import type { AgentProfile, AgentProfileMcpServer } from '@tangle-network/agent-interface'
import type { BackendType } from '@tangle-network/sandbox'
import { type BackendType, parseBackendType } from '@tangle-network/sandbox'

/** The harnesses we sweep. `cli-base` is the plain-CLI baseline (no agent harness). */
export const harnesses = [
Expand All @@ -21,13 +20,12 @@ export const harnesses = [
'cli-base',
] as const satisfies readonly BackendType[]

/** Read the harness a profile targets. The ONE place metadata.harness is decoded. */
/** Read and validate the executable harness a profile targets. */
export function harnessOf(profile: AgentProfile): BackendType {
const h = profile.metadata?.harness
if (typeof h !== 'string') {
throw new Error(`profile "${profile.name}" is missing metadata.harness — see profiles.ts`)
if (profile.harness === undefined) {
throw new Error(`profile "${profile.name}" is missing harness — see profiles.ts`)
}
return h as BackendType
return parseBackendType(profile.harness)
}

/** The default model each harness runs (override per-harness via env). The model id MUST
Expand Down Expand Up @@ -57,8 +55,8 @@ const harnessModel: Record<BackendType, string> = {
/** One bare baseline profile per harness — the harness's out-of-the-box behavior. */
export const harnessProfiles: AgentProfile[] = harnesses.map((harness) => ({
name: `${harness}-baseline`,
harness,
model: { default: harnessModel[harness] },
metadata: { harness },
}))

// ── the tool knob ─────────────────────────────────────────────────────────────
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-runtime",
"version": "0.130.0",
"version": "0.131.0",
"description": "Shared task-lifecycle skeleton for agents: a recursive loop kernel for chat turns, one-shot tasks, and multi-attempt loops, with trace capture and eval-gated self-improvement. Domain behavior lives in adapters; scoring and ship-gates in @tangle-network/agent-eval.",
"homepage": "https://github.com/tangle-network/agent-runtime#readme",
"repository": {
Expand Down Expand Up @@ -171,7 +171,7 @@
"peerDependencies": {
"@tangle-network/agent-eval": ">=0.144.6 <0.145.0",
"@tangle-network/agent-interface": ">=0.46.1 <0.47.0",
"@tangle-network/sandbox": ">=0.19.3 <0.20.0"
"@tangle-network/sandbox": ">=0.19.4 <0.20.0"
},
"peerDependenciesMeta": {
"@tangle-network/sandbox": {
Expand Down
49 changes: 16 additions & 33 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions pnpm-workspace.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ catalog:
'@tangle-network/agent-eval': 0.144.6
'@tangle-network/agent-interface': 0.46.1
'@tangle-network/agent-knowledge': 7.1.2
'@tangle-network/agent-profile-materialize': 0.12.0
'@tangle-network/agent-profile-materialize': 0.13.1
'@tangle-network/agent-trace-contract': ^1.0.2
'@tangle-network/sandbox': 0.19.3
'@tangle-network/sandbox': 0.19.4
publint: 0.3.22
tsdown: 0.22.14
tsx: 4.23.1
Expand Down
33 changes: 10 additions & 23 deletions src/candidate-execution/prepare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,7 @@ import {
agentCandidateWorkspaceSnapshotEvidenceSchema,
sha256DigestSchema,
} from '@tangle-network/agent-interface'
import {
applyAgentCandidateWorkspacePlan,
materializeCandidateProfile,
} from '@tangle-network/agent-profile-materialize'
import { applyAgentCandidateWorkspacePlan } from '@tangle-network/agent-profile-materialize'

import {
readMaterializedWorkspaceFiles,
Expand Down Expand Up @@ -63,8 +60,7 @@ import {
} from './knowledge'
import { sealAgentCandidateModelSettlement, usdToNanos } from './model-settlement'
import { createPreparedCandidateExecution } from './prepared-state'
import { candidateMaterializerHarness } from './profile'
import { projectCandidateSystemPrompt } from './system-prompt'
import { materializeAgentCandidateProfilePlan } from './profile'
import {
type AgentCandidateExecutionPorts,
type AgentCandidateTaskExecution,
Expand Down Expand Up @@ -109,7 +105,6 @@ export async function prepareAgentCandidateExecution(
maxAttempts: benchmarkTask.attempt.maxAttempts,
retryPolicy: benchmarkTask.attempt.retryPolicy,
} as const
const harness = candidateMaterializerHarness(bundle.execution.harness)
assertTaskInput(task, bundle.execution.instructionDelivery)
const resultTimeoutMs = candidateResultTimeout(
options.resultTimeoutMs,
Expand Down Expand Up @@ -178,13 +173,14 @@ export async function prepareAgentCandidateExecution(
}

await assertEmptyDirectory(task.stagingRoots.profileRoot)
const profileWorkspacePlan = projectCandidateSystemPrompt(
materializeCandidateProfile(bundle.profile, harness, {
resolvedResources: verifiedResourceTextByDigest(candidate),
}),
bundle.execution.launch,
profileSystemPromptExecutionPath(bundle.execution.cwd.workspace, task.executionRoots),
)
const profileWorkspacePlan = materializeAgentCandidateProfilePlan({
profile: bundle.profile,
harness: bundle.execution.harness,
launch: bundle.execution.launch,
workspace: bundle.execution.cwd.workspace,
workspaces: task.executionRoots,
resolvedResources: verifiedResourceTextByDigest(candidate),
})
const profileApplication = applyAgentCandidateWorkspacePlan(
profileWorkspacePlan,
task.stagingRoots.profileRoot,
Expand Down Expand Up @@ -935,15 +931,6 @@ function absoluteExecutionCwd(
return absolute
}

function profileSystemPromptExecutionPath(
workspace: VerifiedAgentCandidate['bundle']['execution']['cwd']['workspace'],
roots: AgentCandidateTaskExecution['executionRoots'],
): string {
const root = workspace === 'task' ? roots.taskRoot : roots.candidateRoot
if (!root) throw new Error('candidate profile target is missing its execution workspace root')
return posix.join(root, '.tangle/system-prompt.md')
}

function validateProtectedModelReservation(
reservation: {
preparationId: string
Expand Down
Loading