Skip to content

benchmark: untrusted candidate-agent execution — sandbox, limits, attested when hardware allows #9264

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 4 of that issue: candidate agents are untrusted by construction — arbitrary third-party code, submitted to be scored. Running it needs the isolation the attested-evaluation stack already builds, plus the sandbox properties that matter regardless of whether attestation hardware is present.

Requirements

  1. Reuse the shipped runner, do not build a second one. The pinned, digest-manifested replay-runner image (orb(attest): reproducible pinned replay-runner image + digest manifest enforced in CI #9214), the attester seam (orb(attest): attested-run harness with pluggable attester — sample now, SNP by config #9211), and the CoCo/KBS deploy manifests (metal-prep: CoCo stack in no-TEE dev mode — pinned manifests, Trustee topology decision, sample-attester E2E #9213, k8s/coco-dev/) already exist. This issue adapts them to execute a candidate agent, it does not start a parallel execution path.
  2. Sandbox properties that hold with or without SNP hardware, since attestation is not a sandbox: no network egress during scoring (the runner already proves --network none works), read-only task input, bounded CPU/memory/wall-clock, non-root, no host filesystem access. A candidate agent that can reach the network can fetch the realized future and score perfectly.
  3. Attestation when hardware is present, honestly labeled when it is not. Runs emit the trust tier defined in Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215: attested only under genuine SNP, reproducible otherwise. Never label an unattested run as attested because the plumbing supports it.
  4. Deterministic replay inputs: the agent receives exactly the frozen snapshot (benchmark: frozen-repo snapshot builder — leak-proof repo state at commit T #9259) and nothing else — same leak-proofing property, enforced at the runtime boundary as well as in the data.
  5. Fail-closed: a run that cannot establish its claimed isolation does not score, matching decideAttestedRunOutcome's existing fail-closed posture under a claimed TEE.

Deliverables

  • Candidate-agent execution harness built on the existing runner image and attester seam
  • Tests proving each sandbox property, including a real fail-closed test (a run claiming isolation it does not have must not produce a score)
  • Documented resource limits and the reasoning behind each

Expected outcome

Third-party code can be scored without trusting it, and without a second, weaker execution path existing alongside the attested one.

References

Sub-issue of #9216. Reuses #9214 (scripts/replay-runner/), #9211 (scripts/attested-backtest-run.ts), #9213 (k8s/coco-dev/). Trust tiers: #9215. Depends on #9259. Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions