Add fullsend run agent command and experiment for test it

[maruiz93]

fullsend run — agent runner with sandbox isolation

Adds the fullsend run <agent-name> command, which reads a harness YAML definition, provisions an OpenShell sandbox (optionally from a container image), bootstraps it with agent config/skills/credentials, runs a Claude Code agent inside it, and validates the output.

What's included

Go CLI (internal/cli/run.go, internal/harness/, internal/sandbox/):

• fullsend run <agent-name> --harness-dir <path> — new subcommand on root
• Harness YAML schema: agent definition, skills, policies, env files, host files, container image, pre/post scripts, validation loop
• OpenShell sandbox lifecycle: create (with --from for container images), bootstrap via SCP/SSH, policy application, cleanup
• TARGET_REPO_DIR env var to specify the target codebase directory (for two-repo setups where the harness and target repo are in separate repositories)
• Pre-script runs on the host before sandbox creation; post-script runs after sandbox cleanup
• Validation loop with configurable retries and host-side validation scripts

Hello-world experiment (experiments/runner-hello-world/):

• End-to-end test of the full fullsend run flow
• Container image (Containerfile) with Claude Code and tool binaries pre-installed, based on the OpenShell base image
• Agent that runs a hello-world tool and summarizes a target repository
• Two-repo layout: harness files in a .fullsend repo, target codebase in a separate repo
• Automated build/deploy/trigger script (run-experiment.sh)

Test org

The experiment runs against the test-fullsend org with two repos:

• test-fullsend/.fullsend — harness definitions, agent config, workflow
• test-fullsend/test-repo — target codebase (bookstore microservice docs)

Successful workflow runs:

• Run 24372870045 — two-repo layout, flat .fullsend repo
• Run 24372746135 — first run against new org

What's not tested by this experiment

• Pre/post scripts, providers, API servers, agent input, feedback mode — implemented in the CLI but not exercised
• Providers
• Skills with its own scripts

How to reproduce

See experiments/runner-hello-world/HOW_TO.md.

[rh-hemartin]

Not that really matters, but I passed REPO_NAME=kubearchive/kubearchive and tell the agent to exit early with:

3. If there is any conflict, exit early giving the explanation of the problem.

Leading to:

## Conflict Detected — Exiting Early

There is a **conflict** between the expected and actual repository:

- **`REPO_NAME`** is set to `kubearchive/kubearchive`, but **no such repository exists** in the workspace.
- The only repository present is `fullsend-ai/fullsend` (located at `/tmp/workspace/fullsend`).

Since the task requires exploring the repository specified by `REPO_NAME` and that repository is not available, I cannot proceed with the remaining steps (exploring the repo and writing the summary).

**What was completed:**
- ✅ `output/hello-world.md` was successfully written by the `hello-world-bin` tool.

**What could not be completed:**
- ❌ Repository exploration — `kubearchive/kubearchive` does not exist in the workspace.
- ❌ `output/summary.md` — cannot summarize a repository that isn't present.

I could test this quickly locally.

[waynesun09]

5 critical security issues that need addressing before merge. The most dangerous is the policy race window (#2) — verified against OpenShell's architecture that the fix (applying policy immediately after Create) is safe since SSH/SCP uses the control plane, not the restricted network namespace.

[waynesun09]

Round 2: High, Medium, and Low issues

18 additional comments covering supply chain risks (experiment files), test coverage gaps, code bugs, and documentation nits. See the first review for the 5 critical security issues in the core runner code.

[ascerra]

Alright so I ran this locally and hit the same issue that @rh-hemartin raised on slack where I needed to add --model.

Then the other thing that I had to work around was making my agent skill script available. Here's so info on that and what I did.

Skills can depend on scripts, but there's no way to declare that in the harness as of right now

I tested this locally with a real agent whose code-implementation skill references a scan-secrets helper script (a gitleaks wrapper). The skill's SKILL.md says run $SCAN_SECRETS — but the harness has no way to express "this skill needs this script on $PATH inside the sandbox."

The skills: field copies the skill directory into .claude/skills/, but scripts that skills depend on aren't part of that directory. I had to work around it with host_files:

  - src: scripts/scan-secrets
    dest: /tmp/workspace/bin/scan-secrets
plus an env file that sets export SCAN_SECRETS=/tmp/workspace/bin/scan-secrets so the skill can find it.

This works but it's brittle — the skill and its script dependency are declared in two separate, unrelated places in the harness. If someone copies the skill without the host_files entry, the agent fails at runtime with a confusing "command not found."

A few options:

1. Add a scripts: field to the harness (parallel to skills:), where listed scripts are copied into /tmp/workspace/bin/ and that directory is already on $PATH
2. Let skills declare their own scripts/ subdirectory that gets copied alongside the SKILL.md (I'm thinking this might need to be the way to go, which I know is outside the scope of this PR but still wanted to raise this)
3. At minimum, document the host_files workaround pattern in HOW_TO.md so others don't have to discover it

[maruiz93]

Alright so I ran this locally and hit the same issue that @rh-hemartin raised on slack where I needed to add --model.

Right! This is a bug with an easy fix, I'll take care of it.

Then the other thing that I had to work around was making my agent skill script available. Here's so info on that and what I did.

Skills can depend on scripts, but there's no way to declare that in the harness as of right now

I tested this locally with a real agent whose code-implementation skill references a scan-secrets helper script (a gitleaks wrapper). The skill's SKILL.md says run $SCAN_SECRETS — but the harness has no way to express "this skill needs this script on $PATH inside the sandbox."

The skills: field copies the skill directory into .claude/skills/, but scripts that skills depend on aren't part of that directory. I had to work around it with host_files:

  - src: scripts/scan-secrets
    dest: /tmp/workspace/bin/scan-secrets
plus an env file that sets export SCAN_SECRETS=/tmp/workspace/bin/scan-secrets so the skill can find it.

This works but it's brittle — the skill and its script dependency are declared in two separate, unrelated places in the harness. If someone copies the skill without the host_files entry, the agent fails at runtime with a confusing "command not found."

A few options:

1. Add a scripts: field to the harness (parallel to skills:), where listed scripts are copied into /tmp/workspace/bin/ and that directory is already on $PATH
2. Let skills declare their own scripts/ subdirectory that gets copied alongside the SKILL.md (I'm thinking this might need to be the way to go, which I know is outside the scope of this PR but still wanted to raise this)
3. At minimum, document the host_files workaround pattern in HOW_TO.md so others don't have to discover it

I think the way to go for tools that a skill relies on is baking them into the sandbox image. The skill's SKILL.md would reference them in its front-matter as Bash(my-tool *), and since the tool is already on $PATH inside the image, it just works. This gives us reproducibility — the image is the single source of truth for what tools are available.

I added host_files support for files that are dynamic by nature — things like the output of a previous agent, or files generated by a pre-script. Your workaround of using it for the scan-secrets script works, but I'd treat it as a short-term solution. For the long term, if a tool is a stable dependency of a skill, it belongs in the image.

For the short term, I'd go with your option 3 — document the host_files workaround pattern so others don't have to discover it on their own.

On a related but separate topic: there are also tools that run outside the sandbox entirely, like REST APIs that a skill might call.
The harness already has an api_servers field for declaring these, but there's no runtime logic in fullsend run to start and manage them yet. That's a different problem from "script on $PATH" — just flagging it as upcoming scope.

As for having skills declare their own tool dependencies and having fullsend run parse that, I'd rather not go down that path:

• Skills are markdown, unstructured by definition
• We could use the front-matter, but that would mean departing from the SKILL.md format natively supported by Claude
• Even with front-matter, how would we differentiate between a tool already available in the image and one that needs to be copied in?

Agreed that we should add this to the HOW_TO — both the host_files workaround for now and the image-based approach as the recommended pattern going forward. Also about the REST APIs that isn't covered yet.

[ralphbean]

Regarding scripts, know that skills that bundle their own scripts/ are part of the agentskills specification: https://agentskills.io/specification

We should support copying in any scripts/ and references/ and assets/ that a skill's directory includes.

[maruiz93]

Regarding scripts, know that skills that bundle their own scripts/ are part of the agentskills specification: https://agentskills.io/specification

We should support copying in any scripts/ and references/ and assets/ that a skill's directory includes.

yep, but from the agent creator POV we should know that with this schema the scripts aren't reused, they aren't tools they are just for the skill, right?

[github-actions]

Site preview

Preview: https://2637a2e1-site.fullsend-ai.workers.dev

Commit: 5d4d1610ee483fe761a769398a5834366c006113

[maruiz93]

Added --model

Skills can depend on scripts, but there's no way to declare that in the harness as of right now

If, as ralph mentioned, your script is specific to a skill, you can add it under the skill dir, it's supported as the fullsend run command copies the whole dir in the sandbox.

If it's something you want to reuse, either the image, or the host_file option. There is new doc about this in the README.md

[maruiz93]

Not that really matters, but I passed REPO_NAME=kubearchive/kubearchive and tell the agent to exit early with:

3. If there is any conflict, exit early giving the explanation of the problem.

Leading to:

## Conflict Detected — Exiting Early

There is a **conflict** between the expected and actual repository:

- **`REPO_NAME`** is set to `kubearchive/kubearchive`, but **no such repository exists** in the workspace.
- The only repository present is `fullsend-ai/fullsend` (located at `/tmp/workspace/fullsend`).

Since the task requires exploring the repository specified by `REPO_NAME` and that repository is not available, I cannot proceed with the remaining steps (exploring the repo and writing the summary).

**What was completed:**
- ✅ `output/hello-world.md` was successfully written by the `hello-world-bin` tool.

**What could not be completed:**
- ❌ Repository exploration — `kubearchive/kubearchive` does not exist in the workspace.
- ❌ `output/summary.md` — cannot summarize a repository that isn't present.

I could test this quickly locally.

Fixed. The agent was running in the wrong directory — it was seeing the fullsend source repo instead of the target repo. Two changes:

1. Added --target-repo CLI flag. The target repo is now copied into the sandbox at /tmp/workspace/{repoName}/ and the agent cds into it before running.
2. FULLSEND_TARGET_REPO_DIR env var is set inside the sandbox so agents/skills can reference it.

If you test again with REPO_NAME=kubearchive/kubearchive and --target-repo pointing at a kubearchive clone, the agent should see the right repo.

[waynesun09]

Inline comment on validation exit code — see thread.

[waynesun09]

Two high-severity findings from second-round review — both involve the host trusting file paths and content originating from inside the sandbox, which is the security boundary being protected against.

[waynesun09]

Second-round review — 2 high, 7 medium, 4 low, 1 informational.

The first-round security fixes (credential handling, command injection, policy race) are all solid. The remaining issues center on the sandbox-to-host trust boundary — ExtractOutputFiles, ExtractTranscripts, and the repo write-back all trust file paths and content originating from inside the sandbox, which is the security boundary being protected against.

Must-fix (High):

• Path traversal via sandbox-controlled find output in ExtractOutputFiles
• Unrestricted scp -r write-back follows symlinks and enables git hook injection

Should-fix (Medium):

• runAgent returns nil on validation failure (exit code always 0)
• ProcessState nil panic when ssh binary fails to start
• sandbox.Delete silently swallows errors
• Stale sandbox-side output contaminates iterations after the first
• Transcript path injection from sandbox-controlled find output
• host_files src path traversal — no containment check
• No model field validation (inconsistent with agent name regex)

[waynesun09]

Future improvement: replace exec.Command SSH/SCP wrappers with Go libraries

Several of the review findings (path traversal, symlink following, no SCP timeout, ProcessState nil panic) stem from shelling out to ssh/scp via exec.Command. A Go-native SSH + SFTP stack would eliminate these classes of issues structurally:

Need	Library	Notes
SSH commands	github.com/melbahja/goph (wraps x/crypto/ssh)	Context cancellation with SIGINT, no ProcessState, streaming output
File transfer	github.com/pkg/sftp	Lstat() detects symlinks without following, Walk() gives per-file control, concurrent transfers
Safe local writes	os.Root (Go 1.24+ stdlib)	Kernel-level path containment, eliminates TOCTOU

Key wins:

• Path traversal — sftp.Walk() + os.Root provides containment that scp -r cannot
• Symlink safety — sftp.Lstat() detects symlinks; reject in walk loop instead of trusting scp -r
• Timeouts — native context support on all operations, no more indefinite SCP hangs
• No process panics — pure Go, no cmd.ProcessState to dereference

Also worth noting: OpenShell has a gRPC ExecSandbox RPC that could replace SSH for command execution entirely (native timeout, streaming, env vars). No file transfer RPC exists yet and there's no Go SDK, but the proto files are public.

Not blocking this PR — the immediate fixes (containment checks, nil guards) are sufficient. Suggesting this as a follow-up issue for the next iteration. All libraries are BSD/MIT licensed, compatible with Apache 2.0. Minimum x/crypto/ssh version: v0.45.0 (4 CVEs patched below that).

[waynesun09]

All issues from both review rounds have been addressed. Verified each fix in the diff:

Critical (Round 1) — all fixed:

• Command injection: regex validation [a-zA-Z0-9_-] + defense-in-depth escaping
• Policy race: policy applied at sandbox.Create() via --policy flag
• Credential exposure in errors: redaction via strings.ReplaceAll
• CLI credential leak: bare-key --credential KEY, secrets via cmd.Env
• Pre/post script semantics: file paths + ValidateFilesExist()

High (Round 2) — both fixed:

• Path traversal in ExtractOutputFiles: filepath.Clean + prefix containment check
• scp -r symlink following: replaced with RsyncFrom using --no-links + --exclude .git/hooks/

Medium (Round 2) — all fixed:

• runAgent validation exit code: returns error when !validationPassed
• ProcessState nil guard in SSH()/SSHStream()
• Delete() returns error, caller logs as warning
• Stale sandbox output cleared between iterations
• Transcript path containment check added
• host_files src traversal rejected in ResolveRelativeTo
• Model field validated with same regex as agent name

Low (Round 2) — all fixed:

• SCP/rsync timeout (5 min via CommandContext)
• ApplyPolicy dead code removed
• Single os.ExpandEnv pass (no double expansion)
• Split on newlines instead of whitespace

No new security issues found. The code is solid.