diff --git a/labs/15-pitot/KIMI_CONTROLLED_ACTION_PLAN.md b/labs/15-pitot/KIMI_CONTROLLED_ACTION_PLAN.md new file mode 100644 index 000000000..ac419a171 --- /dev/null +++ b/labs/15-pitot/KIMI_CONTROLLED_ACTION_PLAN.md @@ -0,0 +1,203 @@ +# Plan: Pitot's first truthful, reproducible Kimi controlled action + +**Branch:** `worktree-pitot-kimi-controlled-action` (off `origin/main`) +**PR title:** Make Pitot's first controlled action truthful and reproducible +**Public promise:** *Keep your coding agent. Add the behavior it is missing.* + +Goal: prove `clone → start one shell Controller → launch Kimi → allow one action → deny one +action → prove denied command never ran → show denial reached Kimi`, then rewrite the README +around that tested outcome. Everything below is grounded in the current code. + +--- + +## Ground truth (verified against the tree) + +All paths under `labs/15-pitot/`. + +| Claim in the brief | Verified reality | File | +|---|---|---| +| Controller registered for `test.approval` | Confirmed, hardcoded regardless of language | `pitot/cmd/pitot/workbench.go:198-205` | +| Kimi normalizes `PreToolUse` → kind `shell` | Confirmed | `pitot/adapters/adapters.go:196-215` | +| Dispatch keyed by `event.action.kind` | Confirmed; controllers map keyed by kind | `pitot/runtime/runtime.go:118-144` | +| Generated Controller can't control Kimi | Confirmed — `test.approval` ≠ `shell`, never matches | — | +| Next-step hint points `--exec` at the Controller | Confirmed: `Next: ... pitot dev --host claude --exec "go run main.go"` | `pitot/cmd/pitot/workbench.go:135-137` | +| `--host` validates only, never configures a host | Confirmed — only `adapters.IsSupported` + banner | `pitot/cmd/pitot/workbench.go:503-508,540` | +| README claims `pitot dev` configures the host | Confirmed, line 257 of the public README | `public-readme-preview/README.md:257` | +| README use-case gallery link is unprojected | Confirmed — points into excluded `brand-exploration/` | `public-readme-preview/README.md:66` | +| Generated manifests assume `0.1.0` | Confirmed via `pitotPackageVersion = "0.1.0"` | `pitot/cmd/pitot/workbench.go:22,154,348,373,430,465` | +| Python dist name `pitot` occupied on PyPI | Confirmed — SDK **and** generated manifest both use bare `pitot` | `pitot-distribution/sdk/python/pyproject.toml:6`; `workbench.go:154,348` | + +**Corrections to the brief (things that differ from its assumptions):** + +1. **No `--template` flag exists.** `pitot init` supports only `--language`, `--role`, `--dir`, + `--force`. Templates must be added from scratch (`workbench.go:37-62`). +2. **`pitot setup` does not exist. `pitot doctor` does** but has **no `--host` flag** — it loops + every host and prints a decoder PASS/FAIL only (`cmd/pitot/main.go:199-217`). Part 2's check + is an *extension of `doctor`* to accept `--host` and add PATH/config/hook checks. Recommend + `pitot doctor --host kimi` (smaller surface than a new `setup` verb). +3. **Python rename is bigger than the brief implies.** The published SDK `pyproject.toml` itself + uses `name = "pitot"`, not just the generated manifest. Both must change to + `operatorstack-pitot` while keeping `import pitot`. +4. **Command extraction shape:** in Full mode `Content.Full` is a **JSON-encoded string** of the + command (`projection/projection.go:40-46`). The shell-policy controller must + `json.Unmarshal(event.Content.Full, &command)` after confirming `event.Content.Mode == "full"`. +5. **README is `public-readme-preview/README.md`** (the only projected README). Any surface change + requires regenerating `pitot-distribution/UPSTREAM.json` via `scripts/build_pitot.py --write`, + or CI's `--check` fails (`scripts/build_pitot.py:153-176`). +6. **No `docs/public-claims.json` exists in this lab.** The support-matrix truthfulness lives in + the README + `adapter-verification.json`. Do not invent a claims file unless we choose to. +7. **Kimi already has an allow-path unit test** (`cmd/pitot/main_test.go:172-196`) but **no + deny-path test**, and live-CLI E2E is intentionally deferred + (`pitot-distribution/release-notes/2026-07-22-kimi-code-adapter.md`). + +--- + +## Work breakdown (single PR, ordered so each step is independently green) + +### Step 1 — `shell-policy` template + template selection (`workbench.go`) + +- Add `--template` flag to `runInit` (`workbench.go:37-62`); allowed values: + `shell-policy`, `release-approval`, `blank-controller`, `blank-consumer`. Default preserves + today's behavior (map `blank-controller`/`blank-consumer` to the existing templates). +- New `shell-policy` controller template per language. Go example (the reference language, since + `main.go` is the sample Controller): + ```go + func main() { + sdk.RunController("local-shell-policy", func(req schema.ControlRequested) sdk.Outcome { + var event schema.Event + if err := json.Unmarshal(req.Data, &event); err != nil { + return sdk.Deny("Pitot sample policy could not decode the event.") + } + if event.Content == nil || event.Content.Mode != schema.ContentFull { + return sdk.Deny("Pitot sample policy requires full content mode.") + } + var command string + if err := json.Unmarshal(event.Content.Full, &command); err != nil { + return sdk.Deny("Pitot sample policy could not decode the command.") + } + if strings.Contains(command, "PITOT_DENY_ME") { + return sdk.Deny("Pitot sample policy blocked the PITOT_DENY_ME canary.") + } + return sdk.Allow("Pitot sample policy allowed the shell request.") + }) + } + ``` + Include a code comment stating the substring check is a **sample canary, not production shell + security** (brief requirement; no general security claim). +- `pitotConfig` (`workbench.go:187-206`): when template is `shell-policy`, emit the controller + keyed under **`shell`** (not `test.approval`) with `id: local-shell-policy`, `deadline_ms: 2000`, + `on_timeout: deny`, `on_unavailable: deny`. This is the change that actually lets the Controller + govern Kimi (dispatch keys on `shell`). +- Keep this PR to those 4 templates only. + +### Step 2 — Fix generated guidance (`workbench.go:135-137`) + +- Stop pointing `--exec` at the Controller. The runtime already launches `main.go` from + `.pitot.yaml`; `--exec`/`-- CMD` is for the **agent**. +- If host known: `pitot dev --host kimi -- kimi` (or `-- kimi -p ""`). +- If host unknown, print the two-line "configure a host, then `pitot dev --host HOST -- AGENT`" + hint from the brief. + +### Step 3 — Truthful host setup: `pitot doctor --host kimi` (`cmd/pitot/main.go:199-217`) + +- Extend `runDoctor` to accept `--host`. With a host, verify and report: + `kimi` on `PATH`; Kimi config path resolvable; a `PreToolUse`/`Bash` hook exists; hook command + invokes `pitot hook kimi`; config parses (use `kimi doctor` when available). +- **Do not** have `pitot dev` edit global Kimi config in this PR. Document the canonical TOML: + ```toml + [[hooks]] + event = "PreToolUse" + matcher = "Bash" + command = "pitot hook kimi" + timeout = 5 + ``` +- Document Kimi hooks are **fail-open** on crash/timeout per host semantics; do not market as a + sandbox. + +### Step 4 — Two levels of Kimi testing + +- **Test A (deterministic, no model)** — new Go test (extend `cmd/pitot/main_test.go`, or a new + `e2e` test). Flow: start runtime + sample shell Controller → submit canonical Kimi allow payload + → assert hook exit 0 → harness executes canary → assert `/tmp/pitot-allowed-canary` exists → + submit deny payload → assert hook exit 2 → harness does **not** execute canary → assert + `/tmp/pitot-denied-canary` absent → assert deny stderr carries the Controller reason → assert + decision receipt has `kind=shell`, `outcome=deny`. Uses the exact allow/deny payloads from the + brief. This is the gate for the whole PR and for rewriting the README. +- **Test B (opt-in real Kimi smoke)** — manual/opt-in (needs Kimi auth). Preflight + (`kimi --version`, `kimi doctor`, `pitot doctor --host kimi`), clean canaries, run the brief's + `pitot dev --host kimi -- kimi -p '...'` prompt, assert postconditions, and emit a **bounded JSON + evidence artifact** (Kimi version, hook config hash w/ secrets omitted, pitot commit + binary + SHA-256, runtime descriptor identity, allow/deny action IDs, Controller outcomes, canary states, + final Kimi text, exit status). Prose alone is not proof. + +### Step 5 — Workbench test hardening (`cmd/pitot/workbench_test.go`) + +Replace "files exist" with contract + build tests: +- **Init contract**, per language: expected files exist; `.pitot.yaml` parses; role+template map to + the correct request kind (**`shell` for shell-policy**); generated source references the right SDK + API; non-destructive; next-step guidance launches an **agent**, not the Controller. +- **Build tests** (gated on available toolchains, no remote registries): Go `go test/build` with + local module `replace`; Python import; TS local package smoke; Rust `cargo check` with path dep. +- **Dev tests**: unsupported host rejected; args preserved; unique runtime paths; readiness before + child; child gets `PITOT_RUNTIME`; runtime dir removed on exit; decision timeline renders + allow+deny; `--exec` vs `-- CMD ARGS` semantics distinct. + +### Step 6 — Package naming truthfulness + +- **Python:** rename distribution to `operatorstack-pitot` in + `pitot-distribution/sdk/python/pyproject.toml` **and** generated `requirements.txt`/`pyproject.toml` + (`workbench.go:154,340-349`), keeping import package `pitot`. Generated dep: + `operatorstack-pitot>=0.1.0`. +- **TypeScript:** keep `@operatorstack/pitot`; README install claim only if actually published. +- **Rust:** use `operatorstack-pitot` if `pitot` crate name isn't controlled; lib name may stay `pitot`. +- **Go:** README must label SDK setup source-based/unreleased until `github.com/operatorstack/pitot v0.1.0` + is tagged. +- Until registries/tags exist, README labels every unpublished install as source-based. + +### Step 7 — README value translation (`public-readme-preview/README.md`) — **only after Test A passes** + +- New opening: `# Pitot` / `## Keep your coding agent. Add the behavior it is missing.` + the + Kimi-first framing. Keep "Pitot reports. Your controller decides." as *supporting* vocabulary. +- First visible section **"See it work with Kimi"** = the tested canary path, before protocol, + adapter matrix, envelope reference, manual runtime, privacy. +- Reader sequence: problem → outcome → two-command path → small Controller → proof → hosts/languages + → mechanism → guarantees → advanced. +- Remove the `brand-exploration/` gallery link (line 66). Remove the "`pitot dev` configures the + host" claim (line 257). Remove `--exec ""` examples. Move the big E2E evidence + block below the quickstart. +- Add the truthful **support matrix** (Surface / Implemented / Tested / Published); every "Yes" + backed by CI or a release artifact. + +### Step 8 — Projection sync (mandatory, or CI fails) + +- After README/SDK/surface edits: `python3 scripts/build_pitot.py --write` to regenerate + `pitot-distribution/UPSTREAM.json`; verify with `--check`. Confirm the gallery link removal and + README rewrite are reflected. Keep public projection and `UPSTREAM.json` synchronized. + +--- + +## Definition of done (from the brief, mapped to steps) + +Clean checkout builds the CLI (existing); sample shell Controller runs without a published SDK dep +(Step 1, Go template uses local module) ; config registers Controller for **`shell`** (Step 1); +allow payload → exit 0, deny payload → exit 2, denied side effect never occurs (Step 4A); +`pitot dev --host kimi -- kimi -p ...` runs against a manually configured hook, one allow + one +deny, reason returns to Kimi, canaries correct (Step 4B); README first quickstart is the tested +path, no auto-host-config claim, no `--exec`→Controller, gallery link gone, install claims match +artifacts (Steps 2,6,7); existing adapter/runtime E2E stay green (Step 5); public projection + +`UPSTREAM.json` synchronized (Step 8). + +**Excluded (do not build):** browser UI, automatic global host config, more SDK languages, +LLM-based policy generation, marketplace, cloud runtime, general shell-security claims. + +## Suggested commit sequence + +1. init: add `--template`, shell-policy template, register controller under `shell` +2. init: fix next-step guidance (agent, not controller) +3. doctor: add `--host` checks (PATH/config/hook) +4. test: deterministic Kimi allow/deny adapter+control test (Test A) +5. test: opt-in real Kimi smoke + bounded JSON evidence (Test B) +6. test: workbench init-contract / build / dev hardening +7. packaging: Python `operatorstack-pitot` rename + honest version labels +8. docs: README value translation + support matrix +9. chore: regenerate UPSTREAM.json / projection sync diff --git a/labs/15-pitot/pitot-distribution/UPSTREAM.json b/labs/15-pitot/pitot-distribution/UPSTREAM.json index 4736af131..ab1fdcbc9 100644 --- a/labs/15-pitot/pitot-distribution/UPSTREAM.json +++ b/labs/15-pitot/pitot-distribution/UPSTREAM.json @@ -1,7 +1,7 @@ { "files": { "CONTRIBUTING.md": "23728d8a132d62b8adfb2e5c3eb9d9bfcf8a4d04543765b1e22ad8d55424af8f", - "README.md": "77995d36de1a6ac5f5c687fad4a152b8824ea3ad25abe65325a7c4ee9a2ef52d", + "README.md": "bd662302b629066b630dbb4c274174a4df61e98ab699417b481d7a48590de40b", "adapter-verification.json": "f8ad4e206571650f698826a8b66d8c00822be425e8d2de8ae98d98239e575eb4", "adapters/adapters.go": "1b46ba131fa3b2c93eed23526330275a3506451ba4bbd4f497e5378dfab2b6a8", "assets/pitot-boundary.png": "8a0ddb7d81831d94e14813f50ea4ca8670d77417f339ed2f91f0c653bf52f41d", @@ -14,16 +14,23 @@ "bridge/bridge.go": "5adfcd3f743cae46e4446a6e030d53464ada97de0261a8588fa2a9fcd62136b8", "bridge/bridge_test.go": "6dcc6d05f2b39c25955fc0b2d21d3d148dd9d77600fb12799941f86bdb1acb61", "cmd/generate-schema/main.go": "6e9d0030290d99e36967433f96e38385a122974f899ad9421aac1ef7e50d8fcb", - "cmd/pitot/main.go": "14de1d6bf7ef7a75172ffd015c7cab1b66c38706b879de8d354137b34596c7f3", + "cmd/pitot/doctor_host.go": "7ecade40618bfb3510ae8e55fa802361371b4f7fbafedcd61233d19ef46cb219", + "cmd/pitot/doctor_host_test.go": "4e6e327f6cf27cf94a0a608e10eb6790d6c11fcd53e6dfd7370007190749952f", + "cmd/pitot/kimi_control_test.go": "b3803a9bbdecf5f7e9a3dca90e317bdb94b4ba9bfa21d369d5a152d6742eac63", + "cmd/pitot/kimi_smoke_test.go": "6c2b92a8d3257955617d1387bc3f788846e0be091742c074a762f2b5cd04fbdf", + "cmd/pitot/main.go": "27d00919d7cc687e2b58c930024aba2ae7009a768af0abd9e73abfe123f1c8c3", "cmd/pitot/main_test.go": "544997295e0c4b75ef8f3d698b3de0883153f671b6b8f62057cc6e3452d6dc93", - "cmd/pitot/workbench.go": "2e2522491437c624b241fd594d678be2dab6824fc5aa3235db1ec82e00a669c7", + "cmd/pitot/workbench.go": "70497ca0fd5579d8c1df5350e037cb46096038449293814f995b8c209a99b215", + "cmd/pitot/workbench_build_test.go": "8e9ae497a03c9f1ca0a1fc9ebf821df3a1869993cbb2cd568da3996d437551ec", + "cmd/pitot/workbench_contract_test.go": "f88ba5a34d16d1a18fd2a4fed54c7b4cbb62fb3ffbd0eb6091e2143ba89234c5", + "cmd/pitot/workbench_dev_test.go": "9693e84f24facd7d97cefcc92d95c0e6950d9422a7c23f8b07487bbd35df2eac", "cmd/pitot/workbench_test.go": "457caa11cd4b73c1fb4e0dad806b3050b196ddac690a8125f1615a4c695cc073", "config/config.go": "e6666567d0c0cca41de69361e8f1243adda1ec0a54a9300b39a84d2290bff319", "config/config_test.go": "87d3e5ddc4a3b43c736070de671d03e03ffe29cdd759771526ad27fd9bc0034c", "conformance/conformance.go": "43b692114f45c8b52958e34b35aee1cee339d8321c90f92ab4f5b963e79935bb", "conformance/conformance_test.go": "83ab0bcc15371265a954d177e4e97d81ad3ea734bbf736a29a54628ef64b52cd", "conformance/fixtures/negative.jsonl": "503ea76988df595d96ebf695f991b8ea6c892be4a578522dff4ddb0d39b647e4", - "conformance/fixtures/positive.jsonl": "23010bb2306f90fec40dc870cd922089550dbdfc977f778561c81032f4912a4c", + "conformance/fixtures/positive.jsonl": "881efdf58b66ee7d03171c6b4410bf1bce9e1b8db5c8bf969d0e8ec467420c3f", "doc.go": "a8abdafac969b1bf4372c8bb023aa51125dc073f03218f4ab9913dfc5ffa877d", "e2e/e2e_coverage_test.go": "6235bd1df7212e4e229be324ac50f592aef70dce8855519d02e6b843656ea109", "e2e/e2e_hook_test.go": "5e184dc8907b6e36daeab90bbbb1654fa5336312866412031805a13ba535d1f8", @@ -70,7 +77,7 @@ "sdk/python/pitot/__init__.py": "9cab11b333536f167d4e7bb6089ef00488690ec7b5cd5701f08e8bbb13cef0f7", "sdk/python/pitot/runner.py": "8d4537ffc3aee2ea22b1d691559ac1eb1e95521df9ad5bf241ee6ac54c9925b1", "sdk/python/pitot/types.py": "c9a7221f1ad6627f152f26148155d3c74f249c52262c73a515251f469e8eb4ad", - "sdk/python/pyproject.toml": "4b43380a1ebc12350ee3c30633695160ba0fc0bb64501b1e16f990a6fbe56d02", + "sdk/python/pyproject.toml": "ecd3d54f2e31ae4a95ab168d6a6674d218ae5d4cf1e91bf8c64a8cc1ce3e74cb", "sdk/runner.go": "e9a8db96d3cf6df7ea7e661e6755650f97e580970995dfe881b4268db0c3f832", "sdk/rust/Cargo.toml": "b8435f6c600ad0791bd29ada4bc396b14e54b96ee3804896c8e610583cc70c1a", "sdk/rust/src/lib.rs": "fd2dcb9bf9df47fb58e52e4e94136f95867616d941ba66b4b324413d1c5b1777", diff --git a/labs/15-pitot/pitot-distribution/release-notes/2026-07-23-kimi-controlled-action.md b/labs/15-pitot/pitot-distribution/release-notes/2026-07-23-kimi-controlled-action.md new file mode 100644 index 000000000..2f8f38ee6 --- /dev/null +++ b/labs/15-pitot/pitot-distribution/release-notes/2026-07-23-kimi-controlled-action.md @@ -0,0 +1,10 @@ +### Prove one truthful Kimi controlled-action path + +Pitot now ships an end-to-end, locally reproducible proof that a coding agent's shell action can be allowed or denied through a Pitot Controller, and the README leads with it. + +- **`pitot init --template shell-policy`**: scaffolds a sample shell Controller in Go, Python, TypeScript, or Rust that requires full content mode and denies only the `PITOT_DENY_ME` canary (a demo tripwire, not a general shell-security control). The generated Rust manifest now declares `serde_json`, so the Rust project compiles offline like the other three. +- **`pitot doctor --host HOST`**: reports whether the host's PreToolUse hook is wired to `pitot hook`, prints the resolved config path, and flags fail-open gaps — the one-time host wiring the README describes. +- **`pitot dev --host kimi -- kimi -p ...`**: launches the real agent behind the runtime and prints an allow/deny decision timeline carrying the Controller's reason. +- **Tests**: a deterministic control-path test drives canonical Kimi allow/deny payloads through the built controller (asserting exit 0/2, canary side effects, and that the deny reason reaches the caller), plus dev end-to-end, multi-language build, and `doctor --host` coverage. An opt-in real-Kimi smoke test (`PITOT_KIMI_SMOKE`) emits a bounded JSON evidence artifact. + +The README now opens with "Keep your coding agent. Add the behavior it is missing.", walks through the tested Kimi allow/deny canary, replaces the false "`pitot dev` configures the host for you" claim with truthful one-time hook wiring, and adds a supported-hosts matrix keyed to what is verified in this repo. diff --git a/labs/15-pitot/pitot-distribution/sdk/python/pyproject.toml b/labs/15-pitot/pitot-distribution/sdk/python/pyproject.toml index 0cdaab29c..8177fa88a 100644 --- a/labs/15-pitot/pitot-distribution/sdk/python/pyproject.toml +++ b/labs/15-pitot/pitot-distribution/sdk/python/pyproject.toml @@ -2,8 +2,11 @@ requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" +# The bare name "pitot" is already taken on PyPI by an unrelated aeronautics +# project, so the distribution is published as operatorstack-pitot. The importable +# package stays "pitot" (packages below), so `import pitot` is unchanged. [project] -name = "pitot" +name = "operatorstack-pitot" version = "0.1.0" description = "Pitot: The passive, protocol-first measurement boundary for coding agents." authors = [ @@ -13,6 +16,9 @@ license = { text = "MIT" } requires-python = ">=3.10" dependencies = [] +[tool.setuptools] +packages = ["pitot"] + [project.urls] Homepage = "https://github.com/operatorstack/pitot" Repository = "https://github.com/operatorstack/pitot" diff --git a/labs/15-pitot/pitot/cmd/pitot/doctor_host.go b/labs/15-pitot/pitot/cmd/pitot/doctor_host.go new file mode 100644 index 000000000..c10095733 --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/doctor_host.go @@ -0,0 +1,129 @@ +package main + +import ( + "context" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "github.com/operatorstack/pitot/adapters" +) + +// hostProbe describes what `pitot doctor --host` inspects for a coding agent: +// the agent binary, where its config lives, and the hook it must declare so its +// native blocking boundary reaches `pitot hook `. +type hostProbe struct { + binary string + configPath func() (string, error) + hookEvent string + matcher string +} + +// hostProbes holds per-host inspection metadata. Only Kimi is wired for this +// release; other hosts fall back to a decoder-only note. +var hostProbes = map[adapters.Host]hostProbe{ + adapters.Kimi: { + binary: "kimi", + configPath: func() (string, error) { + if home := os.Getenv("KIMI_CODE_HOME"); home != "" { + return filepath.Join(home, "config.toml"), nil + } + home, err := os.UserHomeDir() + if err != nil { + return "", err + } + return filepath.Join(home, ".kimi-code", "config.toml"), nil + }, + hookEvent: "PreToolUse", + matcher: "Bash", + }, +} + +// doctorHost reports whether a host is configured to route its blocking shell +// boundary to Pitot. It never edits configuration — it only inspects and +// reports, returning a non-nil error when a blocking issue is found so callers +// (and CI) get a clear signal. +func doctorHost(host adapters.Host, stdout, stderr io.Writer) error { + if !adapters.IsSupported(host) { + return fmt.Errorf("pitot doctor: unsupported host %q (want one of: %s)", host, hostList()) + } + fmt.Fprintf(stdout, "Pitot %s — host check: %s\n", adapters.AdapterVersion, host) + + probe, known := hostProbes[host] + if !known { + fmt.Fprintf(stdout, " host-config inspection is not implemented for %q in this release; run `pitot doctor` for the decoder status\n", host) + return nil + } + + var problems []string + + // 1. Agent binary on PATH. + if path, err := exec.LookPath(probe.binary); err == nil { + fmt.Fprintf(stdout, " binary on PATH: %s\n", path) + } else { + fmt.Fprintf(stdout, " binary on PATH: NOT FOUND (%s)\n", probe.binary) + problems = append(problems, fmt.Sprintf("%q not on PATH", probe.binary)) + } + + // 2. Config path resolvable. + cfgPath, err := probe.configPath() + if err != nil { + fmt.Fprintf(stdout, " config path: UNRESOLVED (%v)\n", err) + problems = append(problems, "config path unresolved") + } else { + fmt.Fprintf(stdout, " config path: %s\n", cfgPath) + } + + // 3. Hook present and invoking `pitot hook `. + hookCmd := "hook " + string(host) + if cfgPath != "" { + data, readErr := os.ReadFile(cfgPath) + switch { + case readErr != nil: + fmt.Fprintf(stdout, " config file: ABSENT (%v)\n", readErr) + problems = append(problems, "config file absent") + default: + text := string(data) + hasEvent := strings.Contains(text, `event = "`+probe.hookEvent+`"`) + hasHookCmd := strings.Contains(text, "pitot "+hookCmd) || strings.Contains(text, hookCmd) + if hasEvent && hasHookCmd { + fmt.Fprintf(stdout, " %s hook: FOUND invoking `pitot %s`\n", probe.hookEvent, hookCmd) + } else { + fmt.Fprintf(stdout, " %s hook: MISSING — add a [[hooks]] entry (event = %q, matcher = %q, command = \"pitot %s\")\n", + probe.hookEvent, probe.hookEvent, probe.matcher, hookCmd) + problems = append(problems, "hook not configured") + } + } + } + + // 4. Config parses — best effort via the agent's own doctor, if present. + if _, err := exec.LookPath(probe.binary); err == nil { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + out, derr := exec.CommandContext(ctx, probe.binary, "doctor").CombinedOutput() + if derr == nil { + fmt.Fprintf(stdout, " config parses: PASS (%s doctor)\n", probe.binary) + } else { + fmt.Fprintf(stdout, " config parses: FAIL (%s doctor: %v)\n", probe.binary, derr) + if len(out) > 0 { + fmt.Fprintf(stderr, " %s doctor: %s\n", probe.binary, strings.TrimSpace(string(out))) + } + problems = append(problems, "config did not parse") + } + } else { + fmt.Fprintf(stdout, " config parses: skipped (%s not available)\n", probe.binary) + } + + // Kimi's PreToolUse hook is fail-open on crash/timeout per host semantics. + fmt.Fprintf(stdout, " note: %s hooks are fail-open on hook crash or timeout per host semantics; this sample controller is not a security sandbox\n", host) + + if len(problems) > 0 { + return fmt.Errorf("pitot doctor: %s host check found %d issue(s): %s", host, len(problems), strings.Join(problems, "; ")) + } + fmt.Fprintf(stdout, " ready: %s can route its shell boundary to Pitot\n", host) + return nil +} diff --git a/labs/15-pitot/pitot/cmd/pitot/doctor_host_test.go b/labs/15-pitot/pitot/cmd/pitot/doctor_host_test.go new file mode 100644 index 000000000..19c821eec --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/doctor_host_test.go @@ -0,0 +1,63 @@ +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" +) + +const kimiHookConfig = `default_model = "pitot-control" +[[hooks]] +event = "PreToolUse" +matcher = "Bash" +command = "pitot hook kimi" +timeout = 5 +` + +func TestDoctorHostKimiReportsConfiguredHook(t *testing.T) { + home := t.TempDir() + t.Setenv("KIMI_CODE_HOME", home) + if err := os.WriteFile(filepath.Join(home, "config.toml"), []byte(kimiHookConfig), 0o600); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + // Return value may be non-nil (the kimi binary is absent in CI); we assert on + // the reported diagnostics, which must recognize the configured hook. + _ = doctor([]string{"--host", "kimi"}, &stdout, &stderr) + out := stdout.String() + for _, want := range []string{ + "host check: kimi", + filepath.Join(home, "config.toml"), + "PreToolUse hook: FOUND", + "fail-open", + } { + if !strings.Contains(out, want) { + t.Errorf("doctor --host kimi output missing %q:\n%s", want, out) + } + } +} + +func TestDoctorHostKimiReportsMissingHook(t *testing.T) { + home := t.TempDir() + t.Setenv("KIMI_CODE_HOME", home) + if err := os.WriteFile(filepath.Join(home, "config.toml"), []byte("default_model = \"x\"\n"), 0o600); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + err := doctor([]string{"--host", "kimi"}, &stdout, &stderr) + if err == nil { + t.Fatal("expected doctor to report an issue when the hook is missing") + } + if !strings.Contains(stdout.String(), "PreToolUse hook: MISSING") { + t.Errorf("expected MISSING hook diagnostic:\n%s", stdout.String()) + } +} + +func TestDoctorHostRejectsUnknownHost(t *testing.T) { + var stdout, stderr bytes.Buffer + if err := doctor([]string{"--host", "notahost"}, &stdout, &stderr); err == nil { + t.Fatal("expected unsupported host to be rejected") + } +} diff --git a/labs/15-pitot/pitot/cmd/pitot/kimi_control_test.go b/labs/15-pitot/pitot/cmd/pitot/kimi_control_test.go new file mode 100644 index 000000000..1c05f34b7 --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/kimi_control_test.go @@ -0,0 +1,185 @@ +package main + +import ( + "bytes" + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + goruntime "runtime" + "strings" + "testing" + "time" +) + +// buildGeneratedShellPolicy scaffolds a shell-policy Go controller with `pitot +// init`, points it at the in-tree module with a filesystem replace, and compiles +// it offline. This proves the *generated* sample Controller builds and runs +// without a published SDK dependency (Definition of Done), rather than testing a +// hand-written stand-in. +func buildGeneratedShellPolicy(t *testing.T) (binPath, configWithBinary string) { + t.Helper() + + proj := filepath.Join(t.TempDir(), "shell-policy-proj") + var out, errb bytes.Buffer + if err := runInit([]string{"--language", "go", "--template", "shell-policy", "--dir", proj}, strings.NewReader(""), &out, &errb); err != nil { + t.Fatalf("init shell-policy: %v\n%s", err, errb.String()) + } + + // The generated config must register the controller under the shell kind. + cfg, err := os.ReadFile(filepath.Join(proj, ".pitot.yaml")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(cfg), "shell:") || !strings.Contains(string(cfg), "local-shell-policy") { + t.Fatalf("generated config does not register the shell-policy controller under shell:\n%s", cfg) + } + + // Resolve the in-tree module root (cmd/pitot -> module root) and add a + // filesystem replace so the generated project resolves the SDK locally. + moduleRoot, err := filepath.Abs(filepath.Join("..", "..")) + if err != nil { + t.Fatal(err) + } + gomod := filepath.Join(proj, "go.mod") + existing, err := os.ReadFile(gomod) + if err != nil { + t.Fatal(err) + } + replace := fmt.Sprintf("\nreplace github.com/operatorstack/pitot => %s\n", moduleRoot) + if err := os.WriteFile(gomod, append(existing, []byte(replace)...), 0o644); err != nil { + t.Fatal(err) + } + + name := "shell-policy" + if goruntime.GOOS == "windows" { + name += ".exe" + } + binPath = filepath.Join(proj, name) + build := exec.Command("go", "build", "-o", binPath, ".") + build.Dir = proj + build.Env = append(os.Environ(), "GOFLAGS=-mod=mod", "GOPROXY=off") + if output, err := build.CombinedOutput(); err != nil { + t.Fatalf("build generated shell-policy controller: %v\n%s", err, output) + } + + // A config that runs the compiled controller by absolute path, avoiding a + // per-spawn `go run` compile and any working-directory coupling. + configWithBinary = fmt.Sprintf(`controllers: + shell: + id: local-shell-policy + command: [%q] + deadline_ms: 2000 + on_timeout: deny + on_unavailable: deny +`, binPath) + return binPath, configWithBinary +} + +// TestKimiShellPolicyAllowAndDeny is the deterministic (no-model) proof of the +// controlled-action path: a canonical Kimi PreToolUse/Bash payload is allowed +// and its canary runs, while the PITOT_DENY_ME payload is blocked (exit 2), its +// canary never runs, and the Controller's reason reaches the caller. +func TestKimiShellPolicyAllowAndDeny(t *testing.T) { + if goruntime.GOOS == "windows" { + // The canary harness drives POSIX shell commands (`sh -c`, `printf`, + // `VAR=1 sh -c ...`) with Unix path semantics; Git Bash on Windows mangles + // the backslashed temp paths. The control path itself is exercised on the + // POSIX runners. + t.Skip("canary harness uses POSIX shell commands and Unix path semantics") + } + t.Setenv("PITOT_RUNTIME", "") + + _, configBody := buildGeneratedShellPolicy(t) + + dir := t.TempDir() + configPath := filepath.Join(dir, ".pitot.yaml") + runtimePath := filepath.Join(dir, "runtime.json") + if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil { + t.Fatal(err) + } + + // Per-test canary paths keep the test hermetic (no shared /tmp files). + allowedCanary := filepath.Join(dir, "pitot-allowed-canary") + deniedCanary := filepath.Join(dir, "pitot-denied-canary") + + // Start the runtime backing the shell controller. + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var runtimeOut bytes.Buffer + var runtimeErr lockedBuffer + done := make(chan error, 1) + go func() { + done <- runWithIO(ctx, []string{"run", "--config", configPath, "--runtime", runtimePath}, strings.NewReader(""), &runtimeOut, &runtimeErr) + }() + waitForRuntimeFile(t, runtimePath, &runtimeErr) + + // --- Allow path: hook returns 0, then the harness executes the canary. --- + allowCmd := fmt.Sprintf("printf allowed > %s", allowedCanary) + allowPayload := fmt.Sprintf(`{"hook_event_name":"PreToolUse","session_id":"pitot-local-test","tool_name":"Bash","tool_input":{"command":%q}}`, allowCmd) + var allowOut, allowErrOut bytes.Buffer + allowErr := runWithIO(context.Background(), []string{"hook", "kimi", "--runtime", runtimePath}, strings.NewReader(allowPayload), &allowOut, &allowErrOut) + if allowErr != nil { + t.Fatalf("allow hook: expected exit 0, got err=%v stderr=%s", allowErr, allowErrOut.String()) + } + if !strings.Contains(allowOut.String(), `"kind":"shell"`) { + t.Errorf("allow receipt missing kind=shell:\n%s", allowOut.String()) + } + // Host execution is simulated ONLY after an allow decision. + runCanary(t, allowCmd) + if got, err := os.ReadFile(allowedCanary); err != nil || string(got) != "allowed" { + t.Fatalf("allowed canary: want %q, got %q (err=%v)", "allowed", string(got), err) + } + + // --- Deny path: hook returns exit 2 and the canary is never executed. --- + denyCmd := fmt.Sprintf("PITOT_DENY_ME=1 sh -c 'printf blocked > %s'", deniedCanary) + denyPayload := fmt.Sprintf(`{"hook_event_name":"PreToolUse","session_id":"pitot-local-test","tool_name":"Bash","tool_input":{"command":%q}}`, denyCmd) + var denyOut, denyErrOut bytes.Buffer + denyErr := runWithIO(context.Background(), []string{"hook", "kimi", "--runtime", runtimePath}, strings.NewReader(denyPayload), &denyOut, &denyErrOut) + if !errors.Is(denyErr, errBlocked) { + t.Fatalf("deny hook: expected errBlocked (exit 2), got err=%v stdout=%s stderr=%s", denyErr, denyOut.String(), denyErrOut.String()) + } + if !strings.Contains(denyOut.String(), `"kind":"shell"`) { + t.Errorf("deny receipt missing kind=shell:\n%s", denyOut.String()) + } + // The Controller's exact reason must reach the caller (Kimi) via stderr. + if !strings.Contains(denyErrOut.String(), "PITOT_DENY_ME canary") { + t.Errorf("deny stderr missing Controller reason:\n%s", denyErrOut.String()) + } + // Because the hook denied, the harness must NOT execute the command. + if denyErr == nil { + runCanary(t, denyCmd) + } + if _, err := os.Stat(deniedCanary); !os.IsNotExist(err) { + t.Fatalf("denied canary must be absent, stat err=%v", err) + } + + cancel() + if err := <-done; err != nil { + t.Fatal(err) + } +} + +// waitForRuntimeFile blocks until the runtime descriptor is published. +func waitForRuntimeFile(t *testing.T, runtimePath string, runtimeErr *lockedBuffer) { + t.Helper() + for i := 0; i < 250; i++ { + if _, err := os.Stat(runtimePath); err == nil { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatalf("runtime did not become ready:\n%s", runtimeErr.String()) +} + +// runCanary simulates the host executing an allowed command. It is only ever +// invoked after an allow decision, mirroring how a real host runs a command the +// PreToolUse hook approved. +func runCanary(t *testing.T, command string) { + t.Helper() + if output, err := exec.Command("sh", "-c", command).CombinedOutput(); err != nil { + t.Fatalf("canary command %q failed: %v\n%s", command, err, output) + } +} diff --git a/labs/15-pitot/pitot/cmd/pitot/kimi_smoke_test.go b/labs/15-pitot/pitot/cmd/pitot/kimi_smoke_test.go new file mode 100644 index 000000000..dac66db02 --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/kimi_smoke_test.go @@ -0,0 +1,257 @@ +package main + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + goruntime "runtime" + "strings" + "testing" + "time" +) + +// kimiEvidence is the bounded, content-safe artifact emitted by the real-Kimi +// smoke test. It carries only identities, hashes, decision metadata, and canary +// states — never raw commands or secrets — so a run can be proven after the fact. +type kimiEvidence struct { + Schema string `json:"schema"` + PitotCommit string `json:"pitot_commit"` + PitotBinarySHA256 string `json:"pitot_binary_sha256"` + KimiVersion string `json:"kimi_version"` + HookConfigSHA256 string `json:"hook_config_sha256"` + RuntimeDescriptor string `json:"runtime_descriptor"` + Runs []kimiRun `json:"runs"` +} + +type kimiRun struct { + Name string `json:"name"` + Prompt string `json:"prompt"` + ExitStatus int `json:"exit_status"` + Decisions []decisionRecord `json:"decisions"` + CanaryPath string `json:"canary_path"` + CanaryPresent bool `json:"canary_present"` + CanaryContent string `json:"canary_content,omitempty"` + KimiTextTail string `json:"kimi_text_tail"` +} + +type decisionRecord struct { + Outcome string `json:"outcome"` + Kind string `json:"kind"` + ActionID string `json:"action_id"` + Message string `json:"message"` +} + +const kimiSmokeHookConfig = `[[hooks]] +event = "PreToolUse" +matcher = "Bash" +command = "pitot hook kimi" +timeout = 5 +` + +var runtimeDescriptorRE = regexp.MustCompile(`(?m)^runtime:\s+(.+)$`) + +// TestKimiSmokeRealCLI is the opt-in (Test B) proof that the *real* Kimi CLI +// honors the Pitot PreToolUse hook end to end. It is skipped unless +// PITOT_KIMI_SMOKE is set and a `kimi` binary is on PATH, because it needs a live, +// authenticated Kimi install. It runs one allow prompt and one deny prompt through +// `pitot dev --host kimi -- kimi -p ...`, asserts the canary side effects, and +// writes a bounded JSON evidence artifact (path logged, or PITOT_KIMI_EVIDENCE). +func TestKimiSmokeRealCLI(t *testing.T) { + if os.Getenv("PITOT_KIMI_SMOKE") == "" { + t.Skip("set PITOT_KIMI_SMOKE=1 to run the real-Kimi smoke test (needs an authenticated kimi CLI)") + } + if goruntime.GOOS == "windows" { + t.Skip("smoke harness uses a POSIX config path and canary commands") + } + if _, err := exec.LookPath("kimi"); err != nil { + t.Skip("kimi binary not on PATH; skipping real-Kimi smoke test") + } + + pitotBin := buildPitotBinary(t) + _, configBody := buildGeneratedShellPolicy(t) + + proj := t.TempDir() + if err := os.WriteFile(filepath.Join(proj, ".pitot.yaml"), []byte(configBody), 0o600); err != nil { + t.Fatal(err) + } + + // A private Kimi home whose config.toml wires the PreToolUse hook to pitot. + kimiHome := t.TempDir() + hookConfigPath := filepath.Join(kimiHome, "config.toml") + if err := os.WriteFile(hookConfigPath, []byte(kimiSmokeHookConfig), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("KIMI_CODE_HOME", kimiHome) + // Make the freshly built pitot resolvable to Kimi's `pitot hook kimi` hook. + t.Setenv("PATH", filepath.Dir(pitotBin)+string(os.PathListSeparator)+os.Getenv("PATH")) + t.Setenv("PITOT_RUNTIME", "") + + evidence := kimiEvidence{ + Schema: "pitot.kimi-smoke-evidence/1", + PitotCommit: gitCommit(t), + PitotBinarySHA256: sha256File(t, pitotBin), + KimiVersion: commandOutput(t, "kimi", "--version"), + HookConfigSHA256: sha256Bytes([]byte(kimiSmokeHookConfig)), + } + + allowCanary := filepath.Join(proj, "kimi-allow-canary") + denyCanary := filepath.Join(proj, "kimi-deny-canary") + + allowRun := runKimiSmoke(t, pitotBin, proj, kimiRunSpec{ + name: "allow", + prompt: fmt.Sprintf("Run exactly this shell command and nothing else: printf allowed > %s", allowCanary), + canaryPath: allowCanary, + }) + denyRun := runKimiSmoke(t, pitotBin, proj, kimiRunSpec{ + name: "deny", + prompt: fmt.Sprintf("Run exactly this shell command and nothing else: PITOT_DENY_ME=1 printf blocked > %s", denyCanary), + canaryPath: denyCanary, + }) + evidence.Runs = []kimiRun{allowRun, denyRun} + if d := firstRuntimeDescriptor(allowRun, denyRun); d != "" { + evidence.RuntimeDescriptor = d + } + + // Persist the evidence artifact before asserting, so a failing run is still + // documented. + evidencePath := os.Getenv("PITOT_KIMI_EVIDENCE") + if evidencePath == "" { + evidencePath = filepath.Join(proj, "kimi-evidence.json") + } + blob, err := json.MarshalIndent(evidence, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(evidencePath, blob, 0o600); err != nil { + t.Fatalf("write evidence: %v", err) + } + t.Logf("kimi smoke evidence written to %s:\n%s", evidencePath, blob) + + // Postconditions. Allow path: the command ran, the canary exists. + if !allowRun.CanaryPresent || allowRun.CanaryContent != "allowed" { + t.Errorf("allow run: canary = present:%v content:%q, want present:true content:%q", allowRun.CanaryPresent, allowRun.CanaryContent, "allowed") + } + // Deny path: the command was blocked, the canary never appeared, and a shell + // deny carrying the Controller reason reached the decision timeline. + if denyRun.CanaryPresent { + t.Errorf("deny run: canary must be absent, found content %q", denyRun.CanaryContent) + } + if !hasDeny(denyRun.Decisions) { + t.Errorf("deny run: no shell deny decision observed:\n%+v", denyRun.Decisions) + } +} + +type kimiRunSpec struct { + name string + prompt string + canaryPath string +} + +// runKimiSmoke executes one `pitot dev --host kimi -- kimi -p PROMPT` run and +// collects its evidence. +func runKimiSmoke(t *testing.T, pitotBin, projDir string, spec kimiRunSpec) kimiRun { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) + defer cancel() + + var combined lockedBuffer + cmd := exec.CommandContext(ctx, pitotBin, "dev", "--host", "kimi", "--", "kimi", "-p", spec.prompt) + cmd.Dir = projDir + cmd.Stdout = &combined + cmd.Stderr = &combined + runErr := cmd.Run() + + exitStatus := 0 + if runErr != nil { + var exitErr *exec.ExitError + if errors.As(runErr, &exitErr) { + exitStatus = exitErr.ExitCode() + } else { + exitStatus = -1 + t.Logf("%s run: pitot dev did not complete cleanly: %v", spec.name, runErr) + } + } + + out := combined.String() + run := kimiRun{ + Name: spec.name, + Prompt: spec.prompt, + ExitStatus: exitStatus, + CanaryPath: spec.canaryPath, + KimiTextTail: tail(out, 4000), + } + for _, d := range parseDecisions(out) { + run.Decisions = append(run.Decisions, decisionRecord{Outcome: d.outcome, Kind: d.kind, ActionID: d.actionID, Message: d.message}) + } + if data, err := os.ReadFile(spec.canaryPath); err == nil { + run.CanaryPresent = true + run.CanaryContent = string(data) + } + return run +} + +func hasDeny(decisions []decisionRecord) bool { + for _, d := range decisions { + if d.Outcome == "DENY" && d.Kind == "shell" { + return true + } + } + return false +} + +func firstRuntimeDescriptor(runs ...kimiRun) string { + for _, r := range runs { + if m := runtimeDescriptorRE.FindStringSubmatch(r.KimiTextTail); len(m) == 2 { + return strings.TrimSpace(m[1]) + } + } + return "" +} + +// gitCommit returns the current HEAD, or "unknown" when git is unavailable. +func gitCommit(t *testing.T) string { + t.Helper() + out, err := exec.Command("git", "rev-parse", "HEAD").Output() + if err != nil { + return "unknown" + } + return strings.TrimSpace(string(out)) +} + +func commandOutput(t *testing.T, name string, args ...string) string { + t.Helper() + out, err := exec.Command(name, args...).CombinedOutput() + if err != nil { + return "unknown" + } + return strings.TrimSpace(string(out)) +} + +func sha256File(t *testing.T, path string) string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("hash %s: %v", path, err) + } + return sha256Bytes(data) +} + +func sha256Bytes(data []byte) string { + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} + +// tail returns at most the last n bytes of s, prefixed to signal truncation. +func tail(s string, n int) string { + if len(s) <= n { + return s + } + return "...(truncated)...\n" + s[len(s)-n:] +} diff --git a/labs/15-pitot/pitot/cmd/pitot/main.go b/labs/15-pitot/pitot/cmd/pitot/main.go index ffa67f082..ad6ceb586 100644 --- a/labs/15-pitot/pitot/cmd/pitot/main.go +++ b/labs/15-pitot/pitot/cmd/pitot/main.go @@ -46,7 +46,7 @@ func runWithIO(ctx context.Context, args []string, stdin io.Reader, stdout, stde case "dev": return runDev(ctx, args[1:], stdout, stderr) case "doctor": - return doctor(stdout) + return doctor(args[1:], stdout, stderr) case "run": return runRuntime(ctx, args[1:], stdout, stderr) case "hook": @@ -196,7 +196,24 @@ func runRequest(ctx context.Context, args []string, stdout io.Writer) error { return nil } -func doctor(stdout io.Writer) error { +func doctor(args []string, stdout, stderr io.Writer) error { + host := "" + for i := 0; i < len(args); i++ { + switch args[i] { + case "--host": + if i+1 >= len(args) { + return errors.New("pitot doctor: --host requires a value") + } + host = args[i+1] + i++ + default: + return fmt.Errorf("pitot doctor: unexpected argument %q", args[i]) + } + } + if host != "" { + return doctorHost(adapters.Host(host), stdout, stderr) + } + fmt.Fprintf(stdout, "Pitot %s — local boundary\n", schema.Version) fmt.Fprintf(stdout, "adapter version: %s\n", adapters.AdapterVersion) fmt.Fprintln(stdout, "unauthenticated local socket: none") @@ -283,9 +300,9 @@ func usage() string { return `pitot — the open sensor and control transport for coding-agent tooling usage: - pitot init [--language python|typescript|go|rust] [--role consumer|controller] [--dir PATH] [--force] - pitot dev --host HOST --exec "CMD ARGS" - pitot doctor + pitot init [--language python|typescript|go|rust] [--role consumer|controller] [--template shell-policy|release-approval|blank-controller|blank-consumer] [--dir PATH] [--force] + pitot dev --host HOST -- AGENT [ARGS...] + pitot doctor [--host HOST] pitot run --config PATH --runtime PATH pitot hook HOST [--runtime PATH] pitot request KIND [--data JSON] --runtime PATH diff --git a/labs/15-pitot/pitot/cmd/pitot/workbench.go b/labs/15-pitot/pitot/cmd/pitot/workbench.go index 124ca23bf..ce40994b4 100644 --- a/labs/15-pitot/pitot/cmd/pitot/workbench.go +++ b/labs/15-pitot/pitot/cmd/pitot/workbench.go @@ -21,9 +21,20 @@ import ( // manifests so a freshly initialized project can resolve its dependency. const pitotPackageVersion = "0.1.0" +// pitotPythonDistribution is the PyPI distribution name for the Python SDK. The +// bare name "pitot" is already taken on PyPI by an unrelated aeronautics +// project, so the distribution is published as operatorstack-pitot while the +// importable package remains "pitot". +const pitotPythonDistribution = "operatorstack-pitot" + var supportedLanguages = []string{"python", "typescript", "go", "rust"} var supportedRoles = []string{"consumer", "controller"} +// supportedTemplates enumerates the project scaffolds pitot init can emit. +// shell-policy is the only one that registers a controller for the shell action +// kind; the others target the test.approval kind or the consumer role. +var supportedTemplates = []string{"shell-policy", "release-approval", "blank-controller", "blank-consumer"} + // runInit scaffolds a complete, runnable Pitot project. It validates its inputs, // detects or interactively selects the language and role, refuses to overwrite // existing files unless --force is set, and writes a package manifest alongside @@ -31,6 +42,7 @@ var supportedRoles = []string{"consumer", "controller"} func runInit(args []string, stdin io.Reader, stdout, stderr io.Writer) error { lang := "" role := "" + template := "" dir := "pitot-project" force := false @@ -48,6 +60,12 @@ func runInit(args []string, stdin io.Reader, stdout, stderr io.Writer) error { } role = args[i+1] i++ + case "--template": + if i+1 >= len(args) { + return fmt.Errorf("pitot init: --template requires a value (%s)", strings.Join(supportedTemplates, ", ")) + } + template = args[i+1] + i++ case "--dir": if i+1 >= len(args) { return errors.New("pitot init: --dir requires a path") @@ -83,6 +101,12 @@ func runInit(args []string, stdin io.Reader, stdout, stderr io.Writer) error { return fmt.Errorf("pitot init: unsupported language %q (want python, typescript, go, rust)", lang) } + // An explicit template implies its role, so we never prompt for a role the + // template already dictates. + if role == "" && template != "" && contains(supportedTemplates, template) { + role = templateRole(template) + } + // Resolve role: explicit flag, then prompt, then default. if role == "" { if interactive { @@ -99,7 +123,15 @@ func runInit(args []string, stdin io.Reader, stdout, stderr io.Writer) error { return fmt.Errorf("pitot init: unsupported role %q (want consumer, controller)", role) } - files, err := projectFiles(lang, role) + // Resolve template: explicit flag validated for consistency with the role, + // otherwise defaulted from the role so existing (role-only) callers keep the + // blank scaffolds they got before templates existed. + template, err := resolveTemplate(role, template) + if err != nil { + return err + } + + files, err := projectFiles(lang, template) if err != nil { return err } @@ -132,61 +164,150 @@ func runInit(args []string, stdin io.Reader, stdout, stderr io.Writer) error { } } - fmt.Fprintf(stdout, "Initialized %s %s in %s\n", lang, role, dir) + fmt.Fprintf(stdout, "Initialized %s %s (%s) in %s\n", lang, role, template, dir) fmt.Fprintf(stdout, "Files written: %s\n", strings.Join(sorted(keys(files)), ", ")) - fmt.Fprintf(stdout, "Next: cd %s && pitot dev --host claude --exec %q\n", dir, runCommandString(lang)) + // The runtime launches the generated program from .pitot.yaml; the command + // after `--` is the coding agent Pitot supervises, never the Controller. + fmt.Fprintln(stdout, "Next:") + fmt.Fprintf(stdout, " 1. cd %s\n", dir) + fmt.Fprintln(stdout, " 2. Configure a supported host hook (see: pitot doctor --host HOST).") + fmt.Fprintln(stdout, " 3. Run: pitot dev --host HOST -- AGENT [ARGS...]") + fmt.Fprintln(stdout, " example: pitot dev --host kimi -- kimi -p \"\"") return nil } -// projectFiles returns the complete file set for a language/role: source, +// resolveTemplate reconciles the explicit --template value with the resolved +// role. An empty template defaults from the role (controller -> blank-controller, +// consumer -> blank-consumer); an explicit template must not contradict an +// explicit role. +func resolveTemplate(role, template string) (string, error) { + if template == "" { + if role == "consumer" { + return "blank-consumer", nil + } + return "blank-controller", nil + } + if !contains(supportedTemplates, template) { + return "", fmt.Errorf("pitot init: unsupported template %q (want %s)", template, strings.Join(supportedTemplates, ", ")) + } + if templateRole(template) != role { + return "", fmt.Errorf("pitot init: template %q implies role %q but role %q was requested", template, templateRole(template), role) + } + return template, nil +} + +// templateRole reports the role a template scaffolds. +func templateRole(template string) string { + if template == "blank-consumer" { + return "consumer" + } + return "controller" +} + +// controllerKind returns the .pitot.yaml request kind a controller template +// registers under. Only shell-policy governs the shell action kind that host +// adapters (Kimi, Claude, Codex, ...) normalize Bash/PreToolUse events into. +func controllerKind(template string) string { + if template == "shell-policy" { + return "shell" + } + return "test.approval" +} + +// controllerID returns the controller id embedded in the generated config and +// source for a template. +func controllerID(template string) string { + if template == "shell-policy" { + return "local-shell-policy" + } + return "local-controller" +} + +// projectFiles returns the complete file set for a language/template: source, // package manifest(s), and the .pitot.yaml runtime configuration. -func projectFiles(lang, role string) (map[string]string, error) { +func projectFiles(lang, template string) (map[string]string, error) { files := map[string]string{} - controller := role == "controller" + src, err := sourceTemplate(lang, template) + if err != nil { + return nil, err + } switch lang { case "python": - if controller { - files["main.py"] = pythonControllerTemplate - } else { - files["main.py"] = pythonConsumerTemplate - } - files["requirements.txt"] = fmt.Sprintf("pitot>=%s\n", pitotPackageVersion) + files["main.py"] = src + files["requirements.txt"] = fmt.Sprintf("%s>=%s\n", pitotPythonDistribution, pitotPackageVersion) files["pyproject.toml"] = pythonProjectManifest case "typescript": - if controller { - files["main.ts"] = tsControllerTemplate - } else { - files["main.ts"] = tsConsumerTemplate - } + files["main.ts"] = src files["package.json"] = tsProjectManifest files["tsconfig.json"] = tsProjectTSConfig case "go": - if controller { - files["main.go"] = goControllerTemplate - } else { - files["main.go"] = goConsumerTemplate - } + files["main.go"] = src files["go.mod"] = goProjectManifest case "rust": - if controller { - files["main.rs"] = rustControllerTemplate - } else { - files["main.rs"] = rustConsumerTemplate - } + files["main.rs"] = src files["Cargo.toml"] = rustProjectManifest default: return nil, fmt.Errorf("pitot init: unsupported language %q", lang) } - files[".pitot.yaml"] = pitotConfig(lang, role) + files[".pitot.yaml"] = pitotConfig(lang, template) return files, nil } -// pitotConfig renders the .pitot.yaml wiring the generated role to its run command. -func pitotConfig(lang, role string) string { +// sourceTemplate selects the program source for a language/template pair. +func sourceTemplate(lang, template string) (string, error) { + shellPolicy := template == "shell-policy" + consumer := template == "blank-consumer" + switch lang { + case "python": + switch { + case consumer: + return pythonConsumerTemplate, nil + case shellPolicy: + return pythonShellPolicyTemplate, nil + default: + return pythonControllerTemplate, nil + } + case "typescript": + switch { + case consumer: + return tsConsumerTemplate, nil + case shellPolicy: + return tsShellPolicyTemplate, nil + default: + return tsControllerTemplate, nil + } + case "go": + switch { + case consumer: + return goConsumerTemplate, nil + case shellPolicy: + return goShellPolicyTemplate, nil + default: + return goControllerTemplate, nil + } + case "rust": + switch { + case consumer: + return rustConsumerTemplate, nil + case shellPolicy: + return rustShellPolicyTemplate, nil + default: + return rustControllerTemplate, nil + } + default: + return "", fmt.Errorf("pitot init: unsupported language %q", lang) + } +} + +// pitotConfig renders the .pitot.yaml wiring the generated template to its run +// command. Consumers subscribe to events; controllers register for a request +// kind — shell-policy under "shell" (the kind host adapters normalize Bash +// events into), every other controller under "test.approval". +func pitotConfig(lang, template string) string { cmdList := runCommandList(lang) - if role == "consumer" { + if templateRole(template) == "consumer" { return `consumers: - id: local-consumer command: ` + cmdList + ` @@ -196,13 +317,13 @@ func pitotConfig(lang, role string) string { ` } return fmt.Sprintf(`controllers: - test.approval: - id: local-controller + %s: + id: %s command: %s deadline_ms: 2000 on_timeout: deny on_unavailable: deny -`, cmdList) +`, controllerKind(template), controllerID(template), cmdList) } // runCommandList is the JSON array form embedded in .pitot.yaml. @@ -221,22 +342,6 @@ func runCommandList(lang string) string { } } -// runCommandString is the human-readable command shown in the init next-step hint. -func runCommandString(lang string) string { - switch lang { - case "python": - return "python3 main.py" - case "typescript": - return "npx tsx main.ts" - case "go": - return "go run main.go" - case "rust": - return "cargo run --quiet" - default: - return "" - } -} - // detectLanguage inspects an existing directory for a language's marker manifest. func detectLanguage(dir string) string { markers := []struct { @@ -345,7 +450,8 @@ build-backend = "setuptools.build_meta" name = "pitot-project" version = "0.1.0" requires-python = ">=3.10" -dependencies = ["pitot>=0.1.0"] +# The PyPI distribution is operatorstack-pitot; the import package is "pitot". +dependencies = ["operatorstack-pitot>=0.1.0"] ` const tsControllerTemplate = `import { runController, allow, ControlRequested } from '@operatorstack/pitot'; @@ -463,6 +569,102 @@ path = "main.rs" [dependencies] pitot = "0.1.0" +serde_json = "1" +` + +// The shell-policy templates below decode the Pitot Event carried in the control +// request, require full content projection, extract the normalized shell command, +// and deny only the PITOT_DENY_ME canary. The substring check is a sample +// tripwire, NOT production-grade shell security. + +const goShellPolicyTemplate = `package main + +import ( + "encoding/json" + "strings" + + "github.com/operatorstack/pitot/schema" + "github.com/operatorstack/pitot/sdk" +) + +func main() { + sdk.RunController("local-shell-policy", func(req schema.ControlRequested) sdk.Outcome { + var event schema.Event + if err := json.Unmarshal(req.Data, &event); err != nil { + return sdk.Deny("Pitot sample policy could not decode the event.") + } + if event.Content == nil || event.Content.Mode != schema.ContentFull { + return sdk.Deny("Pitot sample policy requires full content mode.") + } + var command string + if err := json.Unmarshal(event.Content.Full, &command); err != nil { + return sdk.Deny("Pitot sample policy could not decode the command.") + } + // Sample tripwire only — not a general shell-security control. + if strings.Contains(command, "PITOT_DENY_ME") { + return sdk.Deny("Pitot sample policy blocked the PITOT_DENY_ME canary.") + } + return sdk.Allow("Pitot sample policy allowed the shell request.") + }) +} +` + +const pythonShellPolicyTemplate = `from pitot.runner import run_controller, allow, deny +from pitot.types import ControlRequested + +def handler(req: ControlRequested): + event = req.data or {} + content = event.get("content") or {} + if content.get("mode") != "full": + return deny("Pitot sample policy requires full content mode.") + command = content.get("full") or "" + # Sample tripwire only — not a general shell-security control. + if "PITOT_DENY_ME" in command: + return deny("Pitot sample policy blocked the PITOT_DENY_ME canary.") + return allow("Pitot sample policy allowed the shell request.") + +if __name__ == "__main__": + run_controller("local-shell-policy", handler) +` + +const tsShellPolicyTemplate = `import { runController, allow, deny, ControlRequested, Event } from '@operatorstack/pitot'; + +runController("local-shell-policy", async (req: ControlRequested) => { + const event = (req.data ?? {}) as Event; + if (!event.content || event.content.mode !== "full") { + return deny("Pitot sample policy requires full content mode."); + } + const command = String(event.content.full ?? ""); + // Sample tripwire only — not a general shell-security control. + if (command.includes("PITOT_DENY_ME")) { + return deny("Pitot sample policy blocked the PITOT_DENY_ME canary."); + } + return allow("Pitot sample policy allowed the shell request."); +}); +` + +const rustShellPolicyTemplate = `use pitot::{run_controller, allow, deny, ControlRequested, Event, Outcome}; + +fn handler(req: ControlRequested) -> Outcome { + let event: Event = match req.data.and_then(|d| serde_json::from_value(d).ok()) { + Some(e) => e, + None => return deny(Some("Pitot sample policy could not decode the event.".to_string())), + }; + let content = match event.content { + Some(c) if c.mode == "full" => c, + _ => return deny(Some("Pitot sample policy requires full content mode.".to_string())), + }; + let command = content.full.and_then(|v| v.as_str().map(str::to_string)).unwrap_or_default(); + // Sample tripwire only — not a general shell-security control. + if command.contains("PITOT_DENY_ME") { + return deny(Some("Pitot sample policy blocked the PITOT_DENY_ME canary.".to_string())); + } + allow(Some("Pitot sample policy allowed the shell request.".to_string())) +} + +fn main() { + run_controller("local-shell-policy", Box::new(handler)); +} ` // runDev launches the runtime and a single agent against a chosen host, waits diff --git a/labs/15-pitot/pitot/cmd/pitot/workbench_build_test.go b/labs/15-pitot/pitot/cmd/pitot/workbench_build_test.go new file mode 100644 index 000000000..f974a2fdb --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/workbench_build_test.go @@ -0,0 +1,152 @@ +package main + +import ( + "bytes" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// labRoot resolves labs/15-pitot from the package working directory +// (cmd/pitot -> module root -> lab root), where the local SDK sources live. +func labRoot(t *testing.T) string { + t.Helper() + p, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatal(err) + } + return p +} + +// initInto scaffolds a shell-policy project for lang into a fresh dir and returns +// the directory. It fails the test on any init error. +func initInto(t *testing.T, lang string) string { + t.Helper() + dir := filepath.Join(t.TempDir(), "proj") + var out, errb bytes.Buffer + if err := runInit([]string{"--language", lang, "--template", "shell-policy", "--dir", dir}, strings.NewReader(""), &out, &errb); err != nil { + t.Fatalf("init %s: %v\n%s", lang, err, errb.String()) + } + return dir +} + +// TestBuildGeneratedGoShellPolicy makes the Go build a first-class Step-5 +// assertion: the generated controller compiles offline against the in-tree SDK +// via a filesystem module replace (no published dependency, no network). +func TestBuildGeneratedGoShellPolicy(t *testing.T) { + if _, err := exec.LookPath("go"); err != nil { + t.Skip("go toolchain not available") + } + // buildGeneratedShellPolicy performs runInit + local replace + offline build + // and fails the test if the generated project does not compile. + buildGeneratedShellPolicy(t) +} + +// TestBuildGeneratedPythonImports verifies the generated Python controller +// imports cleanly against the local SDK (proving the SDK API surface it depends +// on exists), without contacting PyPI. +func TestBuildGeneratedPythonImports(t *testing.T) { + py, err := exec.LookPath("python3") + if err != nil { + t.Skip("python3 not available") + } + sdk := filepath.Join(labRoot(t), "pitot-distribution", "sdk", "python") + if _, err := os.Stat(filepath.Join(sdk, "pitot", "runner.py")); err != nil { + t.Skipf("local python SDK not present at %s", sdk) + } + proj := initInto(t, "python") + + cmd := exec.Command(py, "-c", "import main; assert hasattr(main, 'handler'), 'generated main.py must define handler'") + cmd.Dir = proj + cmd.Env = append(os.Environ(), + "PYTHONPATH="+sdk+string(os.PathListSeparator)+proj, + "PYTHONDONTWRITEBYTECODE=1", + ) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("python import of generated controller failed: %v\n%s", err, out) + } +} + +// TestTypeCheckGeneratedTypeScript type-checks the generated controller against +// the local SDK using a locally installed tsc. It skips (rather than fetching +// from a registry) when tsc or the local SDK is unavailable. +func TestTypeCheckGeneratedTypeScript(t *testing.T) { + node, err := exec.LookPath("node") + if err != nil { + t.Skip("node not available") + } + sdk := filepath.Join(labRoot(t), "pitot-distribution", "sdk", "typescript") + if _, err := os.Stat(filepath.Join(sdk, "package.json")); err != nil { + t.Skipf("local typescript SDK not present at %s", sdk) + } + proj := initInto(t, "typescript") + + // Resolve the genuine TypeScript compiler via node module resolution rather + // than an ambient PATH `tsc` (which may be an unrelated decoy). Skip when the + // typescript package is not locally installed — no registry fetch. + resolve := exec.Command(node, "-e", "process.stdout.write(require.resolve('typescript/bin/tsc'))") + resolve.Dir = proj + tscJS, err := resolve.Output() + if err != nil || len(tscJS) == 0 { + t.Skip("typescript package not resolvable via node; skipping hermetic TS type-check") + } + + // Make '@operatorstack/pitot' resolvable to the local SDK without installing. + scope := filepath.Join(proj, "node_modules", "@operatorstack") + if err := os.MkdirAll(scope, 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(sdk, filepath.Join(scope, "pitot")); err != nil { + t.Fatalf("link local ts SDK: %v", err) + } + cmd := exec.Command(node, string(tscJS), + "--noEmit", "--skipLibCheck", "--esModuleInterop", + "--moduleResolution", "node", "--target", "es2022", "main.ts") + cmd.Dir = proj + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("tsc type-check of generated controller failed: %v\n%s", err, out) + } +} + +// TestCargoCheckGeneratedRust type-checks the generated Rust controller against +// the local SDK via a path dependency. It skips when cargo or the local SDK is +// unavailable. +func TestCargoCheckGeneratedRust(t *testing.T) { + if _, err := exec.LookPath("cargo"); err != nil { + t.Skip("cargo not available") + } + sdk := filepath.Join(labRoot(t), "pitot-distribution", "sdk", "rust") + if _, err := os.Stat(filepath.Join(sdk, "Cargo.toml")); err != nil { + t.Skipf("local rust SDK not present at %s", sdk) + } + proj := initInto(t, "rust") + + // Repoint the crates.io dependency at the local SDK by path. + manifestPath := filepath.Join(proj, "Cargo.toml") + manifest, err := os.ReadFile(manifestPath) + if err != nil { + t.Fatal(err) + } + patched := strings.Replace(string(manifest), + `pitot = "0.1.0"`, + `pitot = { path = `+quoteToml(sdk)+` }`, 1) + if patched == string(manifest) { + t.Fatalf("could not repoint rust SDK dependency in:\n%s", manifest) + } + if err := os.WriteFile(manifestPath, []byte(patched), 0o644); err != nil { + t.Fatal(err) + } + + cmd := exec.Command("cargo", "check", "--quiet") + cmd.Dir = proj + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("cargo check of generated controller failed: %v\n%s", err, out) + } +} + +// quoteToml renders a filesystem path as a TOML basic string. +func quoteToml(s string) string { + return `"` + strings.ReplaceAll(s, `\`, `\\`) + `"` +} diff --git a/labs/15-pitot/pitot/cmd/pitot/workbench_contract_test.go b/labs/15-pitot/pitot/cmd/pitot/workbench_contract_test.go new file mode 100644 index 000000000..7b40018cb --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/workbench_contract_test.go @@ -0,0 +1,165 @@ +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/operatorstack/pitot/config" +) + +// shellPolicyExpect maps each language to the source file init writes and the +// SDK controller-entry token that source must reference for the shell-policy +// template. The PITOT_DENY_ME marker is asserted separately for every language. +var shellPolicyExpect = map[string]struct { + sourceFile string + sdkToken string +}{ + "python": {"main.py", "run_controller"}, + "typescript": {"main.ts", "runController"}, + "go": {"main.go", "sdk.RunController"}, + "rust": {"main.rs", "run_controller"}, +} + +// TestInitShellPolicyContract asserts the shell-policy scaffold is coherent per +// language: expected files exist, the config parses and registers the controller +// under the shell kind, and the source references the SDK controller API plus +// the deny canary. +func TestInitShellPolicyContract(t *testing.T) { + for lang, wantFiles := range initExpectations { + lang, wantFiles := lang, wantFiles + t.Run(lang, func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + if err := runInit([]string{"--language", lang, "--template", "shell-policy", "--dir", dir}, strings.NewReader(""), &stdout, &stderr); err != nil { + t.Fatalf("init shell-policy %s: %v", lang, err) + } + + for _, name := range wantFiles { + if _, statErr := readIf(dir, name); statErr != nil { + t.Errorf("%s: expected generated file %q: %v", lang, name, statErr) + } + } + + // The generated config must parse and register the controller for shell. + loaded, err := config.Load(filepath.Join(dir, ".pitot.yaml")) + if err != nil { + t.Fatalf("%s: generated .pitot.yaml did not parse: %v", lang, err) + } + ctrl, ok := loaded.Config.Controllers["shell"] + if !ok { + t.Fatalf("%s: controller not registered under shell kind: %+v", lang, loaded.Config.Controllers) + } + if ctrl.ID != "local-shell-policy" { + t.Errorf("%s: controller id = %q, want local-shell-policy", lang, ctrl.ID) + } + if len(ctrl.Command) == 0 { + t.Errorf("%s: controller command is empty", lang) + } + + // The source must use the SDK controller API and the deny canary. + want := shellPolicyExpect[lang] + src, err := readIf(dir, want.sourceFile) + if err != nil { + t.Fatalf("%s: read source: %v", lang, err) + } + if !strings.Contains(src, want.sdkToken) { + t.Errorf("%s: source missing SDK controller API %q:\n%s", lang, want.sdkToken, src) + } + if !strings.Contains(src, "PITOT_DENY_ME") { + t.Errorf("%s: source missing PITOT_DENY_ME canary:\n%s", lang, src) + } + if !strings.Contains(src, "local-shell-policy") { + t.Errorf("%s: source missing local-shell-policy id:\n%s", lang, src) + } + }) + } +} + +// TestInitNextStepLaunchesAgentNotController guards the corrected guidance: the +// hint must point users at `pitot dev --host HOST -- AGENT`, never at `--exec` +// with the Controller command. +func TestInitNextStepLaunchesAgentNotController(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + if err := runInit([]string{"--language", "go", "--template", "shell-policy", "--dir", dir}, strings.NewReader(""), &stdout, &stderr); err != nil { + t.Fatalf("init: %v", err) + } + out := stdout.String() + if !strings.Contains(out, "pitot dev --host HOST -- AGENT") { + t.Errorf("next-step guidance does not launch an agent:\n%s", out) + } + if strings.Contains(out, "--exec") { + t.Errorf("next-step guidance still uses --exec (points at the Controller):\n%s", out) + } + if strings.Contains(out, "go run main.go") { + t.Errorf("next-step guidance still passes the Controller command as the agent:\n%s", out) + } +} + +// TestInitBlankControllerUsesApprovalKind confirms non-shell controller +// templates keep the test.approval kind (and do not leak the shell canary). +func TestInitBlankControllerUsesApprovalKind(t *testing.T) { + for _, template := range []string{"blank-controller", "release-approval"} { + template := template + t.Run(template, func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + if err := runInit([]string{"--language", "go", "--template", template, "--dir", dir}, strings.NewReader(""), &stdout, &stderr); err != nil { + t.Fatalf("init %s: %v", template, err) + } + loaded, err := config.Load(filepath.Join(dir, ".pitot.yaml")) + if err != nil { + t.Fatalf("%s: config did not parse: %v", template, err) + } + if _, ok := loaded.Config.Controllers["test.approval"]; !ok { + t.Errorf("%s: expected test.approval controller, got %+v", template, loaded.Config.Controllers) + } + if _, ok := loaded.Config.Controllers["shell"]; ok { + t.Errorf("%s: must not register a shell controller", template) + } + }) + } +} + +// TestInitTemplateRoleConsistency verifies template/role validation. +func TestInitTemplateRoleConsistency(t *testing.T) { + t.Run("mismatch rejected", func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + err := runInit([]string{"--language", "go", "--role", "consumer", "--template", "shell-policy", "--dir", dir}, strings.NewReader(""), &stdout, &stderr) + if err == nil || !strings.Contains(err.Error(), "implies role") { + t.Fatalf("expected role/template mismatch error, got %v", err) + } + }) + t.Run("unsupported template rejected", func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + err := runInit([]string{"--language", "go", "--template", "nonesuch", "--dir", dir}, strings.NewReader(""), &stdout, &stderr) + if err == nil || !strings.Contains(err.Error(), "unsupported template") { + t.Fatalf("expected unsupported template error, got %v", err) + } + }) + t.Run("blank-consumer infers consumer role", func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "proj") + var stdout, stderr bytes.Buffer + if err := runInit([]string{"--language", "go", "--template", "blank-consumer", "--dir", dir}, strings.NewReader(""), &stdout, &stderr); err != nil { + t.Fatalf("init blank-consumer: %v", err) + } + loaded, err := config.Load(filepath.Join(dir, ".pitot.yaml")) + if err != nil { + t.Fatal(err) + } + if len(loaded.Config.Consumers) == 0 { + t.Errorf("blank-consumer did not produce a consumer config: %+v", loaded.Config) + } + }) +} + +// readIf returns a file's contents under dir, or the read error. +func readIf(dir, name string) (string, error) { + data, err := os.ReadFile(filepath.Join(dir, name)) + return string(data), err +} diff --git a/labs/15-pitot/pitot/cmd/pitot/workbench_dev_test.go b/labs/15-pitot/pitot/cmd/pitot/workbench_dev_test.go new file mode 100644 index 000000000..c3ff5a16c --- /dev/null +++ b/labs/15-pitot/pitot/cmd/pitot/workbench_dev_test.go @@ -0,0 +1,232 @@ +package main + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + goruntime "runtime" + "strings" + "testing" +) + +// decisionLine models one entry from `pitot dev`'s decision timeline. +type decisionLine struct { + outcome string // ALLOW or DENY + kind string + actionID string + message string +} + +var decisionRE = regexp.MustCompile(`\[(ALLOW|DENY)\]\s+(\S+)\s+\((act_[^)]+)\)(?:\s+\p{Pd}+\s+(.*))?`) + +// parseDecisions extracts the decision timeline rendered by `pitot dev` (and the +// real-Kimi smoke test) from a captured stdout stream. +func parseDecisions(output string) []decisionLine { + var out []decisionLine + for _, m := range decisionRE.FindAllStringSubmatch(output, -1) { + out = append(out, decisionLine{outcome: m[1], kind: m[2], actionID: m[3], message: strings.TrimSpace(m[4])}) + } + return out +} + +// buildPitotBinary compiles the reference CLI from the in-tree package so tests +// can invoke `pitot hook`/`pitot dev` as a real subprocess. Callers must invoke +// this before any t.Chdir, since it builds from the package's working directory. +func buildPitotBinary(t *testing.T) string { + t.Helper() + binDir := t.TempDir() + name := "pitot" + if goruntime.GOOS == "windows" { + name += ".exe" + } + bin := filepath.Join(binDir, name) + build := exec.Command("go", "build", "-o", bin, ".") + if out, err := build.CombinedOutput(); err != nil { + t.Fatalf("build pitot binary: %v\n%s", err, out) + } + return bin +} + +// devAgentScript writes a POSIX agent stand-in that records the runtime it was +// handed and its own argv, then drives one allow and one deny decision through +// the running runtime via `pitot hook kimi`. It stands in for a real coding +// agent so the dev harness can be exercised deterministically. +func devAgentScript(t *testing.T, seenDir string) string { + t.Helper() + allowPayload := `{"hook_event_name":"PreToolUse","session_id":"dev","tool_name":"Bash","tool_input":{"command":"echo ok"}}` + denyPayload := `{"hook_event_name":"PreToolUse","session_id":"dev","tool_name":"Bash","tool_input":{"command":"PITOT_DENY_ME=1 echo no"}}` + script := fmt.Sprintf(`#!/bin/sh +printf '%%s' "$PITOT_RUNTIME" > %q +printf '%%s' "$#" > %q +printf '%%s' "$*" > %q +printf '%s' | "$PITOT_BIN" hook kimi --runtime "$PITOT_RUNTIME" +printf '%s' | "$PITOT_BIN" hook kimi --runtime "$PITOT_RUNTIME" || true +`, + filepath.Join(seenDir, "runtime.txt"), + filepath.Join(seenDir, "argc.txt"), + filepath.Join(seenDir, "args.txt"), + allowPayload, denyPayload) + path := filepath.Join(seenDir, "agent.sh") + if err := os.WriteFile(path, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + return path +} + +// TestDevRunsAgentBehindShellController is the end-to-end proof of the dev +// harness: it starts the runtime + the compiled shell-policy controller, launches +// a scripted agent, and verifies the decision timeline, PITOT_RUNTIME +// propagation, argv preservation, and runtime-dir teardown. +func TestDevRunsAgentBehindShellController(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("dev agent stand-in is a POSIX shell script") + } + // Build inputs before we chdir into the project directory. + pitotBin := buildPitotBinary(t) + _, configBody := buildGeneratedShellPolicy(t) + + proj := t.TempDir() + if err := os.WriteFile(filepath.Join(proj, ".pitot.yaml"), []byte(configBody), 0o600); err != nil { + t.Fatal(err) + } + seenDir := t.TempDir() + agent := devAgentScript(t, seenDir) + + t.Setenv("PITOT_BIN", pitotBin) + t.Setenv("PITOT_RUNTIME", "") + t.Chdir(proj) + + var stdout lockedBuffer + var stderr lockedBuffer + // `-- sh AGENT a b` exercises the explicit argv form (args after --). + if err := runDev(context.Background(), []string{"--host", "kimi", "--", "sh", agent, "alpha", "beta"}, &stdout, &stderr); err != nil { + t.Fatalf("pitot dev: %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + + // Decision timeline must show exactly one allow and one deny, both shell-kind, + // and the deny must carry the Controller's canary reason. + decisions := parseDecisions(stdout.String()) + var gotAllow, gotDeny bool + for _, d := range decisions { + if d.kind != "shell" { + t.Errorf("decision kind = %q, want shell: %+v", d.kind, d) + } + switch d.outcome { + case "ALLOW": + gotAllow = true + case "DENY": + gotDeny = true + if !strings.Contains(d.message, "PITOT_DENY_ME canary") { + t.Errorf("deny message missing Controller reason: %q", d.message) + } + } + } + if !gotAllow || !gotDeny { + t.Fatalf("decision timeline missing allow+deny (allow=%v deny=%v):\n%s", gotAllow, gotDeny, stdout.String()) + } + + // The agent must have received a real PITOT_RUNTIME pointing at the per-run dir. + runtimeSeen := readFileString(t, filepath.Join(seenDir, "runtime.txt")) + if !strings.Contains(runtimeSeen, "pitot-dev-") { + t.Errorf("agent PITOT_RUNTIME = %q, want a per-invocation dev runtime path", runtimeSeen) + } + // argv after `--` is preserved verbatim: two positional args, joined "alpha beta". + if argc := readFileString(t, filepath.Join(seenDir, "argc.txt")); argc != "2" { + t.Errorf("agent argc = %q, want 2", argc) + } + if args := readFileString(t, filepath.Join(seenDir, "args.txt")); args != "alpha beta" { + t.Errorf("agent args = %q, want \"alpha beta\"", args) + } + // The per-invocation runtime directory must be removed once dev exits. + if _, err := os.Stat(filepath.Dir(runtimeSeen)); !os.IsNotExist(err) { + t.Errorf("runtime dir %q must be removed on exit, stat err=%v", filepath.Dir(runtimeSeen), err) + } +} + +// TestDevRuntimePathsAreUniquePerRun confirms concurrent-safe isolation: two dev +// runs of the same project hand the agent two different runtime descriptors. +func TestDevRuntimePathsAreUniquePerRun(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("dev agent stand-in is a POSIX shell script") + } + pitotBin := buildPitotBinary(t) + _, configBody := buildGeneratedShellPolicy(t) + + proj := t.TempDir() + if err := os.WriteFile(filepath.Join(proj, ".pitot.yaml"), []byte(configBody), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("PITOT_BIN", pitotBin) + t.Setenv("PITOT_RUNTIME", "") + t.Chdir(proj) + + run := func() string { + seenDir := t.TempDir() + agent := devAgentScript(t, seenDir) + var stdout lockedBuffer + var stderr lockedBuffer + if err := runDev(context.Background(), []string{"--host", "kimi", "--", "sh", agent}, &stdout, &stderr); err != nil { + t.Fatalf("pitot dev: %v\n%s", err, stderr.String()) + } + return readFileString(t, filepath.Join(seenDir, "runtime.txt")) + } + first, second := run(), run() + if first == "" || first == second { + t.Errorf("runtime paths not unique per run: %q vs %q", first, second) + } +} + +// TestDevExecSplitsWhereArgvDoesNot pins the distinct semantics of the two agent +// forms: --exec field-splits its string, while `-- CMD ARGS` preserves argv. +func TestDevExecSplitsWhereArgvDoesNot(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("dev agent stand-in is a POSIX shell script") + } + pitotBin := buildPitotBinary(t) + _, configBody := buildGeneratedShellPolicy(t) + + proj := t.TempDir() + if err := os.WriteFile(filepath.Join(proj, ".pitot.yaml"), []byte(configBody), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("PITOT_BIN", pitotBin) + t.Setenv("PITOT_RUNTIME", "") + t.Chdir(proj) + + argc := func(devArgs []string) string { + seenDir := t.TempDir() + agent := devAgentScript(t, seenDir) + // Substitute the agent path into the caller's arg template. + filled := make([]string, len(devArgs)) + for i, a := range devArgs { + filled[i] = strings.ReplaceAll(a, "{AGENT}", agent) + } + var stdout lockedBuffer + var stderr lockedBuffer + if err := runDev(context.Background(), filled, &stdout, &stderr); err != nil { + t.Fatalf("pitot dev %v: %v\n%s", filled, err, stderr.String()) + } + return readFileString(t, filepath.Join(seenDir, "argc.txt")) + } + + // --exec "sh AGENT a b" splits on whitespace -> sh sees 2 positional args. + if got := argc([]string{"--host", "kimi", "--exec", "sh {AGENT} a b"}); got != "2" { + t.Errorf("--exec field-split argc = %q, want 2", got) + } + // -- sh AGENT "a b" preserves argv -> sh sees 1 positional arg. + if got := argc([]string{"--host", "kimi", "--", "sh", "{AGENT}", "a b"}); got != "1" { + t.Errorf("-- argv-preserving argc = %q, want 1", got) + } +} + +func readFileString(t *testing.T, path string) string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + return string(data) +} diff --git a/labs/15-pitot/pitot/conformance/fixtures/positive.jsonl b/labs/15-pitot/pitot/conformance/fixtures/positive.jsonl index 562d9e57b..33a93b106 100644 --- a/labs/15-pitot/pitot/conformance/fixtures/positive.jsonl +++ b/labs/15-pitot/pitot/conformance/fixtures/positive.jsonl @@ -7,3 +7,5 @@ {"name":"qwen-pre-tool","host":"qwen","mode":"sha256","input":{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git status --short"}},"expect_kind":"shell"} {"name":"qwen-native-shell-tool","host":"qwen","mode":"sha256","input":{"hook_event_name":"PreToolUse","tool_name":"run_shell_command","tool_input":{"command":"git status --short"}},"expect_kind":"shell"} {"name":"pi-tool-call","host":"pi","mode":"sha256","input":{"hook_event_name":"tool_call","tool_name":"bash","tool_input":{"command":"git status --short"}},"expect_kind":"shell"} +{"name":"kimi-pre-tool-allow","host":"kimi","mode":"full","input":{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"printf allowed > /tmp/pitot-allowed-canary"}},"expect_kind":"shell"} +{"name":"kimi-pre-tool-deny-canary","host":"kimi","mode":"full","input":{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"PITOT_DENY_ME=1 sh -c 'printf blocked > /tmp/pitot-denied-canary'"}},"expect_kind":"shell"} diff --git a/labs/15-pitot/public-readme-preview/README.md b/labs/15-pitot/public-readme-preview/README.md index d175b0e0f..3c615eb17 100644 --- a/labs/15-pitot/public-readme-preview/README.md +++ b/labs/15-pitot/public-readme-preview/README.md @@ -3,7 +3,7 @@

- The open sensor and control transport for coding-agent tooling. + Keep your coding agent. Add the behavior it is missing.

@@ -22,13 +22,62 @@ One language-neutral boundary for the coding agents your team already uses.

-Pitot lets you build above coding agents without rebuilding every host -integration or forking an agent runtime. It converts host-specific activity -into a stable local event stream and carries correlated responses from your -controller when a host is waiting synchronously. +Your coding agent runs shell commands, edits files, and calls tools. Pitot lets +you put your own code in the loop at that boundary — to allow, deny, or record +each action — without forking the agent or rewriting a host integration for +every tool. Pitot reports what happened. Your code decides what it means. +## See it work with Kimi + +The fastest way to understand Pitot is to watch one real command get allowed and +another get denied. This walkthrough is exactly what Pitot's automated test suite +exercises on every commit, so the behavior below is verified, not aspirational. + +**1. Scaffold a sample shell policy.** This writes a runnable Controller that +allows shell commands by default and denies any command containing the canary +string `PITOT_DENY_ME`: + +```bash +pitot init --template shell-policy --language go --dir ./kimi-policy +cd ./kimi-policy +``` + +**2. Check your Kimi host wiring** (Pitot does not edit your Kimi config for you): + +```bash +pitot doctor --host kimi +``` + +If the `PreToolUse` hook is missing, add it to `~/.kimi-code/config.toml`: + +```toml +[[hooks]] +event = "PreToolUse" +matcher = "Bash" +command = "pitot hook kimi" +``` + +**3. Run Kimi behind the Controller.** `pitot dev` starts the runtime, launches +the agent you name after `--`, and prints each decision: + +```bash +pitot dev --host kimi -- kimi -p "Run: echo hello" +``` + +An ordinary command is allowed and runs. Now ask for the canary: + +```bash +pitot dev --host kimi -- kimi -p "Run: PITOT_DENY_ME=1 echo nope" +``` + +The Controller denies it. The denied command never executes, and the denial +reason — `Pitot sample policy blocked the PITOT_DENY_ME canary.` — is returned to +Kimi as the blocked tool result. The `shell-policy` sample is a demonstration +tripwire, not a general shell-security control; the point is that *your* code +made the decision. + ## Why Pitot? Building above coding agents usually forces one of two expensive choices: @@ -57,20 +106,6 @@ A passive Consumer cannot reach the response channel. A Controller is statically registered for one request kind and returns at most one response for the pending action. -## Use-case gallery - -### Operational patterns (grid) - -If you want to see concrete integration ideas, start with the **Use-Cases Grid**: - -- [04 Use-Cases Editorial Grid](./brand-exploration/design-demos/04-use-cases-editorial-grid.html) - -This gallery shows practical ways teams can compose Consumers and Controllers -without forcing each workflow into the host or into a single monolithic runtime. -It includes both engineering patterns (action auditing, approvals, audit hooks) and -non-coding workflows (email triage, file movement, and local automation), -so you can quickly evaluate where Pitot helps before building. - ## Two small programs

@@ -166,62 +201,61 @@ pitot doctor ## Quickstart -From a clean repository to one real allow/deny decision in two commands. - -**1. Scaffold a project.** `pitot init` detects the language from the files -already in the directory, or prompts you to choose when it cannot. It writes a -runnable project — source, package manifest, and `.pitot.yaml` — and never -overwrites existing files unless you pass `--force`: +**1. Scaffold a Controller.** `pitot init` writes a runnable project — source, a +package manifest, and `.pitot.yaml` — and never overwrites existing files unless +you pass `--force`. Pick a starting template with `--template`: ```bash -pitot init +pitot init --template shell-policy --language go --dir ./kimi-policy ``` ``` -Detected python project in . -Initialized python controller in . -Files written: .pitot.yaml, main.py, pyproject.toml, requirements.txt -Next: cd . && pitot dev --host claude --exec "python3 main.py" +Initialized go controller (shell-policy) in ./kimi-policy +Files written: .pitot.yaml, go.mod, main.go +Next: + 1. cd ./kimi-policy + 2. Configure a supported host hook (see: pitot doctor --host HOST). + 3. Run: pitot dev --host HOST -- AGENT [ARGS...] + example: pitot dev --host kimi -- kimi -p "" ``` -You can skip detection and prompts with flags — handy for CI: +Available templates are `shell-policy` (allow/deny shell commands), +`release-approval` and `blank-controller` (request/response controllers), and +`blank-consumer` (a passive event reader). Without `--template`, `pitot init` +detects the language from the current directory or prompts you to choose. The +four first-class languages (`python`, `typescript`, `go`, `rust`) each generate a +complete project that builds after installing dependencies. -```bash -pitot init --language python --role controller --dir ./approval -``` - -The four first-class languages (`python`, `typescript`, `go`, `rust`) each -generate a complete project: `python3 main.py`, `npx tsx main.ts`, -`go run main.go`, and `cargo run` all work after installing dependencies. - -**2. Run it against an agent.** `pitot dev` starts the runtime on a private -loopback endpoint, waits until it is ready, launches your Controller, and prints -each decision as the agent makes it. `--exec` takes the full command line (or use -`-- CMD ARGS`): +**2. Run your agent behind it.** `pitot dev` starts the runtime and the +Controllers declared in `.pitot.yaml`, waits until the runtime is ready, then +launches the agent you name after `--` with `PITOT_RUNTIME` set so its host hook +finds the runtime. It prints each decision as the agent makes it: ```bash -pitot dev --host claude --exec "python3 main.py" +pitot dev --host kimi -- kimi -p "Run: PITOT_DENY_ME=1 echo nope" ``` ``` -Starting Pitot dev environment for host claude... -Runtime ready. Starting agent: python3 main.py +Starting Pitot dev environment for host kimi... +Runtime ready. Starting agent: kimi -p Run: PITOT_DENY_ME=1 echo nope Decisions: - [ALLOW] release.approval (act_7f2) — v1.4.0 is approved for publication. - [DENY] shell.exec (act_1a9) — destructive command blocked + [DENY] shell (act_1a9) — Pitot sample policy blocked the PITOT_DENY_ME canary. Agent finished. Runtime stopped. ``` `--host` must name a supported agent (`claude`, `codex`, `copilot`, `cursor`, -`gemini`, `kimi`, `opencode`, `pi`, `qwen`). The runtime descriptor lives in a -per-invocation temporary path and is removed on exit, so concurrent `pitot dev` -sessions never collide. +`gemini`, `kimi`, `opencode`, `pi`, `qwen`), and that agent's host hook must +already be wired to `pitot hook HOST` (see **Connect your agent** and +`pitot doctor --host HOST`). The runtime descriptor lives in a per-invocation +temporary path and is removed on exit, so concurrent `pitot dev` sessions never +collide. **3. Swap the agent.** The same project — the same Controller and `.pitot.yaml` — -works with any other supported host. Change only `--host`: +works with any other supported host whose hook is wired. Change only `--host` and +the agent command: ```bash -pitot dev --host cursor --exec "python3 main.py" +pitot dev --host cursor -- cursor-agent -p "Run: PITOT_DENY_ME=1 echo nope" ``` The boundary is language- and agent-neutral: one Controller, every agent. @@ -251,10 +285,37 @@ $env:PITOT_RUNTIME = Join-Path $env:LOCALAPPDATA "Pitot\project.json" pitot run --config .pitot.yaml --runtime $env:PITOT_RUNTIME ``` +## Supported hosts + +Every host below normalizes its native blocking boundary to a `shell` action and +passes Pitot's language-neutral decoder conformance suite. The E2E column marks +adapters exercised by the cross-platform agent supervisor (the badge at the top) +on Ubuntu, macOS, and Windows. Kimi additionally has an in-repo, no-model test +that asserts the full allow **and** deny control path end to end. + +| Host | Blocking boundary | Hook wiring | Verified in this repo | +|---|---|---|---| +| Kimi Code | `PreToolUse` / Bash | native `config.toml` | decoder + E2E + allow/deny control test | +| Claude | `PreToolUse` | native settings hook | decoder + E2E | +| Cursor | `beforeShellExecution` | bridge (`integrations/cursor`) | decoder + E2E | +| Codex | `PreToolUse` | bridge (`integrations/codex`) | decoder + E2E | +| GitHub Copilot CLI | `PreToolUse` | bridge (`integrations/copilot`) | decoder + E2E | +| Gemini | `BeforeTool` | bridge (`integrations/gemini`) | decoder + E2E | +| OpenCode | `PreToolUse` | bridge (`integrations/opencode`) | decoder + E2E | +| Pi | `tool_call` | extension (`integrations/pi`) | decoder + E2E | +| Qwen Code | `PreToolUse` | bridge (`integrations/qwen`) | decoder + E2E | + +"Decoder" means Pitot correctly normalizes that host's payload into the stable +event envelope. It does not claim Pitot judges whether any command is safe — that +decision belongs to your Controller. + ## Connect your agent -The per-host hooks below wire each agent's native blocking boundary to Pitot for -the manual runtime flow. `pitot dev` configures the selected `--host` for you. +The per-host hooks below wire each agent's native blocking boundary to Pitot. +This wiring is a one-time edit to each host's own configuration; Pitot does not +edit your host config for you. Run `pitot doctor --host HOST` to check whether a +host's hook is correctly configured. Once wired, both `pitot dev` and the manual +runtime flow use the same hook. ### Kimi Code