v0.3.0 — PAL

Summary of what has merged since v0.2.0 (270 merge commits), grouped by area. The
detailed per-feature entries that follow are the record; ADR numbers point at the
decision. Nothing here is tagged yet.

Upgrade notes (breaking or required)

• Helm chart 0.3.0. Ingress is split into an API host and an app host, and the
  chart refuses to render with ingress.enabled unless controlPlane.trustedProxies
  is set (and networkPolicy.ingressFrom when network policy is on). In production
  the control plane refuses to start without PAL_CALLBACK_SIGNING_SECRET, and with
  open or invite sign-up it needs Redis. OAuth sign-up now follows PAL_SIGNUP_MODE.
  Migrations are forward-only; take a pg_dump first. See docs operate/upgrading.
• Rename to PAL (ADR 0032). PAL_* env vars, pal CLI, pal.v1 protos, module
  github.com/dshakes/pal, DB and MinIO names. Legacy names are accepted during a
  transition. The expansion is "Personalized Agentic Layer" (ADR 0032 amendment).

Agent lifecycle and governance

• Promotion: immutable frozen versions, sandbox/staging/prod environments, server-side
  gates, human approval for prod, canary with auto-rollback, append-only audit,
  pal sandbox run (ADR 0054).
• Per-agent identity and default-deny least-privilege policy enforced at every
  dispatch, hash-chained audit (ADR 0055).
• Agentic evals: authoring, LLM-as-judge, hermetic tool mocks (ADR 0052); pal ci
  shift-left gates (ADR 0053).
• Post-deployment observability: trace tree, SLOs and error budgets, sampled
  production evals, anomaly incidents (ADR 0056).
• Debug and support agents that propose fixes a person must approve (ADR 0059).

Protocols and integrations

• A2A v0.3: spec card, JSON-RPC tasks, outbound a2a_call (ADR 0057).
• MCP 2025-06-18 client and PAL MCP server, OAuth 2.1, policy-gated tools (ADR 0058).
• Async completion callbacks and batch invocation (ADR 0033); signed inbound webhook
  triggers (ADR 0028); Stripe billing, off by default (ADR 0049).

Console and onboarding

• Dashboard v2 and v3: PAL / PAL Runtime spaces, Ask PAL, Overview, lifecycle tabs,
  Promotions, Sandboxes, Reliability, Incidents, Debug.
• Onboarding agent, a multi-turn conversation for both products (ADR 0051); PAL
  first-run and pal setup (ADR 0048); device tenancy and bridge enrollment (ADR 0047).
• Sign-up modes, email verification, password reset and data export (ADR 0027).

Security and operations

• mTLS on every internal gRPC hop (ADR 0043); API-key scope and RLS secure defaults in
  production (ADR 0042); supply-chain provenance, SBOMs and digest-pinned images (ADR 0044).
• Egress DLP (ADR 0030) and host-asserted software measurement (ADR 0031).
• Helm chart that installs out of the box, scheduler and surface-gateway, HPA/PDB,
  network policy, backup CronJob and restore drill; SLO burn-rate alerts (ADR 0041);
  multi-region active-passive enablers (ADR 0046).
• Fixes: XFF-safe rate limits, connector SSRF guard, mail header injection, real
  pairing codes, patched dependencies.
• Dependency security updates: OpenTelemetry 0.33 / tracing-opentelemetry 0.34 in the
  Rust services (baggage DoS advisory); jsonwebtoken 10.4 with the aws-lc-rs backend in
  the gateway's tests (auth-bypass advisory; the gateway itself verifies API keys via
  introspection, not JWTs); rcgen 0.14 for VM and egress certificates (cert properties
  unchanged, covered by tests); grpc-go 1.83.2; zod 4 in examples; Next.js 16 in the
  dashboard.
• Known, not fixed in this release: minimatch (ReDoS) reaches the dashboard only through
  ESLint, a dev-time tool that never ships or sees untrusted input. Planned upgrade.
  TypeScript 7 waits on typescript-eslint, tsup and Astro check support.

Examples, docs and site

• Runnable examples (support desk, PR triage, incident responder, daily digest,
  Telegram assistant, voice receptionist, data analyst) with a fixture-tier e2e lane.
• Docs site rebuilt for PAL and PAL Runtime, with interactive diagrams, a landing page
  and, in this release, new pages for promotion, agent policy, the console and upgrading.

Detailed entries

Changed

• Product renamed to PAL (the Personal Agentic Layer; ADR 0032).

Added

• Host-asserted software measurement (ADR 0031, ACE A5) — at every Firecracker
  cold spawn the runtime-manager computes a SHA-256 over the five canonical inputs
  it directly controls: kernel image, rootfs image, harness binary (32 zero bytes
  when baked into rootfs), the kernel cmdline, and the spawn contract — minus
  secrets (declared-secret refs, exec token, injection specs, values of
  credential-named env keys) and minus per-VM identity (vm/run id, IPs, per-VM cert
  paths/CA, agent instance id + token, trace parent, the HashMap-ordered
  pal.env.* cmdline copy). Two VMs from the same images + spec measure
  identically. A restore / resume / fork inherits its source's spawn
  measurement (recorded in the snapshot metadata) instead of re-measuring, and
  re-checks it against the current references; an unmeasured (legacy) snapshot is
  refused under an enforcing gate. The 64-character hex measurement is:
  • Embedded in the per-VM X.509 cert as a custom extension (OID 1.3.6.1.4.1.99999.3,
    a placeholder — a registered IANA PEN is needed).
  • Recorded by the scheduler (persisted in sched_vms.software_measurement) and
    written once to runtime_vms.software_measurement (migration
    0029_software_measurement) by the reconciler — for forked/resumed VMs too.
  • Included in signed run receipts as softwareMeasurement (additive; omitempty
    keeps existing receipts byte-identical and their signatures valid).
  • Returned in VmHandle.software_measurement (proto field 5).
    Gate control: PAL_MEASUREMENT_GATE (off | warn [default] | enforce);
    references: PAL_REFERENCE_MEASUREMENTS (comma-separated 64-hex). An unknown
    gate value or malformed reference refuses startup, as does enforce with no
    references in prod. Image hashes are streamed and cached per file identity.
    HOST-ASSERTED only — proves what the host recorded it launched, NOT what ran
    inside the VM; images are hashed by path and opened later by Firecracker (TOCTOU,
    documented). Hardware attestation (ADR 0023) remains UNVERIFIED. Secret-named env
    values (whole _ segments: TOKEN, SECRET, PASSWORD, KEY, AUTH, PAT, DSN, …) are
    blanked and URL userinfo (scheme://user:pass@) is scrubbed from every env value
    before hashing; the snapshot store is inside the trust boundary (unsigned
    metadata; signing is a follow-up). Live check: make lima-measurement-e2e.
• Breaking (Firecracker guests): caller env is no longer sent on the kernel
  cmdline. spawn.json is the only caller-env channel; the cmdline carries just
  the manager-owned PAL_ENV / PAL_MANAGER_ADDR / PAL_WORKLOAD_UID
  (values whitespace-sanitized). The guest echoes its cmdline to the console the
  manager logs, so caller values there (an instance token, a DATABASE_URL
  password) leaked into logs. Harness builds older than spawn.json support are no
  longer supported for caller env; pal.env.RUST_LOG from a caller no longer
  reaches the harness.
• Egress DLP (ADR 0030) — per-tenant data-loss-prevention policy applied by the egress broker to TLS-terminated and plain-HTTP egress. Detectors: Luhn-validated card numbers, secret-key prefixes (AWS/GitHub/Slack/PEM), email (threshold-based), phone (threshold-based), custom regex. Modes: off | audit | block. Fails closed: unknown detector/mode or bad regex fails the spawn (regex also 400 at write), a block rule on a node without a broker refuses the spawn, ambiguous body framing (CL+TE, conflicting CL, bad chunks) is a 400, and in block mode a body that cannot be fully inspected up to the cap is a 403. Chunked bodies are de-chunked before scanning; Expect: 100-continue is answered by the broker. The policy survives pause/resume/fork; DLP VMs skip the warm pool. Not inspected (documented): opaque CONNECT tunnels, bytes past max_inspect_bytes (64 KiB, ≤1 MiB), keep-alive follow-on requests. Audit action dlp carries rule id, detector, host, byte offset/count, decision — never the matched value (invariant #10). Policy stored in dlp_policies (RLS-enforced, migration 0028). REST CRUD at GET/PUT /v1/egress/dlp-policy + POST/DELETE /v1/egress/dlp-policy/rules. Compiled into the broker at VM spawn via AgentSpec.dlp_policy (proto field 19). e2e: broker-e2e.sh DLP block cases (card number, custom_regex).
• Prod enforces API-key scopes — values-prod.yaml (and staging) set PAL_AUTHZ_ENFORCE=1; a first-class controlPlane.env.PAL_AUTHZ_ENFORCE chart value, a boot log line stating the mode, a ControlPlane403RateElevated alert, a chart render test (infra/helm/pal/tests/authz-enforce.sh, run in CI), and the soak/rollback/re-scoping steps in docs/runbooks/ga-staged-rollout.md. JWT sessions and unscoped legacy keys are unaffected.
• Docs site: Cmd/Ctrl-K search over a build-time index (no external service), per-page titles/descriptions/Open Graph tags,
  sitemap.xml + robots.txt, a top bar (landing link, search, GitHub, theme) and an "Edit this page on GitHub" link.
• CI: fixture-tier e2e lane — .github/workflows/sdlc-e2e.yml (jobs examples, browser, deploy-cli) runs the webhook-example suites, Playwright and the pal deploy e2e on every PR and master push with no provider key; skip = fail. Docs truth pass: e2e/README.md test tiers, docs/GA-READINESS.md, docs/LAUNCH-CHECKLIST.md re-verified against code.
• Kiosk demo mode (ADR 0029, #342) — make demo / pal demo start the stack with Docker only and no API
  key: each visitor gets an ephemeral, auto-deleted tenant (30 minutes) with sample agents, answers come from the
  llm-replay fixture server, RLS and scope enforcement are on. pal demo --down / make demo-down delete
  all demo data. Docs: /demo.
• *...

v0.2.0 — Lantern

The sandbox runtime release: credentials kept out of the VM, idle VMs that
suspend themselves, copy-on-write forks, and the full lifecycle tested on real
KVM.

Added

• Host egress broker (opt-in, LANTERN_EGRESS_MODE=broker, Firecracker;
  ADR 0026, #299) — broker-mode VMs get no network device; egress and the
  harness control channel run over vsock to a per-VM broker on the host, which
  enforces the allowlist and injects credentials from lantern.secret/... refs
  (egress[].inject, exact hostnames only) so the secret never enters the VM.
  HTTPS is terminated only for injected hosts, with a per-VM CA scoped to them.
  Validated live on KVM (infra/firecracker/broker-e2e.sh).
• Idle auto-suspend + transparent auto-resume (#296) — idleTimeoutSecs on
  schedule / lantern run --idle-timeout; idle VMs are snapshotted and stopped
  through the crash-safe pause path (quota slot freed) and resumed on the next
  exec or file request (REST, CLI or SDK).
• Shared-memory copy-on-write forks (#295) — forks hard-link one read-only
  snapshot memory file (MAP_PRIVATE); fork cap configurable
  (LANTERN_MAX_FORK_COUNT, default 100, ceiling 1000).
• Sandbox file API (#294) — list/read/write files under /workspace via
  REST, TS/Python/Go SDKs and lantern vm ls|cat|put.
• Agentic onboarding (#297) — conversational /onboarding flow (LLM-designed
  agent spec, real provider check, streamed first run, sandbox try-out,
  resumable) and interactive lantern onboard; lantern doctor checks the
  sandbox runtime.
• Real-KVM lifecycle e2e (#298) — snapshot → restore → fork ×3 (jailed, with
  an inode check proving shared memory) in Lima and GitHub's KVM runners, with
  per-step timings in docs/benchmarks/runtime-latency.md.
• New brand, README, landing site, docs site and diagrams — 28 diagrams in
  one visual system, animated landing hero and lifecycle walkthrough.
• GA hardening stack (#276–#292): pause/resume/fork, per-call runtime-manager
  authz and service token, re-identification on restore, per-fork identity,
  manager-owned harness config, tenant-wide budget cap, tool-spend metering,
  body-hashed LLM idempotency keys.

Fixed

• Harness reaper raced tokio's pidfd wait (ECHILD from shell_exec after
  restore); stale sockets broke same-vm_id restore; first exec after restore
  waited out a dead address (37 s → ~4 s); unjailed restores shared one rootfs.
• Manager secret relay sent the node's env tenant/vm_id instead of the VM's
  (empty scope on multi-tenant nodes) — now the VM's, empty fails closed.
• Onboarding: generate-spec returned the raw completion envelope; CLI sent
  api_key instead of apiKey; saved keys were never tested.

Security

• wasmtime moved to the 36.x LTS line (20 guest-can-panic-host advisories,
  #293); wasmtime's unused cache feature dropped to remove zstd
  (RUSTSEC-2026-0395/0396/0400); h2/quinn-proto/rustls/webpki/zod bumps.

Earlier unreleased work

• Interactive TTY exec — lantern vm exec <vm> -it -- <cmd> allocates a real
  PTY in the guest (harness openpty + controlling terminal + winsize) and
  streams it bidirectionally; the CLI enters raw mode (no-op on non-terminal
  stdin). Docker backend sets Tty: true. Non-tty exec is byte-identical.
  ExecRequest gains tty/term_rows/term_cols/term.
• Firecracker guest exec — the in-guest harness now serves its first RPC
  (RuntimeHarness.Exec); the manager records each VM's guest address from the
  heartbeat peer and forwards lantern vm exec into firecracker-class VMs over
  that channel (services/harness/src/exec.rs, runtime-manager service.rs).
• Firecracker jailer wrapping — opt-in via FIRECRACKER_JAILER_PATH
  (chroot + drop to non-root FIRECRACKER_JAILER_UID/GID, default 123:100;
  FIRECRACKER_CHROOT_BASE default /srv/jailer). Argv construction unit-tested;
  jailed exec is Linux-runtime-only; jailed snapshot-restore fails closed.
• K8s isolation validation harness (infra/k8s/, make k8s-validate) — kind
  • Calico cluster asserting default-deny egress, seccomp/cap-drop, and admission
    rejection of privileged pods. Ran green (18/18).
• End-to-end runtime suite (e2e/runtime/, make test-e2e) — drives the live
  control-plane REST surface through the full VM lifecycle; skips when the stack
  is down.
• Lima Firecracker dev path (infra/lima/) — run real microVMs on Apple
  Silicon (M3+/macOS 15+) via nested virt; verified ~1.6 s boot on an M4 Max.
• Exec and Stats gRPC handlers for the Docker backend in runtime-manager
  (services/runtime-manager/src/service.rs). Both were previously
  UNIMPLEMENTED stubs; the Docker path is now backed.
• Make targets run-runtime-manager, run-scheduler, run-api-runtime for
  running the full W12 runtime stack locally without Docker Compose.
• Docker Compose runtime profile so make dev can optionally include the
  runtime-manager and scheduler containers.
• Runtime handler tests in the control-plane covering the /v1/runtime/*
  endpoint surface.
• examples/headless-agents/ demo images baked with lantern-harness as PID 1
  so demos work without a separately-deployed harness image.
• services/runtime-manager/README.md and packages/sdk-python/README.md (new).

runtime-manager

• Kata backend (src/backends/kata.rs): RUNTIME_BACKEND=kata activates a
  thin Docker-delegating backend that sets HostConfig.runtime to
  KATA_RUNTIME_NAME (default kata-runtime). Exec and Stats work through the
  inner Docker client unchanged.
• Wasm backend (src/backends/wasm.rs): RUNTIME_BACKEND=wasm runs
  WebAssembly modules in-process via Wasmtime (WASIp1). Memory limits via
  StoreLimitsBuilder; CPU limits via epoch interruption with configurable
  limits.timeout.
• Exec + Stats for docker and kata: both gRPC methods are now backed for the
  Docker and Kata backends.
• SnapshotStore (src/snapshot_store.rs): filesystem-backed snapshot
  persistence under SNAPSHOT_DIR (default /var/lib/lantern/snapshots) with
  ADR 0007 Tier-2 retention (keep last 3 per (agent_version_id, vm_id), delete
  after 7 days). SHA-256 computed over sorted artifact bytes.
• RelaySecretResolver (src/secret_resolver.rs): production secret resolver
  calling POST /v1/runtime/secrets/resolve on the control-plane (ADR 0008).
  Activated when both LANTERN_CONTROL_PLANE_URL and LANTERN_RUNTIME_SECRET_TOKEN
  are set. Fail-closed: unset env → EnvSecretResolver (dev/CI default).
• RuntimeManager.Snapshot RPC added to packages/proto/lantern/v1/runtime.proto
  and hand-stub Go bindings in gen/go/.

harness

• Egress per-rule rate limiting (src/egress.rs): token bucket throttles
  byte throughput per CONNECT tunnel when EgressRule.rate_bps is non-zero.
  Burst = 1 second of traffic; rate_bps == 0 is unlimited.
• Plain-HTTP method filtering (src/egress.rs): EgressRule.http_methods
  is now checked on plain-HTTP (non-CONNECT) requests. CONNECT tunnels are not
  filtered (opaque TLS — no MITM).
• Cgroup v2 stats (src/heartbeat.rs): sample_usage_loop reads
  memory.current and cpu.stat usage_usec from the cgroup v2 unified hierarchy
  first; falls back to /proc/self/status VmRSS on macOS / cgroup v1 / no mount.

runtime-scheduler

• Postgres write-through persistence (internal/store/): when DATABASE_URL
  is set, every cluster-state mutation is written to Postgres (sched_nodes,
  sched_vms, sched_snapshots tables) and state is restored from DB on restart.
  DB write failures degrade to in-memory-only rather than failing placement.
• Snapshot forwarding to manager: Snapshot RPC now calls the winning node's
  manager and persists the returned metadata in sched_snapshots.

control-plane

• POST /v1/runtime/secrets/resolve (internal/handlers/runtime_secrets.go):
  service-to-service endpoint for the runtime-manager's RelaySecretResolver.
  Authenticates via X-Lantern-Runtime-Token / LANTERN_RUNTIME_SECRET_TOKEN;
  returns 403 when the token is unset (fail-closed). See ADR 0008.
• docs/adr/0008-runtime-secret-relay.md: new ADR documenting the relay
  endpoint design and security properties.

CI

• sdlc-qa.yml: Go tests now run against a pgvector/pgvector:pg16 Postgres
  service with DATABASE_URL set, so scheduler and control-plane DB-dependent
  tests run in CI.
• make test-db: new Make target that starts dev Postgres if needed and runs
  all Go tests with DATABASE_URL wired.

---

runtime-manager (this batch)

• K8s exec (src/backends/k8s.rs): exec_command uses the kube ws feature
  (Api::exec / AttachedProcess) to attach to the agent-runner container and
  collect stdout, stderr, and exit code. Previously UNIMPLEMENTED.
• K8s stats (src/backends/k8s.rs): stats_sample queries
  GET /apis/metrics.k8s.io/v1beta1/namespaces/{ns}/pods/{name}. Surfaces a clear
  "metrics-server not installed" error when the API group is absent rather than an
  opaque transport failure.
• Firecracker stats via heartbeat cache (service.rs): the manager now consumes
  RuntimeHarness::Heartbeat bidirectional streams from in-VM harnesses and caches
  ResourceUsage per vm_id. The Stats gRPC method serves the cached value for
  Firecracker VMs; entries older than 30 s yield Status::UNAVAILABLE with an
  explicit staleness message.
• SnapshotStore S3/MinIO tier (src/snapshot_store.rs): put uploads all
  artifacts + meta.json to S3 after writing locally; get falls back to S3 when
  the local copy is missing. Activated by S3_ENDPOINT + S3_BUCKET +
  AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. S3 upload failures are non-fatal
  (warn-logged; local copy is always written first). Verified live against dev MinIO.

control-plane (this batch)

• Secret relay per-VM binding (internal/handlers/runtime_secrets.go):...