v0.3.0 — PAL

Summary of what has merged since v0.2.0 (270 merge commits), grouped by area. The
detailed per-feature entries that follow are the record; ADR numbers point at the
decision. Nothing here is tagged yet.

Upgrade notes (breaking or required)

• Helm chart 0.3.0. Ingress is split into an API host and an app host, and the
  chart refuses to render with ingress.enabled unless controlPlane.trustedProxies
  is set (and networkPolicy.ingressFrom when network policy is on). In production
  the control plane refuses to start without PAL_CALLBACK_SIGNING_SECRET, and with
  open or invite sign-up it needs Redis. OAuth sign-up now follows PAL_SIGNUP_MODE.
  Migrations are forward-only; take a pg_dump first. See docs operate/upgrading.
• Rename to PAL (ADR 0032). PAL_* env vars, pal CLI, pal.v1 protos, module
  github.com/dshakes/pal, DB and MinIO names. Legacy names are accepted during a
  transition. The expansion is "Personalized Agentic Layer" (ADR 0032 amendment).

Agent lifecycle and governance

• Promotion: immutable frozen versions, sandbox/staging/prod environments, server-side
  gates, human approval for prod, canary with auto-rollback, append-only audit,
  pal sandbox run (ADR 0054).
• Per-agent identity and default-deny least-privilege policy enforced at every
  dispatch, hash-chained audit (ADR 0055).
• Agentic evals: authoring, LLM-as-judge, hermetic tool mocks (ADR 0052); pal ci
  shift-left gates (ADR 0053).
• Post-deployment observability: trace tree, SLOs and error budgets, sampled
  production evals, anomaly incidents (ADR 0056).
• Debug and support agents that propose fixes a person must approve (ADR 0059).

Protocols and integrations

• A2A v0.3: spec card, JSON-RPC tasks, outbound a2a_call (ADR 0057).
• MCP 2025-06-18 client and PAL MCP server, OAuth 2.1, policy-gated tools (ADR 0058).
• Async completion callbacks and batch invocation (ADR 0033); signed inbound webhook
  triggers (ADR 0028); Stripe billing, off by default (ADR 0049).

Console and onboarding

• Dashboard v2 and v3: PAL / PAL Runtime spaces, Ask PAL, Overview, lifecycle tabs,
  Promotions, Sandboxes, Reliability, Incidents, Debug.
• Onboarding agent, a multi-turn conversation for both products (ADR 0051); PAL
  first-run and pal setup (ADR 0048); device tenancy and bridge enrollment (ADR 0047).
• Sign-up modes, email verification, password reset and data export (ADR 0027).

Security and operations

• mTLS on every internal gRPC hop (ADR 0043); API-key scope and RLS secure defaults in
  production (ADR 0042); supply-chain provenance, SBOMs and digest-pinned images (ADR 0044).
• Egress DLP (ADR 0030) and host-asserted software measurement (ADR 0031).
• Helm chart that installs out of the box, scheduler and surface-gateway, HPA/PDB,
  network policy, backup CronJob and restore drill; SLO burn-rate alerts (ADR 0041);
  multi-region active-passive enablers (ADR 0046).
• Fixes: XFF-safe rate limits, connector SSRF guard, mail header injection, real
  pairing codes, patched dependencies.
• Dependency security updates: OpenTelemetry 0.33 / tracing-opentelemetry 0.34 in the
  Rust services (baggage DoS advisory); jsonwebtoken 10.4 with the aws-lc-rs backend in
  the gateway's tests (auth-bypass advisory; the gateway itself verifies API keys via
  introspection, not JWTs); rcgen 0.14 for VM and egress certificates (cert properties
  unchanged, covered by tests); grpc-go 1.83.2; zod 4 in examples; Next.js 16 in the
  dashboard.
• Known, not fixed in this release: minimatch (ReDoS) reaches the dashboard only through
  ESLint, a dev-time tool that never ships or sees untrusted input. Planned upgrade.
  TypeScript 7 waits on typescript-eslint, tsup and Astro check support.

Examples, docs and site

• Runnable examples (support desk, PR triage, incident responder, daily digest,
  Telegram assistant, voice receptionist, data analyst) with a fixture-tier e2e lane.
• Docs site rebuilt for PAL and PAL Runtime, with interactive diagrams, a landing page
  and, in this release, new pages for promotion, agent policy, the console and upgrading.

Detailed entries

Changed

• Product renamed to PAL (the Personal Agentic Layer; ADR 0032).

Added

• Host-asserted software measurement (ADR 0031, ACE A5) — at every Firecracker
  cold spawn the runtime-manager computes a SHA-256 over the five canonical inputs
  it directly controls: kernel image, rootfs image, harness binary (32 zero bytes
  when baked into rootfs), the kernel cmdline, and the spawn contract — minus
  secrets (declared-secret refs, exec token, injection specs, values of
  credential-named env keys) and minus per-VM identity (vm/run id, IPs, per-VM cert
  paths/CA, agent instance id + token, trace parent, the HashMap-ordered
  pal.env.* cmdline copy). Two VMs from the same images + spec measure
  identically. A restore / resume / fork inherits its source's spawn
  measurement (recorded in the snapshot metadata) instead of re-measuring, and
  re-checks it against the current references; an unmeasured (legacy) snapshot is
  refused under an enforcing gate. The 64-character hex measurement is:
  • Embedded in the per-VM X.509 cert as a custom extension (OID 1.3.6.1.4.1.99999.3,
    a placeholder — a registered IANA PEN is needed).
  • Recorded by the scheduler (persisted in sched_vms.software_measurement) and
    written once to runtime_vms.software_measurement (migration
    0029_software_measurement) by the reconciler — for forked/resumed VMs too.
  • Included in signed run receipts as softwareMeasurement (additive; omitempty
    keeps existing receipts byte-identical and their signatures valid).
  • Returned in VmHandle.software_measurement (proto field 5).
    Gate control: PAL_MEASUREMENT_GATE (off | warn [default] | enforce);
    references: PAL_REFERENCE_MEASUREMENTS (comma-separated 64-hex). An unknown
    gate value or malformed reference refuses startup, as does enforce with no
    references in prod. Image hashes are streamed and cached per file identity.
    HOST-ASSERTED only — proves what the host recorded it launched, NOT what ran
    inside the VM; images are hashed by path and opened later by Firecracker (TOCTOU,
    documented). Hardware attestation (ADR 0023) remains UNVERIFIED. Secret-named env
    values (whole _ segments: TOKEN, SECRET, PASSWORD, KEY, AUTH, PAT, DSN, …) are
    blanked and URL userinfo (scheme://user:pass@) is scrubbed from every env value
    before hashing; the snapshot store is inside the trust boundary (unsigned
    metadata; signing is a follow-up). Live check: make lima-measurement-e2e.
• Breaking (Firecracker guests): caller env is no longer sent on the kernel
  cmdline. spawn.json is the only caller-env channel; the cmdline carries just
  the manager-owned PAL_ENV / PAL_MANAGER_ADDR / PAL_WORKLOAD_UID
  (values whitespace-sanitized). The guest echoes its cmdline to the console the
  manager logs, so caller values there (an instance token, a DATABASE_URL
  password) leaked into logs. Harness builds older than spawn.json support are no
  longer supported for caller env; pal.env.RUST_LOG from a caller no longer
  reaches the harness.
• Egress DLP (ADR 0030) — per-tenant data-loss-prevention policy applied by the egress broker to TLS-terminated and plain-HTTP egress. Detectors: Luhn-validated card numbers, secret-key prefixes (AWS/GitHub/Slack/PEM), email (threshold-based), phone (threshold-based), custom regex. Modes: off | audit | block. Fails closed: unknown detector/mode or bad regex fails the spawn (regex also 400 at write), a block rule on a node without a broker refuses the spawn, ambiguous body framing (CL+TE, conflicting CL, bad chunks) is a 400, and in block mode a body that cannot be fully inspected up to the cap is a 403. Chunked bodies are de-chunked before scanning; Expect: 100-continue is answered by the broker. The policy survives pause/resume/fork; DLP VMs skip the warm pool. Not inspected (documented): opaque CONNECT tunnels, bytes past max_inspect_bytes (64 KiB, ≤1 MiB), keep-alive follow-on requests. Audit action dlp carries rule id, detector, host, byte offset/count, decision — never the matched value (invariant #10). Policy stored in dlp_policies (RLS-enforced, migration 0028). REST CRUD at GET/PUT /v1/egress/dlp-policy + POST/DELETE /v1/egress/dlp-policy/rules. Compiled into the broker at VM spawn via AgentSpec.dlp_policy (proto field 19). e2e: broker-e2e.sh DLP block cases (card number, custom_regex).
• Prod enforces API-key scopes — values-prod.yaml (and staging) set PAL_AUTHZ_ENFORCE=1; a first-class controlPlane.env.PAL_AUTHZ_ENFORCE chart value, a boot log line stating the mode, a ControlPlane403RateElevated alert, a chart render test (infra/helm/pal/tests/authz-enforce.sh, run in CI), and the soak/rollback/re-scoping steps in docs/runbooks/ga-staged-rollout.md. JWT sessions and unscoped legacy keys are unaffected.
• Docs site: Cmd/Ctrl-K search over a build-time index (no external service), per-page titles/descriptions/Open Graph tags,
  sitemap.xml + robots.txt, a top bar (landing link, search, GitHub, theme) and an "Edit this page on GitHub" link.
• CI: fixture-tier e2e lane — .github/workflows/sdlc-e2e.yml (jobs examples, browser, deploy-cli) runs the webhook-example suites, Playwright and the pal deploy e2e on every PR and master push with no provider key; skip = fail. Docs truth pass: e2e/README.md test tiers, docs/GA-READINESS.md, docs/LAUNCH-CHECKLIST.md re-verified against code.
• Kiosk demo mode (ADR 0029, #342) — make demo / pal demo start the stack with Docker only and no API
  key: each visitor gets an ephemeral, auto-deleted tenant (30 minutes) with sample agents, answers come from the
  llm-replay fixture server, RLS and scope enforcement are on. pal demo --down / make demo-down delete
  all demo data. Docs: /demo.
• Playwright dashboard tier (#343) — headless-Chromium specs for auth, onboarding, settings and theme against a
  private throwaway stack (own Postgres DB, Redis index, log mailer, fixture LLM). Run npm run test:e2e from
  apps/web (apps/web/e2e/README.md). Docs: /testing.
• Docs: pal demo and pal hooks in the CLI reference.
• Data analyst sandbox example (X5) — examples/data-analyst (pandas image, sandbox
  spec, explainer agent) and e2e/examples/data_analyst_test.go: CSV in, exec analysis, JSON +
  PNG out, egress denied, fixture-LLM explanation. Docker tier runs in CI; the files API and
  ExecTool are Firecracker-only, so that tier is the KVM nightly.
• Examples: daily digest, Telegram assistant, voice receptionist — three
  runnable agents under examples/ (spec, workflow, schedule/number, README with
  a fixture path and a real-account path) and make test-e2e-examples, which
  boots a private fixture stack and asserts: schedule fires -> Slack post;
  signed Telegram update -> reply; signed Twilio webhooks -> TwiML, <Reject>
  over budget, cost reconciled in agent_usage_daily. Signature negatives for
  Telegram and WhatsApp in services/surface-gateway/tests. Docs: /testing,
  /surfaces, /scheduling.
• Surface-gateway replies — a Telegram or WhatsApp message now gets the
  run's output back in the chat (the gateway polls the run and sends it, then
  frees the session). Before, the run was created and the answer was dropped.
  TELEGRAM_API_BASE_URL points Bot API calls at a fixture (refused in prod).
• Examples: PR triage + incident responder (examples/pr-triage-webhook, examples/incident-responder) — signed
  GitHub/Sentry/PagerDuty webhooks drive workflow graphs (GitHub comment + Slack thread; Sentry/PagerDuty lookup,
  Slack, approval node, PagerDuty note). make test-e2e-examples runs them on a throwaway fixture stack
  (scripts/e2e-examples.sh, e2e/examples). Docs: /examples.
• PagerDuty add_note connector action; POST /v1/runs/{id}/takeover/{id}/deny (approval node fails the step).

Fixed

• Fork/restore no longer byte-copies the VM disk on reflink filesystems. Since disk-consistent snapshots, every fork and restore staged the disk with a full copy (O(n) in disk size). The manager now clones it copy-on-write (FICLONE) on XFS/btrfs, falls back to a full copy with a disk_staging="full_copy" warning on ext4/tmpfs, and can be made strict with PAL_REQUIRE_REFLINK=1.
• Receipt verification now detects an edited payload. With a stored receipt, POST /v1/runs/receipts/verify compared only the signature string, so a receipt whose costUsd/status had been changed but that carried the original signature verified as valid: true. It now also requires the payload to match the issued one (canonical compare), which keeps receipts valid across key rotation.
• Workflow connector nodes could not call Sentry or PagerDuty ("dispatch not yet wired").
• A workflow whose JSON held a backslash (e.g. an escaped quote in a prompt) failed to load and the agent silently ran as a plain LLM call (::text::bytea).
• Map-form workflow params (inputMapping/parameters) now render {{...}} templates.
• After an approval park, the resumed node's step_completed/step_failed was dropped from the journal (seq collision with step_waiting).

Added

• Self-service onboarding polish (W2-S): a six-template starter gallery in dashboard onboarding
  (served by GET /v1/agents/templates), a verify-email fix path for 403 email_unverified on run
  creation, free-tier caps (PAL_FREE_TIER_*) applied to every new tenant on password and Google
  sign-up, and the pal.onboarding.step.total counter.
• pal deploy is real (W2-D) — reads agent.yaml, creates the agent or
  updates its prompt/model (PATCH /v1/agents/{name}), saves an optional
  workflow (PUT .../workflow), then POST /v1/agents/{name}/deploy. --json
  prints the API response, --dry-run makes no mutating call. Non-2xx maps to
  exit codes 3 (auth) / 4 (402) / 5 (404) / 6 (5xx). Replaces the simulated
  output. New e2e: e2e/deploy. Docs: /cli#deploy.
• Fixture LLM tier — PAL_LLM_BASEURL_{OPENAI,ANTHROPIC} re-points every
  provider call (completions, streaming, tool loop, web_search) at
  cmd/llm-replay, a deterministic OpenAI/Anthropic wire-format server with
  scriptable replies and tool calls and realistic token usage, so CI and kiosk
  mode need no provider key. Refused in production. Docs: /testing.
• Webhook triggers (ADR 0028, #323) — a public POST /v1/hooks/{id} URL per
  agent; GitHub, Sentry, PagerDuty or any HMAC-SHA256 sender starts a run. The
  signature is the only credential; duplicate deliveries return
  200 {"duplicate":true} and start no run. Managed over REST, the TS/Python/Go
  SDKs, pal hooks and the dashboard (/agents/[name]/triggers). Docs:
  /triggers.
• Account lifecycle (ADR 0027, #318) — PAL_SIGNUP_MODE
  (open/invite/closed, closed by default in prod), email verification,
  password reset, HMAC invite codes, Turnstile captcha, Redis rate limits on the
  pre-auth routes, owner-only data export (GET /v1/tenants/{id}/export) and
  workspace deletion from Settings. Pluggable mailer (PAL_MAILER =
  log/smtp/http). Docs: /accounts.
• Connector actions (#316) — Slack reply_thread, GitHub comment_pr,
  Sentry get_issue, Google Calendar create_event, and a PagerDuty connector.
  Recorded test mode: PAL_CONNECTOR_BASEURL_* points a connector at the
  connector-replay fixture server; refused in production.
• End-to-end kit (#317) — make test-e2e against a live stack with a spend
  policy enforced in code (reserved e2e-* agents, hard-fail budget per agent,
  cheapest model tier) and make e2e-clean. Docs: /testing.
• Docs — new pages for webhook triggers, accounts, end-to-end tests and
  running real Firecracker microVMs on a Mac (Lima); budgets rewritten around an
  enforcement matrix; models page lists every capability name per provider.
• Per-agent model — POST/PATCH /v1/agents accept a model (SDK
  capability such as fast / reasoning-small, or a vendor id) and GET
  returns it. Runs, sessions and auto workflow ai-steps try it first; unset
  or auto keeps the default chain. A cheap-tier agent fails over only to the
  other provider's cheap tier and never to a Sonnet / gpt-4o / Opus class model
  — it fails instead (migration 0025, agents.model). Docs: Agents → Model
  selection, Models → Per-agent model.
• Helm: runtime-scheduler and surface-gateway templates, per-tier HPA, PodDisruptionBudgets, default-deny
  NetworkPolicy with the required allow flows, an opt-in Postgres backup CronJob, and values-ci.yaml.
• Control-plane Prometheus /metrics on :8080; the control-plane SLO alert group is now enabled.
• scripts/db-restore-drill.sh and scripts/check-app-grants.sh; a kind + Calico helm-install CI job.

Fixed

• pal test now works against a real API: POST /v1/runs answers 201 and runs asynchronously, but the CLI required
  200 and read the output immediately. It now polls GET /v1/runs/{id} to a terminal status (--timeout, default 2m per
  case); a failed run or a timeout scores as a failed case with its error. --rehearse had the same bug and is fixed too.
• New sign-ups now land on /onboarding: the authenticated-redirect effect raced the explicit redirect and sent them
  to /agents.
• Password reset and the other /auth/* recovery pages work for logged-out visitors: a 401 from the global
  model-context fetch no longer bounces them off the page.
• Docs: install steps no longer point at registry packages that do not exist (@pal/sdk is not on npm; the
  PyPI name pal-sdk belongs to an unrelated project), broken dshakes.github.io/pal/<page>/ README links
  now include /docs/, and make dev-infra is documented as Postgres and Redis only (no MinIO).
• @pal/sdk built output threw module is not defined on import under Node ESM (the runner's
  require.main === module check); it now checks process.argv only.
• Workflow connector nodes could not call Sentry or PagerDuty ("dispatch not yet wired").
• A workflow whose JSON held a backslash (e.g. an escaped quote in a prompt) failed to load and the agent silently ran as a plain LLM call (::text::bytea).
• Map-form workflow params (inputMapping/parameters) now render {{...}} templates.
• After an approval park, the resumed node's step_completed/step_failed was dropped from the journal (seq collision with step_waiting).
• Workflows silently skipped — agent workflow JSON, run input/output and
  connector configs were read with jsonb::text::bytea, which fails on any
  backslash (\n, \"); the error was swallowed, so such workflows ran as a
  plain LLM call and crash-recovery could not re-drive runs whose input held a
  quote or newline. Now read as ::text.
• Voice cost reconciliation — AdjustUsageCost let Postgres infer the delta
  as INTEGER, truncating every sub-dollar adjustment to 0, so a call's
  duration-based cost (and refund) never reached agent_usage_daily.
• Workflow connector nodes — only gmail/slack/github/notion/linear worked;
  google-calendar, sentry, pagerduty and the rest failed with "not yet wired".
  They now use the shared executor (including OAuth refresh).
• Templated free text in inputMapping — a multi-line or quoted LLM reply
  spliced into a JSON inputMapping made the params invalid (connector got no
  text); values are now JSON-escaped when the raw render does not parse.
• Voice call record failures are logged instead of swallowed.
• Data export redacts credentials structurally (#330) — any column with a
  secret/token/key/password/hash/enc segment is left out, including webhook
  trigger secrets (agent_triggers.secret_enc) and data_planes.reg_token_hash,
  which were previously exported.
• Cost forecast works (#330) — it read run history with
  status = 'completed', which no run has, so it never estimated cost; it now
  averages succeeded runs, so the per-run and daily-cost checks run.
• Workflow runs count toward runs/day (#330) — once each, on success.
  Failed runs still do not count.
• Agents created from a template now keep their system prompt and labels. The
  template-create and prompt back-fill UPDATEs wrote a column that did not
  exist, failed, and the error was discarded.
• /v1/completions honours the request's model on its failover paths.
• Helm: the default in-cluster Postgres crash-looped (Bitnami chart on the pgvector image) and the Bitnami
  MinIO images were no longer pullable, so a default helm install did not come up.
• values-prod.yaml and values-staging.yaml rendered unpullable image references (/pal/pal-gateway:tag);
  they now use the published ghcr.io/dshakes/pal/<name> images.
• Balanced default picks Sonnet (#326) — auto/balanced routing picked
  Opus because price was only a capped bonus; price is now a real penalty, so the
  default is Sonnet (GPT-4o when only OpenAI is keyed).

Changed

• Helm: Postgres (pgvector) and the S3-compatible store are now first-party templates instead of Bitnami
  subcharts, enabled by default for dev/kind and disabled in values-prod.yaml in favour of
  externalDatabase / externalS3. Redis remains a Bitnami subchart (Chart.lock pins it). Not an in-place upgrade from the
  Bitnami-based chart (StatefulSet selectors differ): dump, uninstall, install, restore.
  externalS3.region sets the S3 signing region (default us-east-1).
• Budgets enforced further (#315, #319) — maxTokensPerDay now blocks;
  workflow runs re-check the tenant cap, daily caps and maxCostUsdPerRun
  before every ai-step and confidence vote; toolLimits gate workflow tool
  and connector nodes. A budget lookup error now fails closed for hard-fail
  budgets (503 budget check unavailable; retry shortly at run create, 402
  elsewhere) instead of allowing the call. Still not enforced: the per-run cap on
  single-call paths and projected cost at run creation.
• web_search tool nodes honour the per-run cap (#324) — they check
  maxCostUsdPerRun against the run's accrued spend and add their LLM cost to
  the run, like ai-steps.
• Fast restore (#325) — after resume or fork the runtime-manager holds the
  first exec until the restored harness has re-identified and heartbeated
  (PAL_HARNESS_READY_TIMEOUT_MS, default 5 s) instead of the caller
  retrying; first exec after restore p50 2.9 s on nested-virt Lima (was 4.2 s).
• Workflow tool nodes execute (#319) — tool nodes dispatch to the sandbox
  runtime or the built-in web_search and fail loudly when the runtime is not
  wired, instead of returning a placeholder; side effects carry an idempotency
  key per loop iteration.
• Capability resolution (#320) — unknown capability names route to the
  balanced default (logged once) instead of a hardcoded model; fast and
  cheap resolve to the cheap tier.
• Disk snapshots (#322) — Firecracker snapshots capture the VM's writable
  disk alongside memory and vmstate; resume, idle wake and fork stage from it, so
  files written before a snapshot survive restore.