v0.2.2 (sync from internal development)

.github/workflows/proto-check.yml

@@ -17,7 +17,7 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
 
       - name: Install protoc
         uses: arduino/setup-protoc@v3

.github/workflows/test.yml

@@ -18,7 +18,7 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
           cache-dependency-path: server/go.sum
 
       - name: Run go vet
@@ -45,7 +45,7 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
           cache-dependency-path: server/go.sum
 
       - name: golangci-lint
@@ -128,12 +128,16 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
           cache-dependency-path: server/go.sum
 
       - name: Run integration tests
         working-directory: server
-        run: go test -race -count=1 -tags=integration ./...
+        # Serialize with -p 1 -parallel 1: these integration tests spin up
+        # embedded NATS clusters and real sidecar runners/tunnels. Running them
+        # concurrently on the 2-core runner starves timing-sensitive setup
+        # (JetStream elections, tunnel handshakes) and causes contention flakes.
+        run: go test -race -count=1 -p 1 -parallel 1 -tags=integration ./...
 
   e2e:
     runs-on: ubuntu-latest
@@ -142,7 +146,7 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
           cache-dependency-path: server/go.sum
 
       # The e2e suite spawns aetherlite as a subprocess and exercises the
@@ -155,7 +159,10 @@ jobs:
         working-directory: server
         env:
           AETHER_ALLOW_DEV_MODE: "true"
-        run: go test -tags=e2e -count=1 -timeout 360s ./internal/proxysidecar/integration_e2e/...
+        # -parallel 1: the e2e tests share one aetherlite subprocess; running the
+        # t.Parallel tunnel tests concurrently against the shared gateway causes
+        # tunnel-lifecycle races (PEER_RESET) under 2-core-runner load. Serialize.
+        run: go test -tags=e2e -count=1 -p 1 -parallel 1 -timeout 360s ./internal/proxysidecar/integration_e2e/...
 
   security:
     runs-on: ubuntu-latest
@@ -164,7 +171,7 @@ jobs:
 
       - uses: actions/setup-go@v5
         with:
-          go-version: '1.25.10'
+          go-version: '1.25.12'
           cache-dependency-path: server/go.sum
 
       - name: Install govulncheck
@@ -173,3 +180,13 @@ jobs:
       - name: Run govulncheck
         working-directory: server
         run: govulncheck ./...
+
+      # Informational scan of the published Go SDK, which reaches the Docker
+      # orchestrator (sdk/go/orchestrators/docker) and thus CALLS the docker/docker
+      # vulnerabilities tracked in SECURITY.md's Known Issues. continue-on-error so
+      # the build stays green on those already-tracked, no-upstream-fix advisories,
+      # while surfacing them (and any genuinely new SDK vuln) in the CI log.
+      - name: Run govulncheck (SDK — informational, non-blocking)
+        working-directory: sdk/go
+        continue-on-error: true
+        run: govulncheck ./...

.github/workflows/version-check.yml

@@ -23,7 +23,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install dependencies
-        run: pip install pyyaml
+        run: pip install pyyaml uv
 
       - name: Check version consistency
         run: python scripts/update-versions.py --check --verbose

.gitignore

@@ -32,3 +32,4 @@ build/
 /RELEASING.md
 /server/coverage.out
 /.run-ci-logs/
+/bin/

CHANGELOG.md

@@ -6,9 +6,42 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Aet
 
 ---
 
-## [Unreleased]
+## [0.2.2] - Unreleased
 
-Internal builds carry a **v0.2.1** gateway/SDK version stamp; no v0.2.1 tag is published yet.
+Work landed since the **v0.2.1** release (2026-05-22); not yet tagged.
+
+### Added
+
+- **[GATEWAY / SDK] `uu::` user-broadcast topic** for cross-window user targeting (`uu::{user_id}`). A platform→user channel carrying ordinary `MessageEnvelope` protos; only Service, WorkflowEngine, and Bridge principals may publish (`routing.go` enforces the sender-type gate). SDKs gain user-broadcast send helpers and the channel is documented.
+- **[GATEWAY / AETHERLITE] OpenTelemetry OTLP metrics + Go runtime metrics** (`internal/tracing/meter.go`), gated on the same `OTEL_EXPORTER_OTLP_ENDPOINT` env var as tracing — when unset, metrics are a no-op. Both the gateway and aetherlite binaries initialize the OTLP gRPC meter alongside the existing tracer.
+- **[GATEWAY] `ServiceIdentity.no_pool_consumer`** proto field lets a Service principal opt OUT of pool-task consumer routing (serve-only), so a service can connect without being enrolled as a POOL task consumer.
+- **[AUTHPROXY] Internal-only `/auth/verify-optional` endpoint** — an optional-auth variant of `/auth/verify` for `ext_authz` optional-auth flows. With no credential present it passes through anonymously and emits EMPTY identity headers; a present-but-invalid credential still fails closed.
+- **[PROXY SIDECAR] Tenant-relay and aggregator service-tunnel modes** for relaying sandbox tunnels through a tenant boundary, plus a `WatchTenants` `snapshot_complete` sentinel and support for dialing `unix://` terminator backends.
+- **[GATEWAY / CLEANUP] Stale-task TTL sweeps.** A stale `agent_startup` pool-task reaper and a stale-interactive-task sweep (`InteractiveTaskTTL` / `InteractiveTaskCancelInterval`) auto-cancel non-terminal tasks left behind by disconnected clients; sweeps run single-node-direct so aetherlite (no leader election) reaps reliably.
+- **[GATEWAY] Task COMPLETE/FAIL outcome logging.** The gateway now logs every task-op outcome — success and each rejection reason (not found, unauthorized, completeFn error) — for the COMPLETE/FAIL branches.
+- **[GO SDK] `ServiceTopic` helper** for workspace-less service principals.
+
+### Changed
+
+- **[AETHERLITE / BADGER ROUTER] Badger message + offset retention TTLs.** The single-node Badger router now expires stored messages (default 24h, override via `AETHER_MESSAGE_RETENTION_TTL`) and orphaned per-consumer offset keys (default 7d, override via `AETHER_OFFSET_RETENTION_TTL`). Active consumers keep re-saving their offset so the TTL refreshes and the key never expires while in use; an offset for a consumer that never returns (e.g. a closed browser tab) is eventually reclaimed instead of growing without bound.
+- **[GATEWAY] Broadcast/progress subscriptions use resume-or-tail semantics.** Global-user (`gu::`), user-workspace (`uw::`), user-broadcast (`uu::`), and per-user progress (`pg::`) lanes now subscribe as named resume-or-tail (exclusive) consumers, so a reconnect replays only the missed gap instead of re-delivering the whole lane; the task-message lane does a full replay on (re)connect.
+- **[GATEWAY / SDK] Resolved OBO subject propagated on delivered messages** so downstream consumers can attribute a delivered message to the on-behalf-of subject rather than only the system actor.
+- **[AUDIT] High-volume proxy-route event coalescing.** A bounded, sharded coalescer collapses repetitive `proxy_http_routed`-class events (e.g. per-token chat streaming) while never coalescing failures, denials, or auth/task/kv/control events, which are always audited individually. Complements the new scheduled audit-retention sweep.
+
+### Fixed
+
+- **[GO SDK] Metric/event topic helpers now use the `::` separator.** `MetricTopic`/`MetricWildcardTopic` and `EventTopic`/`EventWildcardTopic` emitted the legacy dot form (`metric.*` / `event.*`), which the gateway's `validateTopicFormat` rejects as `ERR_INVALID_TOPIC` — so every `SendMetric`/`SendEvent` publish was silently dropped. They now emit the `::` identity separator.
+- **[GATEWAY] Agent principals can publish their own metrics without workspace WRITE.** `checkMessageSend` gated metric publishes on a per-workspace WRITE grant, so intentionally READ-only agents (e.g. sandbox agents seeded READ on their home workspace) could never emit their own token/usage metrics. An agent targeting the metric fan-in plane (`metric::*`) is now additively granted ReadWrite, realizing the "same-workspace publish is implicit" contract; cross-workspace, topic-eligibility, and negative-delta checks are unaffected.
+- **[WORKFLOW] Event dispatch moved off the receive loop (`AsyncMessageHandler`).** Handling events inline on the receive loop let a synchronous KV-coordination call self-deadlock (the response needed the blocked loop), surfacing as `coalesce gate: DEADLINE_EXCEEDED`; event handling now runs asynchronously.
+- **[BADGER ROUTER] Consumer-offset durability.** The router persists a consumer's offset after a replay and retries `saveOffset` on an optimistic-concurrency conflict, so offsets survive reconnects instead of being lost under contention.
+- **[AETHERLITE] Connection-storm write-contention hardening** for the Badger and SQLite lite stores, reducing write contention when many clients connect at once.
+- **[ORCHESTRATION / CLEANUP] `orchestrated_task_queue` retirement.** Queue rows whose task is terminal or missing are now retired reliably and orphans reconciled on a schedule (`QueueReconcileInterval`, default 5m; 0 disables).
+
+---
+
+## [0.2.1] - 2026-05-22
+
+The accumulated work that shipped in the **v0.2.1** SDK/gateway version bump (Go / Python / TypeScript).
 
 ### Added
 

CLAUDE.md

@@ -14,7 +14,7 @@ This project is made by scitrera.ai. Therefore, the naming involves "scitrera";
 - api/ - Protobuf definitions and generated code (own Go module: `github.com/scitrera/aether/api`)
 - sdk/ - Client SDKs (Go, Python, TypeScript)
 - server/ - Go server (module: `github.com/scitrera/aether`)
-  - server/cmd/ - Main entry points (gateway, auth-proxy, migrate, cleanup, loadtest)
+  - server/cmd/ - Main entry points (gateway, aetherlite, auth-proxy, proxy-sidecar, workflow, migrate, cleanup, init-secrets, readiness-check, loadtest)
   - server/configs/ - YAML configuration files (e.g., dev.yaml)
   - server/deployments/ - Kubernetes manifests and Docker Compose files
   - server/docs/ - Server documentation (quickstart, scaling, monitoring, admin API, error codes, etc.)
@@ -24,7 +24,7 @@ This project is made by scitrera.ai. Therefore, the naming involves "scitrera";
   - server/scripts/ - Server dev scripts (infra, test, certs, load test)
   - server/specification.md - Full system specification (v4.0, reflects actual implementation)
   - server/go.mod, server/go.sum - Go module files
-  - server/Dockerfile - Multi-stage build for gateway, cleanup, and migrate binaries
+  - server/Dockerfile - Multi-stage build for gateway, cleanup, migrate, auth-proxy, init-secrets, workflow, aetherlite, and proxy-sidecar binaries
   - server/Makefile - Build/test/run targets (run from server/ directory)
 - scripts/ - Repo-wide scripts (compile_protos.sh)
 - refs/ - Reference materials (external open source code; exclude from general scans)
@@ -111,7 +111,7 @@ cd server
 | **Orchestration** | `server/internal/orchestration/` | Task dispatch via AMQP, claim-based delivery, profile management |
 | **Admin Server** | `server/internal/admin/server.go` | REST API + embedded UI; ops server for health probes + Prometheus metrics |
 | **Auth Proxy** | `server/cmd/auth-proxy/` + `server/internal/authproxy/` | Standalone auth gateway for external services (e.g., MemoryLayer) |
-| **Identity Model** | `server/pkg/models/identity.go` | Seven principal types (Agent, Task, User, Orchestrator, WorkflowEngine, MetricsBridge, Bridge), topic address derivation via `ToTopic()` |
+| **Identity Model** | `server/pkg/models/identity.go` | Eight principal types (Agent, Task, User, Service, Orchestrator, WorkflowEngine, MetricsBridge, Bridge), topic address derivation via `ToTopic()` |
 
 ### Topic Schema and Routing
 
@@ -123,11 +123,13 @@ cd server
 | `tb` | `tb::{workspace}::{impl}` | Task broadcast (load-balancing) |
 | `us` | `us::{user_id}::{window_id}` | User window-specific |
 | `uw` | `uw::{user_id}::{workspace}` | User workspace-scoped |
+| `uu` | `uu::{user_id}` | User broadcast — reaches all of a user's windows regardless of active workspace; workspace-agnostic, ordinary (non-progress) messages. Platform-principal senders only (see permission matrix). |
 | `ga` | `ga::{workspace}` | Global agent broadcast |
 | `gu` | `gu::{workspace}` | Global user broadcast |
 | `pg` | `pg::{workspace}` | Progress updates (server-side recipient filtering) |
-| `event.*` | Write: `event.{workspace}` (gateway rewrites to `event::receiver{shard}`); Subscribe (WE): `event::receiver0` | Workflow Engine fan-in; today 1 shard (`event::receiver0`) |
-| `metric.*` | Write: `metric.{workspace}` (gateway rewrites to `metric::receiver{shard}`); Subscribe (MB): `metric::receiver0` | Metrics Bridge fan-in; today 1 shard (`metric::receiver0`) |
+| `event::` | Write: `event::{workspace}` (gateway rewrites to `event::receiver{shard}`); Subscribe (WE): `event::receiver0` | Workflow Engine fan-in; today 1 shard (`event::receiver0`). A legacy `event.*` (dot) form is rejected as an invalid topic prefix. |
+| `metric::` | Write: `metric::{workspace}` (gateway rewrites to `metric::receiver{shard}`); Subscribe (MB): `metric::receiver0` | Metrics Bridge fan-in; today 1 shard (`metric::receiver0`) |
+| `tk` | `tk::{workspace}::{task_id}::events` (TaskEvent stream) and `tk::{workspace}::{task_id}::msg` (per-task chat) | Per-task lanes; SUBJECT sessions auto-subscribe for the task's lifetime. Task-message lane uses full replay; user broadcast lanes (`gu`/`uw`/`uu`) and `pg` use resume-or-tail. |
 | `br` | `br::{impl}::{spec}` | Bridge (cross-workspace messaging integration) |
 
 ### Connection Flow
@@ -185,7 +187,9 @@ An active gRPC stream connection represents both the distributed lock for that i
 
 Cross-workspace sends are blocked: workspace-scoped principals cannot target topics in other workspaces. Bridges are cross-workspace by design (no workspace component) and check ACL per-message against the target workspace.
 
-**Cross-workspace event/metric broadcast:** Sending to `event.*` or `metric.*` in another workspace requires `capability/event_broadcast` or `capability/metric_broadcast` ACL permission. Sending to the sender's own native workspace is implicitly permitted.
+**User-broadcast (`uu::{user_id}`):** A workspace-agnostic channel that reaches every one of a user's windows regardless of which workspace each window is viewing (the non-progress complement to `pg::us::{user}`). Because the topic carries no workspace segment, the workspace ACL cannot gate it, so authorization is by principal type: **only Service, WorkflowEngine, and Bridge principals may publish** (`enforceTopicPermissions`). Users, Agents, Tasks, and Orchestrators are denied and must reach a user via `us::`/`uw::`/progress or task ownership. Users subscribe to their own `uu::` topic on connect.
+
+**Cross-workspace event/metric broadcast:** Sending to `event::` or `metric::` topics in another workspace requires `capability/event_broadcast` or `capability/metric_broadcast` ACL permission. Sending to the sender's own native workspace is implicitly permitted.
 
 **Metric payloads are structured:** `METRIC` messages must carry a `Metric` proto payload (fields: `trace_id`, `entries` [{`name`, `kind`, `qty`}], `metadata`, `client_timestamp_ms`). All entries are additive deltas; negative `qty` requires the `capability/metric_credit` ACL permission. See spec Section 4.5 for details and error codes.
 
@@ -224,6 +228,12 @@ PostgreSQL is used for persistent features that gracefully degrade when unavaila
 
 Schema is managed by embedded migrations in `server/migrations/` that auto-run on startup.
 
+### Observability
+Both `gateway` and `aetherlite` export OpenTelemetry traces and metrics via OTLP gRPC (`internal/tracing/`: `InitTracer`, `InitMeter`, `NewLogBridge`), gated on the standard `OTEL_EXPORTER_OTLP_ENDPOINT` env var (no-op when unset; OTLP gRPC default port 4317). This is independent of the admin ops server's Prometheus `/metrics` endpoint (port 9090).
+
+### Background Cleanup / Reconcile
+The cleanup service (`internal/cleanup/`, wired in both `gateway` and `aetherlite`) runs periodic sweeps: orphaned `orchestrated_task_queue` reconciliation (`QueueReconcileInterval`, default 5m), audit-log retention sweep, stale `agent_startup` / interactive-task TTL reapers, and a stale regular pool-task sweep (`CancelStalePoolTasks`). On single-node AetherLite these run leader-gated/single-node-direct.
+
 ## Horizontal Scaling
 
 - **Stateless Design:** All state in Redis and PostgreSQL. Gateway instances share nothing.
@@ -235,7 +245,7 @@ Schema is managed by embedded migrations in `server/migrations/` that auto-run o
 ### Deployment Options
 - **Kubernetes:** `server/deployments/k8s/gateway/` — Deployment, Service, Ingress, ConfigMap, cert-manager
 - **Docker Compose:** `server/deployments/docker-compose/multi-instance.yaml` — 3 instances + nginx
-- **Dockerfile:** `server/Dockerfile` — Multi-stage build, non-root user, ports 50051/9090/31880 (build from repo root: `docker build -f server/Dockerfile .`)
+- **Dockerfile:** `server/Dockerfile` — Multi-stage build, non-root user, ports 50051/9090/31880/8080 (build from repo root: `docker build -f server/Dockerfile .`)
 
 ## Client SDKs