(CODUX-216) Add authenticated headless readiness and recovery coverage

[numnx]

🧩 Task: Add authenticated headless readiness and recovery coverage

Automated task PR opened by Code UX for T08.

📋 Summary

Sprint: Sprint 216 · Nodes Full Integration · Task: T08

Turn the existing project-scoped node-flow foundation and browser-local canvas into a canonical, secure, versioned automation control plane with registry-driven nodes, encrypted credentials, durable execution, governed integrations, isolated custom nodes, agent authoring, dashboard integration, and authenticated headless readiness.

🤖 Model & Provider

Provider	Codex CLI
Model	gpt-5.6-sol

⏱️ Timing

Started	2026-07-12 06:51:14 UTC
Finished	2026-07-12 07:09:31 UTC
Duration	18m 17s

📊 Token Usage

Metric	Value
Input tokens	431,940
Cached input tokens	19,547,648
Output tokens	71,505
Total tokens	20,051,093
Invocations	3
Tool calls	153
Active time	29m 17s

Included usage estimate (subscription/local login): $14.08

✅ QA Review

⏳ QA review runs after this PR is opened. Check the task card in the dashboard for the outcome once it lands.

📝 Full Task Prompt

## Objective
Make credentialed automations operable in authenticated headless mode and prove the full authoring-to-execution path with mocked external boundaries and restart-safe side effects.

## Scope
- `src/server/dashboard-security.ts`
- `src/server/dashboard-middleware.ts`
- `src/server/dashboard-route-registration.ts`
- `src/server/dashboard-server.ts`
- `src/worker/index.ts`
- `src/contracts/` for principals, roles, service identities, readiness, and audit records
- `src/app/dependency-factory/core-factory.ts` and `dashboard-factory.ts`
- New authentication/authorization, audit-export, readiness, SLO, and distributed-runner services
- `tests/backend/server/`, `tests/backend/services/`, and new end-to-end fixtures under `tests/e2e/` or `scripts/e2e/`
- `docs/operations/server-mode.md`, `docs/operations/security-hardening.md`, `docs/operations/runbook.md`, architecture documentation, `docs/index.md`, `docs/SUMMARY.md`, and matching `docs-web/content/docs/` pages

## Implementation Requirements
1. Add authenticated dashboard and administrative API enforcement with OIDC or trusted reverse-proxy identity boundaries, roles for credential administration, automation authoring, publishing, running, and viewing, project-scoped authorization, CSRF/origin protection, TLS deployment assumptions, and request-rate limits.
2. Add runner service identities, remote credential-management feature gates, KMS/Vault readiness checks, startup failure when encrypted data exists but keys are unavailable, distributed lease ownership, audit export, and metrics/correlation events for management calls, builds, credential access, runs, attempts, approvals, and outbox delivery.
3. Add an approved local test-project end-to-end scenario using mocked job and email APIs that asks an agent to build the example automation, configures credentials externally, generates/validates a custom node, processes exactly 20 fixture records, drafts and approves selected messages, delivers through the mock outbox, restarts during execution, rotates a credential, and rolls back to an earlier published version.
4. Assert no duplicate side effects after restart/retry, no secret disclosure in API responses, logs, traces, graph exports, prompts, or diagnostics, and correct behavior for missing keys, revoked credentials, unauthorized projects, and unavailable external providers.
5. Document deployment, backup/restore, key rotation, recovery, rollback, SLO baselines, alert conditions, and the remaining limitations of local versus authenticated headless operation in both canonical and public docs.

## Constraints
- Use only mocked external boundaries and the approved local test project for end-to-end execution.
- Do not expose unauthenticated credential-management routes merely because local desktop mode remains available.
- Do not claim production readiness until authentication, key readiness, recovery, audit, and duplicate-prevention checks pass.

## Verification
- Run focused authentication, authorization, readiness, audit, runner, and recovery tests with `pnpm exec vitest run`.
- Run the mocked end-to-end scenario and verify the full authoring, credential, custom-node, approval, restart, rotation, and rollback sequence.
- Run `pnpm run lint`, `pnpm run test:backend`, `pnpm run test:dashboard`, `pnpm run build`, `pnpm run audit`, and `pnpm run check:docs-web`.
- Confirm `/health` remains live, `/ready` fails closed when required key providers are unavailable, and all documented recovery drills produce durable, redacted results.

🌿 Branch Info

Base: feature/CODUX-216-nodes-full-integration
Head: task/feature-codux-21-t08-codex-6bcd6902-mrhf9zca

---

🤖 Generated by Code UX