docs: Update all documentation for Next.js/SSE migration

.github/copilot-instructions.md

@@ -1,18 +1,18 @@
 # Copilot Instructions
 
-- **Scope**: Memory Loop is a Bun monorepo (backend Hono server, React 19 frontend, shared Zod protocol). Core protocol schemas live in [shared/src/protocol.ts](shared/src/protocol.ts) and are the source of truth for WebSocket message types.
-- **Dev commands**: From repo root run `bun install`; `bun run dev` spins up both apps; `bun run test` runs backend then frontend; `bun run typecheck` and `bun run lint` fan out to all workspaces. Backend-only dev is `bun run --cwd backend dev`; frontend-only is `bun run --cwd frontend dev`.
-- **Environments**: Requires `VAULTS_DIR` pointing at folders containing a `CLAUDE.md` per vault. Optional `MOCK_SDK=true` disables real Anthropic calls; backend serves on `PORT`/`HOST` (defaults 3000/0.0.0.0). Production uses `bun run build` then [scripts/launch.sh](scripts/launch.sh).
-- **Backend shape**: Hono app in [backend/src/server.ts](backend/src/server.ts) exposes `/api/health`, `/api/vaults`, `/api/sessions/:vaultId`, vault asset passthrough under `/vault/:vaultId/assets/*` (image-only with traversal checks), and WebSocket `/ws` (per-connection handler instance). Static frontend is served from `frontend/dist`.
-- **WebSocket flow**: Handler in [backend/src/websocket-handler.ts](backend/src/websocket-handler.ts) validates messages with `safeParseClientMessage`, routes by `type`, and emits JSON-serialized server messages. Streaming responses emit `response_start` → `response_chunk`* → `response_end`; tool invocations stream `tool_start`/`tool_input`/`tool_end`. Abort calls interrupt active queries.
-- **Sessions**: Claude Agent SDK sessions are created/resumed via [backend/src/session-manager.ts](backend/src/session-manager.ts) with defaults in `DISCUSSION_MODE_OPTIONS` (auto-allowed read tools, `permissionMode=acceptEdits`, streaming enabled). Session metadata persists to `~/.memory-loop/sessions/*.json` and is pruned to the most recent five per vault; IDs are validated to avoid traversal.
+- **Scope**: Memory Loop is a Bun monorepo (backend library, Next.js 15 App Router frontend, shared Zod schemas). Core types live in [shared/src/types.ts](shared/src/types.ts) and are the source of truth for API and SSE event types.
+- **Dev commands**: From repo root run `bun install`; `bun run --cwd nextjs dev` starts the Next.js dev server on port 3000; `bun run test` runs all workspaces sequentially; `bun run typecheck` and `bun run lint` fan out to all workspaces.
+- **Environments**: Requires `VAULTS_DIR` pointing at folders containing a `CLAUDE.md` per vault. Optional `MOCK_SDK=true` disables real Anthropic calls; Next.js serves on `PORT`/`HOSTNAME` (defaults 3000/0.0.0.0). Production uses `bun run --cwd nextjs build` then [scripts/launch.sh](scripts/launch.sh).
+- **Backend shape**: Backend is a library (no HTTP server) consumed by Next.js API routes. Domain logic lives in [backend/src/](backend/src/): session management, vault operations, streaming, note capture, file browsing. Next.js API routes in [nextjs/app/api/](nextjs/app/api/) are thin wrappers that call backend functions.
+- **SSE streaming**: AI chat uses SSE via POST to `/api/chat`. The [Active Session Controller](nextjs/lib/controller.ts) is a singleton (survives Next.js HMR via globalThis) that manages the live SDK connection. Frontend sends prompts via REST, reads the SSE stream for incremental responses. Stop/permission/answer requests are separate REST calls alongside the stream.
+- **Sessions**: Claude Agent SDK sessions are created/resumed via [backend/src/session-manager.ts](backend/src/session-manager.ts). Session metadata persists to vault metadata directory and is pruned to the most recent five per vault; IDs are validated to avoid traversal.
 - **Vault discovery and goals**: [backend/src/vault-manager.ts](backend/src/vault-manager.ts) scans `VAULTS_DIR` for directories containing `CLAUDE.md`, extracts the first `#` heading as the vault name, detects inbox paths from common patterns, and optionally loads goals from `06_Metadata/memory-loop/goals.md`. Goals are parsed into sections with a 9-item cap.
 - **Note capture**: [backend/src/note-capture.ts](backend/src/note-capture.ts) writes to `00_Inbox/YYYY-MM-DD.md` (creates inbox as needed), appends under `## Capture` with `- [HH:MM] text`, preserves existing content, and normalizes CRLF. Empty text returns a protocol error.
 - **File browsing**: [backend/src/file-browser.ts](backend/src/file-browser.ts) whitelists markdown reads only, rejects symlinks, enforces vault-boundary checks (`realpath`), and truncates reads >1MB. Directory listings skip dotfiles and sort dirs-first.
 - **Inspiration**: [backend/src/inspiration-manager.ts](backend/src/inspiration-manager.ts) pulls contextual prompts from `06_Metadata/memory-loop/contextual-prompts.md` (weekday-only generation) and weekly quotes from `06_Metadata/memory-loop/general-inspiration.md`. Context gathering prioritizes recent daily notes and project/area README/index files with a 3200-char budget.
-- **Frontend state**: [frontend/src/contexts/SessionContext.tsx](frontend/src/contexts/SessionContext.tsx) is the single source of truth (useReducer). Modes: home, note, discussion, browse. Vault ID and browser path persist via localStorage; messages are server-sourced only. Pinned folders are stored per-vault locally.
-- **WebSocket client**: [frontend/src/hooks/useWebSocket.ts](frontend/src/hooks/useWebSocket.ts) builds the ws URL from the current origin, validates server messages with `safeParseServerMessage`, and auto-reconnects with exponential backoff. `useServerMessageHandler` in SessionContext consumes `session_ready`/`response_*` events and updates state.
-- **UI composition**: Root shell in [frontend/src/App.tsx](frontend/src/App.tsx) gates on vault selection (`VaultSelect`), then renders mode-specific components (`HomeView`, `NoteCapture`, `Discussion`, `BrowseMode`) with header actions for new session and vault switch dialogs.
-- **Testing**: Backend `bun run test` iterates all files under [backend/src/__tests__](backend/src/__tests__), often using temp dirs and mocked SDK. Frontend tests use `@testing-library/react` with happy-dom from [frontend/src/test-setup.ts](frontend/src/test-setup.ts). Protocol tests live in [shared/src/__tests__/protocol.test.ts](shared/src/__tests__/protocol.test.ts).
-- **Testing constraints**: Tests cannot run in parallel (filesystem contention causes flaky failures). Do not use `mock.module()` (causes infinite loops in Bun); use dependency injection instead. The SDK uses a provider pattern ([backend/src/sdk-provider.ts](backend/src/sdk-provider.ts)) where only `index.ts` calls `initializeSdkProvider()`; tests use `configureSdkForTesting(mockFn)` to prevent accidental API calls.
-- **Common pitfalls**: WebSocket endpoints expect JSON matching protocol discriminators; invalid JSON returns `VALIDATION_ERROR`. File reads must be `.md` relative to vault root. When mocking, set `MOCK_SDK=true` before starting the backend; otherwise the SDK will try to spawn Claude Code.
+- **Frontend state**: [nextjs/contexts/SessionContext.tsx](nextjs/contexts/SessionContext.tsx) is the single source of truth (useReducer). Modes: home, note, discussion, browse. Vault ID and browser path persist via localStorage; messages are server-sourced only. Pinned folders are stored per-vault locally.
+- **SSE client**: [nextjs/hooks/useChat.ts](nextjs/hooks/useChat.ts) sends a POST to `/api/chat`, reads the SSE response stream, and dispatches events to SessionContext. Event types include `session_ready`, `response_chunk`, `tool_start`/`tool_end`, `request_permission`, and `ask_question`.
+- **UI composition**: Root shell in [nextjs/app/page.tsx](nextjs/app/page.tsx) gates on vault selection (`VaultSelect`), then renders mode-specific components (`HomeView`, `NoteCapture`, `Discussion`, `BrowseMode`) with header actions for new session and vault switch dialogs.
+- **Testing**: Backend tests use Bun's test runner under [backend/src/__tests__](backend/src/__tests__), often using temp dirs and mocked SDK. Next.js tests use `@testing-library/react` with happy-dom via [nextjs/test-setup.ts](nextjs/test-setup.ts). Shared tests live in [shared/src/__tests__](shared/src/__tests__).
+- **Testing constraints**: Tests cannot run in parallel (filesystem contention causes flaky failures). Do not use `mock.module()` (causes infinite loops in Bun); use dependency injection instead. The SDK uses a provider pattern ([backend/src/sdk-provider.ts](backend/src/sdk-provider.ts)); tests use `configureSdkForTesting(mockFn)` to prevent accidental API calls.
+- **Common pitfalls**: File reads must be `.md` relative to vault root. When mocking, set `MOCK_SDK=true`; otherwise the SDK will try to spawn Claude Code. SSE events are newline-delimited JSON with `data:` prefix.

.lore/design/active-session-controller.md → .lore/_archive/active-session-controller.md

@@ -1,10 +1,10 @@
 ---
 title: Active Session Controller interface
 date: 2026-02-03
-status: draft
+status: implemented
 tags: [architecture, interface-design, session-management, pub-sub]
 modules: [active-session-controller, websocket-handler, session-manager]
-related: [.lore/specs/session-viewport-separation.md]
+related: [.lore/_archive/session-viewport-separation-spec.md]
 ---
 
 # Design: Active Session Controller Interface
@@ -13,7 +13,7 @@ related: [.lore/specs/session-viewport-separation.md]
 
 Design the interface for Active Session Controller - the new component that owns the live SDK connection and lets viewports (WebSocket connections) subscribe/unsubscribe.
 
-See [Spec: session-viewport-separation](.lore/specs/session-viewport-separation.md) for requirements.
+See [Spec: session-viewport-separation](.lore/_archive/session-viewport-separation-spec.md) for requirements.
 
 ## Constraints
 

.lore/reference/_infrastructure/communication-layer.md → .lore/_archive/communication-layer.md

@@ -1,7 +1,7 @@
 ---
 title: Communication Layer Infrastructure
 date: 2026-01-28
-status: current
+status: outdated
 tags: [websocket, rest-api, protocol, streaming]
 modules: [websocket-handler, server, protocol]
 ---

.lore/research/daily-planning-science.md → .lore/_archive/daily-planning-science.md

@@ -2,7 +2,7 @@
 title: Science of Daily Planning and Morning Rituals
 date: 2026-02-02
 status: archived
-tags: [daily-planning, productivity, psychology, behavioral-economics, energy-management]
+tags: [daily-planning, productivity, psychology, behavioral-economics, energy]
 related: [.lore/brainstorm/daily-prep-system.md]
 ---
 

.lore/excavations/index.md → .lore/_archive/excavation-index.md

@@ -1,7 +1,7 @@
 ---
 title: Excavation Index
 date: 2026-01-28
-status: current
+status: outdated
 tags: [excavation, index, documentation]
 ---
 

.lore/diagrams/session-auto-resume.md → .lore/_archive/session-auto-resume.md

@@ -1,7 +1,7 @@
 ---
 title: Session Auto-Resume Flow
 date: 2026-01-28
-status: current
+status: outdated
 tags: [architecture, session, websocket, rest-api]
 modules: [session-manager, vault-select]
 ---

.lore/brainstorm/session-viewport-separation.md → .lore/_archive/session-viewport-separation-brainstorm.md

@@ -1,7 +1,7 @@
 ---
 title: Session and viewport separation
 date: 2026-02-03
-status: open
+status: parked
 tags: [architecture, websocket, session-management, separation-of-concerns]
 modules: [websocket-handler, session-manager]
 ---

.lore/specs/session-viewport-separation.md → .lore/_archive/session-viewport-separation-spec.md

@@ -4,7 +4,7 @@ date: 2026-02-03
 status: draft
 tags: [architecture, websocket, session-management, separation-of-concerns, refactor]
 modules: [websocket-handler, session-manager, active-session-controller]
-related: [.lore/brainstorm/session-viewport-separation.md]
+related: [.lore/_archive/session-viewport-separation-brainstorm.md]
 ---
 
 # Spec: Session Viewport Separation
@@ -84,5 +84,5 @@ Three distinct components with clear ownership:
 ## Context
 
 - [Brainstorm: Session and Viewport Separation](/.lore/brainstorm/session-viewport-separation.md)
-- [Diagram: WebSocket Connection Lifecycle](/.lore/diagrams/websocket-connection-lifecycle.md)
-- [Reference: Communication Layer](/.lore/reference/_infrastructure/communication-layer.md)
+- [Diagram: WebSocket Connection Lifecycle](/.lore/_archive/websocket-connection-lifecycle.md) (archived - WebSocket replaced by SSE)
+- [Reference: Communication Layer](/.lore/_archive/communication-layer.md) (archived - WebSocket replaced by SSE)

.lore/design/single-websocket-consolidation.md → .lore/_archive/single-websocket-consolidation.md

@@ -1,10 +1,10 @@
 ---
 title: Single WebSocket Consolidation
 date: 2026-02-03
-status: implemented
+status: superseded
 tags: [websocket, architecture, rest-api, migration]
 modules: [websocket-handler, app, discussion, session-context]
-related: [.lore/specs/session-viewport-separation.md, .lore/reference/_infrastructure/communication-layer.md]
+related: [.lore/_archive/session-viewport-separation-spec.md, .lore/_archive/communication-layer.md]
 ---
 
 # Design: Single WebSocket Consolidation

.lore/diagrams/websocket-connection-lifecycle.md → .lore/_archive/websocket-connection-lifecycle.md

@@ -1,7 +1,7 @@
 ---
 title: WebSocket Connection Lifecycle
 date: 2026-01-28
-status: current
+status: outdated
 tags: [architecture, websocket, connection, reconnect]
 modules: [websocket-handler, use-websocket]
 ---

.lore/lore-agents.md

@@ -1,6 +1,6 @@
 ---
 title: Lore Agents Registry
-date: 2026-01-29
+date: 2026-02-05
 status: current
 tags: [agents, registry, tooling]
 ---
@@ -9,20 +9,29 @@ tags: [agents, registry, tooling]
 
 Specialized agents available for lore-development work in this project.
 
-Last updated: 2026-01-29
+Last updated: 2026-02-05
 
 ## Discovery
 
 | Agent | Purpose | When to Use |
 |-------|---------|-------------|
 | `Explore` | Fast codebase exploration | Finding files by patterns, searching code for keywords, answering questions about Memory Loop structure |
 | `lore-development:surface-surveyor` | Entry point discovery | Finding where features start during excavation (e.g., tracing a mode from NavBar to View component) |
+| `lore-development:lore-researcher` | Searches lore frontmatter for related prior work | Before starting new specs or designs, to surface what's already been explored |
+
+## Documentation Review
+
+| Agent | Purpose | When to Use |
+|-------|---------|-------------|
+| `lore-development:spec-reviewer` | Fresh-context spec review for clarity issues | After completing a spec, when requirements feel ambiguous |
+| `lore-development:design-reviewer` | Fresh-context design review for weak decisions | After completing a design, when technical approach feels uncertain |
+| `lore-development:fresh-lore` | Fresh-context analysis from outside accumulated context | When too deep in the weeds to think clearly, need a second opinion |
 
 ## Architecture
 
 | Agent | Purpose | When to Use |
 |-------|---------|-------------|
-| `Plan` | Software architect for implementation design | Planning new features, choosing between approaches (REST vs WebSocket), architectural trade-offs |
+| `Plan` | Software architect for implementation design | Planning new features, choosing between approaches, architectural trade-offs |
 
 ## Code Quality
 
@@ -31,8 +40,7 @@ Last updated: 2026-01-29
 | `pr-review-toolkit:code-reviewer` | Project guideline adherence | Before commits, after implementing features |
 | `pr-review-toolkit:code-simplifier` | Simplifies code for clarity | Refactoring hooks, reducing component complexity |
 | `pr-review-toolkit:comment-analyzer` | Comment accuracy and maintainability | Reviewing JSDoc comments, protocol documentation |
-| `pr-review-toolkit:type-design-analyzer` | Type design analysis | Reviewing new Zod schemas, WebSocket message types |
-| `code-simplifier:code-simplifier` | Code clarity and consistency | Refining recently modified code |
+| `pr-review-toolkit:type-design-analyzer` | Type design analysis | Reviewing new Zod schemas, SSE event types |
 
 ## Testing
 
@@ -44,12 +52,12 @@ Last updated: 2026-01-29
 
 | Agent | Purpose | When to Use |
 |-------|---------|-------------|
-| `pr-review-toolkit:silent-failure-hunter` | Identifies silent failures | Reviewing WebSocket error handling, API fallbacks, catch blocks |
+| `pr-review-toolkit:silent-failure-hunter` | Identifies silent failures | Reviewing SSE streaming error handling, API fallbacks, catch blocks |
 
 ## Project-Specific Notes
 
 - **Testing is mandatory**: Always consult `pr-review-toolkit:pr-test-analyzer` before marking work complete
-- **Protocol changes**: When modifying `shared/src/protocol.ts`, use `pr-review-toolkit:type-design-analyzer` to review Zod schema design
-- **WebSocket handlers**: Consult `silent-failure-hunter` for any changes to `websocket-handler.ts` or error handling paths
+- **Schema changes**: When modifying `shared/src/types.ts`, use `pr-review-toolkit:type-design-analyzer` to review Zod schema design
+- **SSE streaming**: Consult `silent-failure-hunter` for any changes to SSE chat endpoint or error handling paths
 - **SDK integration**: The Claude Agent SDK provider pattern requires careful review; use `code-reviewer` for any `sdk-provider.ts` changes
-- **Monorepo context**: When exploring, remember the three workspaces (backend, frontend, shared) have distinct concerns
+- **Monorepo context**: Three workspaces (backend, nextjs, shared). Backend is a library consumed by Next.js API routes. No standalone server.

.lore/plans/daily-prep-system.md

@@ -2,7 +2,7 @@
 title: Daily Prep System Implementation Plan
 date: 2026-02-02
 status: executed
-tags: [skill-development, ui-component, rest-api, ground-tab]
+tags: [skill-development, ui, rest-api, ground-tab]
 modules: [daily-prep-manager, session-actions-card, home-view, daily-prep-skill, daily-debrief-command]
 related: [.lore/specs/daily-prep.md]
 ---

.lore/plans/vectorized-hopping-pinwheel.md → .lore/plans/nextjs-consolidation.md

@@ -1,7 +1,7 @@
 ---
 title: Consolidate to Single Next.js Application
 date: 2026-02-05
-status: complete
+status: executed
 tags: [migration, next-js, consolidation, maintainability]
 modules: [nextjs, backend]
 related:

.lore/plans/swirling-toasting-giraffe.md → .lore/plans/nextjs-test-coverage.md

@@ -1,3 +1,11 @@
+---
+title: "Next.js test coverage to 80%+"
+date: 2026-02-05
+status: executed
+tags: [testing, coverage, next-js, dom-environment, websocket-cleanup]
+modules: [nextjs]
+---
+
 # Plan: Next.js Test Coverage to 80%+
 
 ## Current State

.lore/plans/concurrent-imagining-flurry.md → .lore/plans/pairwriting-discussion-fix.md

@@ -1,3 +1,11 @@
+---
+title: Fix PairWriting actions not appearing in Discussion
+date: 2026-02-05
+status: executed
+tags: [bug-fix, pair-writing, discussion, sse, ref-forwarding]
+modules: [pair-writing, discussion, use-chat]
+---
+
 # Plan: Fix PairWriting Actions Not Appearing in Discussion
 
 ## Bug

.lore/plans/vi-mode-pair-writing.md

@@ -4,15 +4,15 @@ date: 2026-01-29
 status: executed
 tags: [vi-mode, pair-writing, modal-editing, keyboard]
 modules: [pair-writing-editor, use-vi-mode]
-related: [.lore/specs/vi-mode-pair-writing.md, .lore/research/vi-mode-implementation.md]
+related: [.lore/specs/vi-mode-pair-writing.md, .lore/_archive/vi-mode-implementation.md]
 ---
 
 # Plan: Vi Mode for Pair Writing
 
 ## Context
 
 - **Spec**: `.lore/specs/vi-mode-pair-writing.md`
-- **Research**: `.lore/research/vi-mode-implementation.md`
+- **Research**: `.lore/_archive/vi-mode-implementation.md`
 - **Integration target**: `PairWritingMode.tsx` and `PairWritingEditor.tsx`
 
 ## Approach