feat: remediate plan 071 audit findings — data integrity, harness, UI/UX

plans/ADRs/027-canonical-state-and-p2p-sync-bridge.md

@@ -0,0 +1,138 @@
+# ADR 027 — Canonical State and P2P Synchronization Bridge
+
+**Date**: 2026-07-19  
+**Status**: Accepted  
+**Related**: ADR 018, ADR 026, Plan 071  
+**Supersedes**: ADR 026 statements that make Yjs primary or Zustand read-only
+
+## Context
+
+ADR 018 establishes Zustand plus `localStorage` as the canonical local-first
+persistence layer. ADR 026 later describes Yjs/IndexedDB as primary and Zustand
+as a read-only view. The implementation currently follows neither complete
+model:
+
+- normal CRUD writes to Zustand only;
+- joining sync copies a snapshot from Zustand into Yjs;
+- no production lifecycle subscribes Yjs changes back into Zustand;
+- deletions have no tombstone/version semantics;
+- manual conflict choices are dismissed without being applied.
+
+This split ownership creates false-success states and makes it unclear which
+copy wins after reconnect, reload, or concurrent deletion.
+
+## Decision
+
+### 1. Zustand remains canonical application state
+
+Zustand is the only state read and mutated by product features. Its validated,
+versioned persisted subset remains the local source of truth. Yjs is an opt-in
+replication transport and collaboration log, not a second canonical database.
+
+This preserves ADR 018, avoids rewriting every feature around CRDT types, and
+keeps the app fully useful when collaboration is disabled or unavailable.
+
+### 2. One lifecycle owner bridges canonical state and Yjs
+
+The sync session owns one bidirectional adapter:
+
+```text
+validated Zustand transaction
+          │
+          ▼
+   outbound Yjs update
+          │
+          ▼
+  WebRTC / IndexedDB replica
+          │
+          ▼
+ validated inbound projection
+          │
+          ▼
+atomic Zustand transaction
+```
+
+The adapter must:
+
+1. seed an empty sync document from canonical state on first join;
+2. subscribe to canonical create/update/delete operations while connected;
+3. subscribe to Yjs changes and validate them before store mutation;
+4. tag transaction origins to prevent echo loops;
+5. commit batches atomically so history and persistence observe one change;
+6. unsubscribe and release resources when the session ends.
+
+### 3. Deletes are versioned operations
+
+Absence is not interpreted as deletion. Entities and claims use tombstones (or
+an equivalent explicit delete operation) containing record ID, deletion time,
+device identity, and logical/version ordering information. Tombstones are
+retained long enough for offline peers to converge and are compacted only under
+a separately tested retention rule.
+
+Concurrent edit-versus-delete behavior must be deterministic. The default is
+delete-wins when the deletion is causally later; unresolved concurrent changes
+are surfaced as a conflict rather than silently resurrecting data.
+
+### 4. Conflict resolution is a canonical transaction
+
+Manual choices are validated, applied to the merged record, persisted to
+Zustand, and propagated to Yjs before the dialog reports success. Dismissal
+without applying a choice does not mutate data or claim resolution.
+
+### 5. Sync status describes canonical commit state
+
+“Synced” means all acknowledged inbound changes passed validation and committed
+to canonical state, and all local operations were submitted to the active Yjs
+document. Transport connection alone is not sufficient.
+
+## Consequences
+
+### Positive
+
+- Local-only behavior remains simple and independent of sync infrastructure.
+- Every feature observes one application state model.
+- Peer data crosses a runtime-validation boundary before persistence.
+- Create, update, delete, reload, and conflict behavior can be integration
+  tested deterministically.
+- The false distinction between two primary stores is removed.
+
+### Negative
+
+- The adapter must maintain origin metadata and guard against update loops.
+- Tombstones require retention and compaction policy.
+- Whole-record Zustand updates cannot expose character-level collaborative
+  editing; that would require a future editor-specific ADR.
+- IndexedDB Yjs state is a replica and must be reconciled against the canonical
+  persisted schema during upgrades.
+
+## Alternatives Considered
+
+1. **Make Yjs canonical and Zustand read-only.** Rejected for the current phase:
+   it contradicts ADR 018 and requires broad application/store migration.
+2. **Keep manual snapshot synchronization.** Rejected: it cannot provide
+   ongoing collaboration and encourages false-success UI.
+3. **Mirror writes independently in every store action.** Rejected: lifecycle,
+   error handling, and echo prevention would be scattered across the product.
+4. **Treat absence as deletion.** Rejected: offline merging resurrects deleted
+   records or deletes records a peer has not received yet.
+
+## Implementation Requirements
+
+- Add a single sync-session bridge at the collaboration boundary.
+- Reuse complete Zod record schemas for inbound peer validation.
+- Add explicit canonical batch actions rather than mutating store internals.
+- Define tombstone schema, ordering, retention, and compaction.
+- Apply conflict selections before clearing conflict state.
+- Do not add an offline queue claim until convergence and retry semantics are
+  implemented.
+
+## Verification
+
+- Two independent documents/stores converge after create, update, and delete.
+- Offline peer reconnect converges without resurrection.
+- Concurrent different-field edits merge as specified.
+- Edit-versus-delete follows the documented deterministic rule.
+- Invalid peer records never enter canonical state.
+- Manual conflict selection produces the selected field values on both peers.
+- Disconnect/reconnect does not duplicate observers or operations.
+- Reload restores canonical state and reconnects the replica without data loss.

plans/ADRs/028-validated-recoverable-local-data-boundaries.md

@@ -0,0 +1,134 @@
+# ADR 028 — Validated and Recoverable Local Data Boundaries
+
+**Date**: 2026-07-19  
+**Status**: Accepted  
+**Related**: ADR 010, ADR 018, Plan 071
+
+## Context
+
+The studio is local-first, so browser persistence and user-controlled export
+files are the durability boundary. Current implementation has two unsafe paths:
+
+1. persisted Zustand hydration trusts unknown data through a no-op migration
+   cast;
+2. JSON import uses shallow guards, silently filters invalid records, and then
+   immediately replaces canonical entities and claims.
+
+Complete Zod schemas already exist for core records. The missing decision is how
+strictly to validate, what to do with partially invalid data, and how users
+recover from destructive replacement.
+
+## Decision
+
+### 1. Validate every external state boundary
+
+The following inputs are untrusted and must be parsed with versioned Zod
+schemas before they can mutate canonical state:
+
+- `localStorage` hydration;
+- JSON and encrypted archive import;
+- future Markdown import metadata;
+- Yjs/peer projections;
+- AI/provider and web-research responses where persisted.
+
+Type assertions and shallow record filters do not satisfy this boundary.
+
+### 2. Imports are atomic and fail closed
+
+An import payload is accepted only when its declared version is supported and
+all records and references required by that version are valid. Invalid records
+are not silently dropped. Parsing produces either:
+
+- a complete validated candidate plus a preview summary; or
+- a structured error list with record paths and no state mutation.
+
+Best-effort salvage is a separate explicit recovery workflow, never the default
+import behavior.
+
+### 3. Replacement requires preview and recovery
+
+Before replacement, the UI shows entity/claim/link counts, detected conflicts,
+schema version, and the consequence that current data will be replaced. A
+validated pre-import snapshot is created before canonical commit.
+
+Replacement is one atomic store transaction. If commit or persistence fails,
+the previous snapshot is restored. The user can also explicitly undo the most
+recent replacement until a later destructive operation supersedes that
+recovery point.
+
+### 4. Hydration uses versioned migrations
+
+Persisted state has an explicit schema version and a migration chain. Hydration:
+
+1. reads the raw persisted envelope;
+2. validates the envelope/version;
+3. applies known migrations sequentially;
+4. validates the resulting current schema;
+5. hydrates canonical state atomically.
+
+If recovery is possible, valid data is exposed through an explicit recovery
+path. Unknown future versions or unrecoverable corruption do not enter the
+store. The application offers reset and raw-export options without logging
+personal content.
+
+### 5. Referential integrity is part of validation
+
+- Claims must reference an imported/existing entity according to import mode.
+- Entity links must target a valid entity unless the schema explicitly permits
+  external references.
+- IDs are unique within each collection.
+- dates, enums, confidence ranges, and version fields are validated.
+- deletion removes or resolves dependent links and claims atomically.
+
+## Consequences
+
+### Positive
+
+- Malformed archives cannot silently destroy valid local data.
+- Schema evolution is explicit and testable.
+- Import errors can identify exact invalid paths.
+- Sync and hydration can reuse the same canonical validators.
+- Recovery behavior matches the importance of local-only user data.
+
+### Negative
+
+- Strict import rejects partially useful legacy files until a dedicated
+  recovery tool exists.
+- Snapshot retention consumes temporary local storage.
+- Migration and rollback tests become mandatory for every persisted schema
+  change.
+- Import UI gains a preview/confirmation step.
+
+## Alternatives Considered
+
+1. **Continue filtering invalid records.** Rejected: silent data loss is worse
+   than an actionable failed import.
+2. **Validate only TypeScript shapes.** Rejected: TypeScript types do not exist
+   at runtime.
+3. **Always merge imports into current data.** Rejected as a default: ID,
+   deletion, and relationship conflicts need explicit semantics.
+4. **Keep an automatic JSON stringify clone as backup.** Rejected: use a
+   validated snapshot and `structuredClone` for in-memory copies.
+5. **Reset automatically on hydration failure.** Rejected: local-first data
+   should not disappear without an explicit recovery choice.
+
+## Implementation Requirements
+
+- Define the current persisted-state envelope schema next to canonical record
+  schemas.
+- Reuse schemas for export, import, hydration, and sync boundaries.
+- Reject unsupported versions before reading records.
+- Build preview data from the validated candidate only.
+- Add one atomic replacement action and one bounded recovery snapshot.
+- Keep raw recovery export local; never log user content or secrets.
+
+## Verification
+
+- Invalid field, enum, date, confidence, duplicate ID, dangling relation, and
+  unsupported-version fixtures leave canonical state byte-for-byte unchanged.
+- Valid export/import round-trip preserves all supported canonical data.
+- A forced persistence failure restores the prior snapshot.
+- Known old persisted versions migrate deterministically and idempotently.
+- Unknown future and corrupt persisted versions present recovery choices.
+- Entity deletion removes dependent claims and incoming/outgoing links in one
+  undoable transaction.