feat(collab-doc): server-side markdown↔Yjs conversion + agent-merge core (Stage A) by waleedlatif1 · Pull Request #6008 · simstudioai/sim

waleedlatif1 · 2026-07-28T18:08:24Z

Summary

Stage A of moving collaborative file docs to a server-authoritative Yjs model — the foundation that lets the server own the document, so the agent and the user write into one shared CRDT (markdown stays the source of truth, as a projection). This PR is the DOM-free, fully-unit-testable conversion + merge core; Stages B (persistence + seeding, deleting the client-seeder) and C (copilot into the doc) build on it.

Precedent: mirrors the pattern in inkeep/open-knowledge (Hocuspocus + @tiptap/y-tiptap + updateYFragment).

What's here

markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc in apps/sim/lib/collab-doc/.
Parity by construction: markdown↔ProseMirror reuses the exact client engine (@tiptap/markdown on the shared extensions), not a second impl — the server can't diverge from the editor. Round-trip test covers tables, footnotes, raw HTML, task lists.
Same binding as the browser: ProseMirror↔Yjs via @tiptap/y-tiptap (what TipTap Collaboration uses), pinned + peer-dep-shared prosemirror-model/yjs, targeting the 'default' fragment → structurally byte-compatible with the client.
Merge, not replace: applyMarkdownToYDoc uses updateYFragment (the primitive ySyncPlugin runs per keystroke); a test proves an agent write and a concurrent remote edit both survive.
Server-only; the markdown engine's editor gets a DOM from a lazily-required jsdom window (kept out of the client bundle).
Adds serializeDocToMarkdown as the single PM-JSON→markdown path; refactors serializeMarkdownBody onto it.

Type of Change

New feature (foundation / non-breaking)

Testing

4 converter tests (round-trip parity incl. custom-fidelity constructs, empty doc, agent write, concurrent-edit CRDT merge).
77 existing fidelity/parse tests green (the serializeMarkdownBody refactor is behavior-preserving).
apps/sim tsc 0 · biome clean.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Notes

Base is realtime-rooms (stacked), not staging — it depends on the file-doc collab infra there.
Not wired into any runtime path yet (no behavior change); Stages B/C do the wiring behind a flag. See apps/sim/lib/collab-doc/README.md.

… core (Stage A) Foundation for server-authoritative collaborative file docs. A DOM-free-testable module that converts a file's markdown to/from its collaborative Yjs document and merges agent writes into a live doc without clobbering concurrent typing. - markdownToYDoc / yDocToMarkdown / applyMarkdownToYDoc. - Parity by construction: the markdown<->ProseMirror step reuses the EXACT client engine (parseMarkdownToDoc / serializeDocToMarkdown via @tiptap/markdown on the shared extensions), not a second markdown impl — so the server can't diverge from the editor. Round-trip test covers tables, footnotes, raw HTML, task lists. - Same binding as the browser: ProseMirror<->Yjs via @tiptap/y-tiptap (what TipTap Collaboration uses), pinned + peer-dep-shared instances, targeting the 'default' fragment. Structurally byte-compatible with the client. - Merge, not replace: applyMarkdownToYDoc uses updateYFragment (the primitive ySyncPlugin runs per keystroke) so Yjs reconciles the agent write with in-flight remote edits. Test proves an agent write + a concurrent remote edit both survive. - Server-only; the markdown engine's editor gets a DOM from a lazily-required jsdom window (kept out of the client bundle). - Adds serializeDocToMarkdown to the client markdown module (single serialize path, reused by the server projection). Refactors serializeMarkdownBody onto it. Stages B (server-authoritative persistence + seeding, deleting the client seeder) and C (copilot into the doc + projection) build on this. See lib/collab-doc/README.md. Tests: 4 converter + 77 fidelity/parse suites green; sim tsc 0; biome.

vercel · 2026-07-28T18:08:29Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jul 28, 2026 8:10pm

cursor · 2026-07-28T18:08:35Z

PR Summary

Medium Risk
Changes the live collab cold-open path (realtime ↔ app internal API) and editor readiness/autosave gating; mis-seeding or auth wiring on the internal route could leave blank docs or block editing, though tests cover dedup, retries, and empty-file readiness.

Overview
Collaborative file docs now seed on the server instead of electing a client. On join, the realtime relay calls POST /api/internal/file-doc/seed (internal API key), applies the returned Yjs update (or sets initialContentLoaded for missing/empty files), and drops the old SEED_REQUEST flow, seeder election, seed deadlines, and in-room retry loops.

Adds @/lib/collab-doc: markdown ↔ Yjs using the same TipTap/markdown engine and @tiptap/y-tiptap as the browser (markdownToYDoc, yDocToMarkdown, applyMarkdownToYDoc with CRDT merge), plus buildFileDocSeed that loads workspace file bytes (frontmatter stripped) and encodes the seed with the seed flag. getWorkspaceFile gains throwOnError so DB failures are not treated as empty files.

Client side: FileDocProvider removes seed-request handling; readiness is synced && initialContentLoaded under a 12s READINESS_TIMEOUT (replacing connect-only timeout). Fatal fallback still renders stored markdown read-only and blocks relaying local writes. The rich markdown editor no longer imports content on the happy path—only on fatal join errors.

Infra: jsdom + @tiptap/y-tiptap as runtime deps; Next config traces jsdom for the seed route. Protocol docs updated; API contract baseline +1 route.

^{Reviewed by Cursor Bugbot for commit 0979739. Configure here.}

greptile-apps · 2026-07-28T18:11:51Z

Greptile Summary

Stage A server-authoritative collab-doc foundation: markdown↔Yjs conversion/merge core, internal seed API, and realtime relay seeding (replacing client seeder election).

Adds apps/sim/lib/collab-doc converter/seed helpers and unit tests.
Realtime file-doc joins fetch seeds via internal HTTP; client readiness becomes synced+seeded.
Production packaging: jsdom as a runtime dependency, serverExternalPackages, and seed-route outputFileTracingIncludes.

Confidence Score: 5/5

Safe to merge from a follow-up perspective; prior jsdom runtime/standalone findings are addressed and no eligible new inline issues remain.

Previous threads on jsdom as a runtime dependency and standalone tracing are reflected at HEAD (dependencies + serverExternalPackages + outputFileTracingIncludes for the seed route). The server-only marker thread was intentionally declined and is not a remaining defect. No incomplete or unsafe residual on those threads warrants a new comment.

Important Files Changed

Filename	Overview
apps/sim/lib/collab-doc/converter.ts	Server markdown↔Yjs via shared TipTap extensions and lazy jsdom; no new follow-up issues vs prior threads.
apps/sim/next.config.ts	Completes prior jsdom packaging fix with serverExternalPackages and seed-route tracing includes.
apps/sim/package.json	Moves jsdom to dependencies and adds @tiptap/y-tiptap for the converter stack.
apps/realtime/src/handlers/file-doc.ts	Replaces client seeder election with ensureServerSeed + fetchFileDocSeed; tests updated accordingly.
apps/sim/app/api/internal/file-doc/seed/route.ts	Internal seed endpoint gated on checkInternalApiKey; sole runtime importer of the converter path.

Sequence Diagram

sequenceDiagram
  participant Client
  participant Realtime as apps/realtime
  participant SeedAPI as POST /api/internal/file-doc/seed
  participant Converter as collab-doc converter
  Client->>Realtime: JOIN file-doc room
  Realtime->>SeedAPI: fetchFileDocSeed (x-api-key)
  SeedAPI->>Converter: markdownToYDoc / buildFileDocSeed
  Converter-->>SeedAPI: Yjs update (base64)
  SeedAPI-->>Realtime: "{ update }"
  Realtime->>Realtime: Y.applyUpdate + seed flag
  Realtime-->>Client: sync update (synced + seeded)

_{Reviews (8): Last reviewed commit: "fix(collab-doc): include jsdom in the Ne..." | Re-trigger Greptile}

buildFileDocSeed(workspaceId, fileId) → a Yjs update encoding the file's current markdown body, built through the Stage A converter (exact client engine). This is the server-authoritative seed source the realtime relay will apply on room create, replacing the client-seeder election. Frontmatter is stripped exactly as the client seed did (it's file metadata, not part of the collaborative body). Tests: seed round-trips to the file body through the client engine; frontmatter stripped; null for a missing file.

…ative seed source (Stage B) POST /api/internal/file-doc/seed (x-api-key, contract-bound): the realtime relay asks the app to build a file's collaborative-document seed (markdown -> Yjs, via the seed builder). Only the app can — it owns the editor extensions + blob access — so realtime delegates rather than embedding a second ProseMirror/markdown stack. Returns the Yjs update base64-encoded, or null for a missing file. Completes the app side of server-authoritative seeding. Bumps the api-validation route baseline (979 -> 980; contract-bound, so zodRoutes tracks). Realtime consumption + retiring the client seeder is the next step (behavior change, live-verified). Gates: api-validation, sim tsc 0, 7 collab-doc tests.

- Realtime relay fetches the initial doc from Sim (/api/internal/file-doc/seed) on first join and applies it authoritatively; deletes client seeder election, seed deadlines, and the SEED_REQUEST protocol event entirely - Seed builder marks initialContentLoaded in the same Yjs update so the client readiness gate (synced && seeded) needs no seed handshake - Client provider drops shouldSeed / seed-request; keeps synced + offline fallback - Rewrites realtime handler tests to cover server seeding; updates provider tests

The seed endpoint (/api/internal/file-doc/seed) calls markdownToYDoc -> ensureDomForTipTap -> require('jsdom') at runtime, so jsdom must survive devDependency pruning in production/standalone builds. Move it to dependencies.

waleedlatif1 · 2026-07-28T18:48:17Z

@greptile

waleedlatif1 · 2026-07-28T18:48:19Z

@cursor review

…indings Blocker: - Seed route used checkInternalAuth (Bearer-JWT only, forbids x-api-key) while the realtime relay authenticates with x-api-key: INTERNAL_API_SECRET — every seed 401'd and every collaborative doc hung in loading. Switch to checkInternalApiKey. Add a route test that exercises the real auth contract (the regression that was missing). Recovery (a null/failed seed left clients stuck once the client-seeder was removed): - getWorkspaceFile gains throwOnError so buildFileDocSeed distinguishes a genuinely absent file (null) from a transient read error (throws) — a DB blip can no longer be mistaken for an empty file and seeded as blank content over the real document. - ensureServerSeed: re-check the room is still live/empty/unseeded after the fetch (never touch a destroyed or client-seeded doc); mark an absent-file doc seeded so clients still reach readiness; retry a failed fetch a bounded number of times against the live room with backoff instead of waiting passively for a new join. - fetchFileDocSeed validates the response body and raises the timeout to fit the 5 MB max seed (a large valid file no longer perpetually times out). Docs/tests: - Rewrite the FILE_DOC_SEED consumer contract #3 to match reality (fatal->read-only on a cross-user client-id collision; same-user is auto-reclaimed) instead of an unimplemented re-mint claim. - Remove stale client-seeder comments (handoff, election, seed-handshake); correct the server-only docstring and the README staging. - Strengthen the concurrent-joiners test to exercise the in-flight guard; add empty-file, retry, and liveness tests; use FILE_DOC_SEED constants.

waleedlatif1 · 2026-07-28T19:16:19Z

@greptile

waleedlatif1 · 2026-07-28T19:16:20Z

@cursor review

The isDocEmpty guard skipped the seed when the doc's state vector was non-empty, but TipTap Collaboration writes a default empty paragraph into the local Y.Doc on mount, which syncs to the server before the seed fetch returns — so a fresh client left the doc "non-empty" but unseeded, the seed was skipped, initialContentLoaded never set, and the readiness gate never opened (stuck loading). Remove the state-vector emptiness guard: isDocSeeded alone is the correct "already handled" check, because content is only ever written alongside the seed flag (this seed or the client's offline fallback set both), so a doc with real content is always already seeded. A bare placeholder is unseeded and MUST still be seeded. Add a regression test.

waleedlatif1 · 2026-07-28T19:25:09Z

@greptile

waleedlatif1 · 2026-07-28T19:25:10Z

@cursor review

…laceholder) The earlier comments implied a fresh client syncs its empty-paragraph placeholder into the room doc ahead of the seed. @tiptap/y-tiptap deliberately does NOT — its sync plugin diffs against createAndFill() and writes nothing when the doc is unchanged from empty — so an unseeded doc is genuinely empty and the seed applies with no merge/duplication. Correct the ensureServerSeed rationale and reframe the defensive regression test accordingly.

waleedlatif1 · 2026-07-28T19:36:38Z

@greptile

waleedlatif1 · 2026-07-28T19:36:39Z

@cursor review

A persistent server-side seed failure (build exhausts its retries) left synced clients blank and read-only forever: the connect deadline was cleared on `synced`, and the offline fallback only ran on a fatal join-error — neither covered "synced but never seeded". Extend the client's single deadline from "connected" to "ready" (synced AND seeded): it now also observes the seed flag and, if readiness isn't reached in time, resets `synced`, latches fatal, and surfaces the same non-retryable join-error the offline path uses — so the editor shows the file's stored content read-only instead of a blank pane. Also gate outbound doc updates on `fatal` so a fatal client's local read-only render never relays unseeded content to peers. Add tests for the synced-but-not-seeded fallback.

waleedlatif1 · 2026-07-28T19:50:15Z

@greptile

waleedlatif1 · 2026-07-28T19:50:16Z

@cursor review

Cut the removed-isDocEmpty archaeology and the y-tiptap internals deep-dive from the shipped comment (both live in git history / the test) — keep the essential what + why.

…deadline The in-room seed retry (3x backoff, up to ~60s) outlasted the client's 12s readiness deadline: once a client gave up and latched fatal, a late server-side success could no longer reach it — the retry actively worked against the recovery it claimed to provide. Replace it with a single seed attempt bounded by an 8s fetch timeout (comfortably under the 12s client readiness deadline, and generous for a real ≤256KB collab seed — the 5MB cap is unreachable since collab is gated at 256KB client-side). On failure the guard is released so the next join re-attempts; a persistent failure lapses the client into its read-only fallback. Removes the retry loop, backoff helpers, and the now-vestigial fake timers from the test. Simpler and with coherent timescales.

waleedlatif1 · 2026-07-28T20:01:54Z

@greptile

waleedlatif1 · 2026-07-28T20:01:55Z

@cursor review

Moving jsdom to dependencies was necessary but not sufficient: the seed converter's lazy require('jsdom') is invisible to Next's standalone file tracer, so a Docker/standalone build omitted jsdom and the seed endpoint 500'd with MODULE_NOT_FOUND (rooms then fell to the client read-only fallback). Add jsdom to serverExternalPackages (webpack keeps it external instead of bundling its dynamic internal requires) and force it into outputFileTracingIncludes for the seed route so the standalone build actually copies it. Matches how ws/sharp are handled.

waleedlatif1 · 2026-07-28T20:10:34Z

@greptile

waleedlatif1 · 2026-07-28T20:10:35Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0979739. Configure here.}

waleedlatif1 requested a review from a team as a code owner July 28, 2026 18:08

greptile-apps Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/sim/lib/collab-doc/converter.ts

Comment thread apps/sim/lib/collab-doc/converter.ts

vercel Bot deployed to Preview July 28, 2026 18:12 View deployment

waleedlatif1 added 2 commits July 28, 2026 11:17

vercel Bot deployed to Preview July 28, 2026 18:26 View deployment

vercel Bot temporarily deployed to Preview July 28, 2026 18:45 Inactive

fix(collab-doc): make jsdom a runtime dependency

4c02e73

The seed endpoint (/api/internal/file-doc/seed) calls markdownToYDoc -> ensureDomForTipTap -> require('jsdom') at runtime, so jsdom must survive devDependency pruning in production/standalone builds. Move it to dependencies.

vercel Bot deployed to Preview July 28, 2026 18:51 View deployment

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/realtime/src/handlers/file-doc.ts

Comment thread apps/realtime/src/handlers/file-doc.ts

vercel Bot temporarily deployed to Preview July 28, 2026 19:16 Inactive

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/realtime/src/handlers/file-doc.ts Outdated

vercel Bot temporarily deployed to Preview July 28, 2026 19:25 Inactive

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/realtime/src/handlers/file-doc.ts Outdated

Comment thread apps/sim/lib/collab-doc/seed.ts

vercel Bot temporarily deployed to Preview July 28, 2026 19:36 Inactive

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread ...ace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/rich-markdown-editor.tsx

vercel Bot temporarily deployed to Preview July 28, 2026 19:50 Inactive

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread ...paceId]/files/components/file-viewer/rich-markdown-editor/collaboration/file-doc-provider.ts

docs(collab-doc): trim the ensureServerSeed docblock

8dd728c

Cut the removed-isDocEmpty archaeology and the y-tiptap internals deep-dive from the shipped comment (both live in git history / the test) — keep the essential what + why.

vercel Bot temporarily deployed to Preview July 28, 2026 19:57 Inactive

vercel Bot temporarily deployed to Preview July 28, 2026 20:01 Inactive

greptile-apps Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/sim/lib/collab-doc/converter.ts

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/realtime/src/handlers/file-doc-seed.ts

vercel Bot temporarily deployed to Preview July 28, 2026 20:10 Inactive

cursor Bot reviewed Jul 28, 2026

View reviewed changes

waleedlatif1 merged commit ba72663 into realtime-rooms Jul 28, 2026
5 checks passed

waleedlatif1 mentioned this pull request Jul 28, 2026

feat(collab-doc): stream copilot edits into the live document (Stage C) #6021

Merged

Uh oh!

Conversation

waleedlatif1 commented Jul 28, 2026

Summary

What's here

Type of Change

Testing

Checklist

Notes

Uh oh!

vercel Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

waleedlatif1 commented Jul 28, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jul 28, 2026 •

edited

Loading

cursor Bot commented Jul 28, 2026 •

edited

Loading

greptile-apps Bot commented Jul 28, 2026 •

edited

Loading