Skip to content

fix(editor): restore inline date pills on the collaborative open path; legible fired reminders - #1866

Merged
h4yfans merged 3 commits into
mainfrom
fix-date-pill-roundtrip
Aug 27, 2026
Merged

fix(editor): restore inline date pills on the collaborative open path; legible fired reminders#1866
h4yfans merged 3 commits into
mainfrom
fix-date-pill-roundtrip

Conversation

@h4yfans

@h4yfans h4yfans commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

What changes for the user

An inline reminder pill written into note body text survives a restart, a vault switch or a rebuild on a new device. Before this, the pill came back as a two-hundred-character base64 run — the date it stood for was unreadable, which is what the reporter hit.

A fired reminder pill now reads as a spent chip: muted text at full opacity, a settled background, a dimmed alarm icon. It keeps its date and time on the page as a record of what was scheduled, and is visually distinct from an armed pill (blue, alarm) and from a date-only pill (muted, no fill, no alarm). Red is left to mean broken.

A date token the parser refuses now degrades to a plain, readable date instead of staying an opaque blob.

Root cause

Not the escape the issue proposed. Two probes against the real converter say so: a realistic payload never reaches the base64 symbols base64url spells -/_ (the JSON is printable ASCII, and only >, ? or ~ can drive base64 onto them — no field can hold one), and when a token is forced to contain _, this serialization path emits it bare rather than escaped.

The actual break is the collaborative open path. Main parses the vault file into the shared Y.Doc with a dateMention spec whose parse claims a data-date-mention element, and markdown has none, so the token reaches the doc as plain text. useEditorSync's load effect returns early once a Y.Doc fragment is bound, so normalizeNoteBlocks — and every promoter in it — runs on the markdown path only. Nothing turned the token back into a pill. Wiki links (#1642) and inline checkboxes already close this exact gap; a date token is worse off than either, because its text carries no readable fallback at all.

promoteDateMentionsInSharedDoc closes it the same way.

Compat plan

  • Token bytes are unchanged. The alphabet is closed at the same time — the encoder spells the two odd base64 symbols , and ; rather than base64url's - and _, because _ opens emphasis and sits on remark-stringify's unconditional escape list for phrasing content. No realistic payload reaches those symbols, so every token already on disk re-encodes to identical bytes. date-mention.test.ts asserts byte-for-byte equality against the old writer over 200 generated payloads. The write-back byte compare therefore rewrites nothing.
  • Both alphabets parse, and a stray \ inside the run is stripped, so any token already written with an escape is healed on read.
  • Graceful null is preserved. parseDateMentionToken still returns null for pre-remind-enum tokens. The degrade is a separate salvageDateMentionToken, and it restores remind: 'none' only — those tokens have armed nothing since the enum landed, so nothing is resurrected.
  • Bytes with no recoverable date are left exactly as they are. An unreadable token beats a destroyed one.
  • Promotion is byte-neutral. A promoted node re-serializes to the token it came from, so a second open writes no CRDT update and "opening a note must not rewrite it" (test(notes): opening a note must not rewrite it — byte stability for wiki links and custom blocks #1434) holds. A salvaged token does change the bytes, once, on a note that was already unreadable.
  • Fired state stays per-device and presentational. It is still applied as a data-fired DOM attribute by use-triggered-date-pills.ts and is never written into pill props or markdown.

Verification

  • pnpm lint clean. typecheck:node / typecheck:web / typecheck:test clean. pnpm typecheck fails only on a pre-existing architecture-boundary violation in apps/mobile/vitest.config.ts, which this branch does not touch.
  • packages/shared/src/date-mention.test.ts — 22 passed. Covers the closed alphabet, byte identity against the old writer, a legacy base64url token with _/- runs, a backslash-escaped token, and the salvage.
  • apps/desktop/src/main/sync/blocknote-converter.test.ts — 247 passed with src/main/notes/note-date-reminders.test.ts. New: a dateMention node serializes to exactly the token the renderer's promoter reads, emits only [A-Za-z0-9,;], hands an un-promotable token back byte-identical, and drops a stray escape rather than carrying it.
  • date-mention-collab-promotion.test.ts — 6 passed. Real BlockNoteEditor on the real schema, real Yjs collaboration, the real hook. Covers the current token, the legacy one, the escaped one, the degrade, the untouched-bytes case, and "promotes once, writes nothing on a second open".
  • Full renderer content-area suite — 792 passed, 6 skipped, 62 files.
  • pnpm docs:impact --base origin/main --strict green, pnpm docs:build green, git diff --check clean.

The full conformance suite is #1848's; the link-mention half is #1844's, and link-mention.ts is untouched here.

Closes #1845

An inline reminder pill turned into its own base64 after a restart or a
vault switch. Main parses the vault file into the shared Y.Doc with a
`dateMention` spec whose `parse` claims a `data-date-mention` element,
which markdown has none of, so the token reaches the doc as plain text --
and `useEditorSync`'s load effect returns before `normalizeNoteBlocks` on
that path, so nothing promoted it back. Wiki links and inline checkboxes
already close the same gap; date mentions now do too. Byte-neutral for a
well-formed token, so a second open writes no CRDT update.

The token alphabet is closed at the same time. base64url's `_` opens
emphasis and sits on remark-stringify's unconditional escape list for
phrasing content, so the encoder now spells the two odd base64 symbols
`,` and `;`, which no markdown writer escapes. Real payloads are
printable-ASCII JSON and never reach those symbols, so every token
already on disk re-encodes to identical bytes and no note is rewritten.
The parser accepts both alphabets and strips a stray `\` escape.

A token the parser refuses now degrades to a plain, readable date instead
of staying an opaque blob. Bytes with no recoverable date are left alone.

Fired reminder pills read as a spent chip -- muted text at full opacity,
a settled background, a dimmed alarm -- rather than as an error in red.

Closes #1845
@github-actions github-actions Bot added bug Something isn't working documentation Improvements or additions to documentation test labels Aug 27, 2026
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

React Doctor found no new issues. 🎉

Reviewed by React Doctor for commit 4afe775.

Comment thread apps/desktop/src/main/sync/blocknote-converter.test.ts Fixed
Comment thread packages/shared/src/date-mention.test.ts Fixed
Comment thread packages/shared/src/date-mention.test.ts Fixed
Comment thread packages/shared/src/date-mention.test.ts Fixed
…replace

CodeQL reads a `replace` that prefixes a backslash as an escaping routine
and files js/incomplete-sanitization against it. These are fixtures that
deliberately corrupt a base64url run, not sanitizers, and a named helper
says so.
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@h4yfans
h4yfans merged commit 3ae78da into main Aug 27, 2026
19 checks passed
@h4yfans
h4yfans deleted the fix-date-pill-roundtrip branch August 27, 2026 23:08
h4yfans added a commit that referenced this pull request Aug 27, 2026
#1866 gave `((date:…))` a promoter on the collaborative path, so the list of
nodes that reach the shared doc as text and get rebuilt there is one longer.
Names hash tags as the one that stays text, and why.
h4yfans added a commit that referenced this pull request Aug 27, 2026
The hostile anchor bytes that forced base64url emphasis runs now assert
the opposite property: the fixed encoder emits no emphasis character at
all, so a future alphabet reopen fails the corpus loudly.
h4yfans added a commit that referenced this pull request Aug 27, 2026
#1866 gave `((date:…))` a promoter on the collaborative path, so the list of
nodes that reach the shared doc as text and get rebuilt there is one longer.
Names hash tags as the one that stays text, and why.
h4yfans added a commit that referenced this pull request Aug 27, 2026
The hostile anchor bytes that forced base64url emphasis runs now assert
the opposite property: the fixed encoder emits no emphasis character at
all, so a future alphabet reopen fails the corpus loudly.
h4yfans added a commit that referenced this pull request Aug 28, 2026
The hostile anchor bytes that forced base64url emphasis runs now assert
the opposite property: the fixed encoder emits no emphasis character at
all, so a future alphabet reopen fails the corpus loudly.
h4yfans added a commit that referenced this pull request Aug 28, 2026
#1866 gave `((date:…))` a promoter on the collaborative path, so the list of
nodes that reach the shared doc as text and get rebuilt there is one longer.
Names hash tags as the one that stays text, and why.
h4yfans added a commit that referenced this pull request Aug 28, 2026
…e reload (#1867)

* fix(editor): close the link-mention token alphabet so mentions survive reload

A link mention persists as literal `((mention:<encoded-url>))` text inside a
paragraph, so it has to survive both the markdown writer and the markdown
reader. Two separate holes meant it did not.

The collaborative load path never rebuilt the chip. Main seeds the shared
Y.Doc straight from the vault file, where the mention is plain text, and the
Yjs branch of `useEditorSync` returns before `normalizeNoteBlocks` runs. Wiki
links and inline checkboxes each have a promoter on that path; mentions had
none. A mention therefore lived only as long as its Y.Doc and came back as
literal text after a restart or a vault switch.

The token's alphabet was open. `encodeURIComponent` leaves `- _ . ! ~ * ' ( )`
raw, and remark-stringify does not escape them inside the token — so two
mentions on one line whose URLs each hold a `*` (or each a `~`) are read back
as a single emphasis or strikethrough run spanning from the first token into
the second, and both mentions are destroyed.

Encoding `! ' ( ) * ~ _` closes the payload to `[A-Za-z0-9.%-]`, which nothing
downstream can reinterpret. `MENTION_TOKEN_REGEX` widens from `[^)\s]+` to
`[^)\n\r]+` — a strict superset, so today's tokens match with identical
captures — and `parseLinkMentionToken` keeps its original code path for any
payload without whitespace or a backslash, adding a repair path that strips
those and requires a parseable URL before accepting the result. Tokens already
mangled on disk heal on open; tokens that are fine keep their exact bytes.

* docs(architecture): note the collaborative-path promoters and the mention token alphabet

* fix(editor): guard the mention pattern against a broken token upstream

Review of the widened payload class turned up a case where it loses a mention
the strict pattern found. Stopping only at `)` runs the match straight through
an unterminated `((mention:` fragment earlier on the line and captures it
together with the next real token, so a note damaged by this very bug — which
is where those fragments come from — would lose a mention that works today:

  x ((mention:foo ((mention:https%3A%2F%2Fx.test)) y
  strict          -> https%3A%2F%2Fx.test
  stop-at-paren   -> foo ((mention:https%3A%2F%2Fx.test

Excluding `(` restores the strict pattern's captures. No serializer has ever
emitted a raw paren in a payload, so nothing real is refused.

The repair path now also requires what it repaired to be spellable in the
token alphabet. `new URL()` alone was not the gate the comment claimed: with
the spaces stripped out, `((mention: see http://x.com))` parses as a URL and
became a chip pointing at `seehttp://x.com`.

Favicon hydration is dropped from the collaborative path. It writes back a
content array captured before its fetch, which on a shared document pushes a
stale block to every device, and it resolves after `clearYjsUndoHistory`, so
opening a note left an undoable step — the invariant the surrounding code
exists to hold. A chip promoted there renders the domain the token carries.
Hydrating a shared document safely is its own change.

* docs(notes): say what a link mention shows after a restart

* docs(architecture): fold the date pill into the promoter list

#1866 gave `((date:…))` a promoter on the collaborative path, so the list of
nodes that reach the shared doc as text and get rebuilt there is one longer.
Names hash tags as the one that stays text, and why.
h4yfans added a commit that referenced this pull request Aug 28, 2026
The hostile anchor bytes that forced base64url emphasis runs now assert
the opposite property: the fixed encoder emits no emphasis character at
all, so a future alphabet reopen fails the corpus loudly.
h4yfans added a commit that referenced this pull request Aug 28, 2026
…zers (#1874)

* test(editor): golden round-trip conformance suite across both serializers

One corpus in @memry/editor-schema/conformance, asserted byte-for-byte by
both pipelines: the renderer save path (real BlockNoteEditor + the real
editor schema, markdown-utils + normalize-note-blocks) and the main/CRDT
converter (markdownToYFragment -> yDocToMarkdown). Each case asserts
round-trip identity and second-pass idempotence against the same
canonical bytes, so cross-serializer agreement is a tested equality.

Covers the payload table of #1848: mention URLs with markdown-significant
characters, date tokens with base64url emphasis runs, all four callout
types plus foreign passthrough / titles / multi-paragraph bodies, toggles
empty / nested / blank-lined / fenced / imaged / unterminated, and tokens
inside table cells, list items and toggle bodies — plus a deterministic
seeded fuzz layer over the token alphabets (no property-testing dep in
the workspace).

Cases broken on current main are marked it.fails pending their sibling
fix (#1844 #1846 #1847, plus #1865 found by this suite), so the sibling
landing turns the inverted expectation red and forces the flag off.

Golden vault gains four roundtrip-*.md fixtures, byte-gated by the
existing frontmatter suite and now also pushed through the converter.

New renderer-side counter reports tokens the normalize chain left
unclaimed through app_error_seen (leading emit, then one aggregated
event per kind per minute). Metric only, no user-facing surface.

* docs(observability): document the unclaimed-token counter

* test(editor): keep pending buckets valid once every sibling fix lands

it.each on an empty list throws, so the it.fails buckets are registered
only while pending cases exist; conformance internals stop being exported.

* test(editor): deepen the fuzz corpus with whole-document mixing

New mixed-documents family assembles notes from every token family plus
lists, paragraphs and gaps. It immediately caught #1877 — blank-line
gaps adjacent to a toggle collapse on write-back — now pinned as a
static pending case; the family itself joins with single gaps and never
puts two lists across one blank line (CommonMark fuses those: accepted
canonicalization, not token damage).

* test(telemetry): cover string-shaped inline content in the token walk

* test(editor): adapt the date corpus to the closed token alphabet (#1866)

The hostile anchor bytes that forced base64url emphasis runs now assert
the opposite property: the fixed encoder emits no emphasis character at
all, so a future alphabet reopen fails the corpus loudly.

* test(editor): assert the fixed callout behavior landed by #1875

Foreign passthrough, titles after the marker and the four claimed types
now round-trip on both pipelines as plain assertions. The two blank-`>`-
line shapes #1875 explicitly declined re-anchor to #1881: the quote path
collapses blank quote lines on ANY blockquote, callout or not, merging
quoted paragraphs on write-back.

* test(editor): assert persisted toggle fold state landed by #1876

Expanded toggles (the new open-attribute on-disk form) join the corpus,
the golden fixture and the fuzz roll. The unterminated shape #1876 did
not reach re-anchors to #1883: the declined region's raw-HTML lines are
still dropped by the markdown parser on both pipelines.

* test(editor): assert the closed mention alphabet landed by #1867

Every mention payload and the mention fuzz family now assert fixed
behavior on both pipelines. The renderer's rich render-in-a-cell rewrite
was not in #1867's scope and joins its wiki-link twin on #1865.
h4yfans added a commit that referenced this pull request Aug 28, 2026
The spec still asserted the pre-#1866 triggered red (#e56458). PR #1866
deliberately restyled fired pills to muted-foreground with a settled fill,
and E2E only runs on push to main, so the stale assertion first surfaced
there. Assert the new contract: muted color plus a background fill that a
date-only pill does not have.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inline reminder pill breaks into literal text; fired reminders become unreadable

2 participants