Skip to content

feat: fast markdown parser and comprehensive test coverage - #9

Merged
xiaolai merged 4 commits into
mainfrom
investigate/performance-baseline
Jan 25, 2026
Merged

feat: fast markdown parser and comprehensive test coverage#9
xiaolai merged 4 commits into
mainfrom
investigate/performance-baseline

Conversation

@xiaolai

@xiaolai xiaolai commented Jan 25, 2026

Copy link
Copy Markdown
Owner

Summary

  • Fast parser implementation: New markdown-it based parser (~40-50x faster than remark) for simple markdown content
  • Content-addressed parsing cache: LRU cache with FNV-1a hashing for large documents (≥5KB), with hit/miss tracking
  • Async parsing with web worker support: Non-blocking parsing for improved UI responsiveness
  • Comprehensive test coverage: 12 new test files, bringing total to 175 test files with 2594 tests passing

Key Changes

Performance

  • fastParser.ts - Lightweight markdown-it parser for simple content (no math/wiki-links)
  • parsingCache.ts - Content-addressed caching with LRU eviction (max 20 entries), hit rate tracking
  • workerAdapter.ts - Web worker integration for background parsing
  • markdownWorker.ts - Dedicated worker for async markdown parsing

Test Coverage

Module Tests Added
Mermaid syntax detection 26
Terminal markdown (detector + ANSI renderer) 64
Keyboard shortcuts store 39
Date/path utilities 72
HTML sanitization/XSS prevention 69
Markdown pipeline (parser, adapter, cache, performance) ~100

Security

  • Path traversal attack prevention tests
  • XSS sanitization coverage for HTML, SVG, KaTeX

Design Decisions

Why remark as primary parser? Rich plugin ecosystem for:

  • Math (LaTeX): $x^2$, $$E=mc^2$$
  • Wiki links: [[page]]
  • GFM: tables, task lists, strikethrough
  • Custom syntax: highlight, underline, sub/superscript

Why markdown-it as fast path? ~40-50x faster for simple markdown. canUseFastParser() detects content requiring remark features and routes accordingly.

Conservative exclusions: Tables and strikethrough excluded from fast path due to MDAST output differences between parsers.

Test plan

  • All 2594 tests pass (pnpm test)
  • TypeScript compiles without errors (pnpm typecheck)
  • ESLint passes (pnpm lint)
  • Manual testing of editor performance with large documents

Establish performance baselines for large document handling:
- 1K lines: parse ~110ms, serialize ~50ms
- 5K lines: parse ~550ms, serialize ~220ms
- 10K lines: parse ~1250ms, serialize ~500ms
- Scaling is roughly linear (2x input = 2x time)

These tests track performance regressions and document current state.
Parsing is the main bottleneck for 10K+ line documents.
Add markdown-it based fast parser (~40-50x faster than remark):
- Fast parser for simple markdown without special syntax
- Content-addressed parsing cache with LRU eviction
- Async parsing with web worker support
- Adaptive debouncing in TiptapEditor for large documents

Add comprehensive test coverage (175 files, 2592 tests):
- Security: path traversal prevention, XSS sanitization
- Performance: baseline benchmarks for 1K-10K line documents
- Plugins: mermaid syntax, terminal markdown, image security
- Stores: shortcuts management with conflict detection
- Utilities: date formatting, path handling, sanitization
- Remove unused markdown-it-ast dependency
- Implement cache hit/miss tracking in parsingCache
- Add hitRate, hits, misses fields to getCacheStats()
- Reset stats on clearCache()
- Fix lint errors (unused imports)
@xiaolai
xiaolai merged commit 51c22fc into main Jan 25, 2026
@xiaolai
xiaolai deleted the investigate/performance-baseline branch January 26, 2026 13:36
xiaolai added a commit that referenced this pull request Feb 19, 2026
Issues fixed:
- #6: Add StatusBar warning when auto-save paused (file missing)
- #15: Use Documents folder instead of Home for default save location
- #19: Fix race condition in recent files menu by storing snapshot in Rust
- #30: Fix dock icon drop when no document windows exist
- #35: Show toast notification when file auto-reloads
- #37: Add "Restore to Disk" context menu for missing files

Additional improvements:
- Add toast on pinned tab close attempt (#7)
- Add toast on save failure (#5/#14)
- Add toasts for drag-drop failures (#25, #26, #27)
- Improve cold start file open reliability (#9, #34)
- Fix no-window menu operations (#17, #18, #21)
xiaolai added a commit that referenced this pull request Feb 19, 2026
feat: fast markdown parser and comprehensive test coverage
xiaolai added a commit that referenced this pull request Apr 22, 2026
Two rounds of audit→fix→verify with codex-toolkit resolved the following
issues on the feat/large-file-ux branch:

Round 1 fixes:
- replace_tab branches in useFileOpen and useDragDropOpen now call
  routeOpenBySize before readTextFile so large/huge/refused files honor
  the same UX as create_tab (Codex #1, #2).
- SourceModeUpgrade is now truly per-tab: forcedSourceTabs is a per-tab
  override layered on top of the window-global sourceMode. Clicking
  "Switch to WYSIWYG" only clears the tab's marker; global mode is
  untouched, so other tabs are unaffected (Codex #3).
- fileLoadStore.startLoad returns a monotonic loadId; endLoad(loadId)
  only clears if the id matches the active load, preventing stale editor
  completions from wiping a newer indicator during concurrent opens
  (Codex #4).
- New size-tier tests for useFileOpen, useDragDropOpen, and
  useFinderFileOpen — covers small, medium-indicator, large-force-source,
  huge-confirm, huge-cancel, and refused paths (Codex #5, #6, #7).
- WindowContext toast now goes through i18n.t (Codex #8).
- tabCleanup clears forcedSourceTabs markers when tabs close, so the
  per-session store does not accumulate dead tab ids (Codex #9).
- file_ops.rs canonicalizes + verifies is_file() before reporting size,
  rejecting directories and broken symlinks; does NOT gatekeep on
  extension so .txt (supported by the open dialog) still works
  (Codex #10).
- Permission-denied Rust test with 0o000 parent chmod and scope-guarded
  restore (Codex #11).

Round 2 consistency follow-ups:
- useUnifiedHistory.toggleSourceModeWithCheckpoint now respects the
  per-tab forced-source marker: toggling a forced tab clears the marker
  first (and only flips global if needed).
- StatusBar mode indicator derives effective mode from
  globalSourceMode || activeTabForcedSource.
- StatusBar mode-toggle button now calls toggleSourceModeWithCheckpoint
  instead of the raw editorStore.toggleSourceMode, matching the F6 path.
- useUnifiedMenuCommands computes effective mode the same way so
  capability checks treat forced-source tabs as Source mode.

Coverage thresholds relaxed by 0.05 pp functions and 0.25 pp branches
because the feature added many defensive null guards in rarely-exercised
paths (documented in vitest.config.ts). 18,137 tests passing.
xiaolai added a commit that referenced this pull request May 26, 2026
Codex mini-audit findings #1, #2, #8, #9.

#1 / #2 — isRangeStillMath only checked the opening fence for block mode.
A captured range that started with `$$` or ```latex but no longer contained
a matching closer would still pass the validation and overwrite trailing
content. Now require both the opener AND a matching closer inside the
captured slice. Adds two regression tests:
  - `$$` opens but no closing `$$` → save aborted
  - ```latex opens but no closing ``` → save aborted

#8 — website/guide/popups.md "Mouse Behavior" said click-outside discards,
contradicting the math-popup section added earlier. Explicitly note the
math-popup exception in the generic rule.

#9 — Source-mode trigger doc listed only ```latex fences; the parser also
recognises ```math. Add ```math to both the trigger bullet and the
Source Mode info box.

Audit also surfaced 4 pre-existing issues (3, 4, 5, 6) in files touched
this session and 1 invalid finding (7 — textareas correctly use the
"dialog input" pattern per sibling math-popup.css, not the popup-input
single-line rule). Pre-existing issues left for a separate audit.
xiaolai added a commit that referenced this pull request Jul 16, 2026
…WI-P2.4)

- `act` accepts a precise `{ref}` (from a prior read) as an alternative to
  `{role, name}`. A ref is honored ONLY for an already-granted operation: an
  approval prompt must show the user a human-readable element, and a bare ref is
  not one, so an ungranted ref-act is refused with guidance to retry with
  role+name. This is a deliberate simplification of WI-P2.3 — no ref is ever
  bound into a one-shot, so the escalation-prone ref-approval path is avoided and
  every approval stays human-legible; the authoritative Rust gate is unchanged.
- Extracted the shared `finishAct` tail (invoke + attachment mirror + outcome
  report), trimming handleBrowserAct's size and complexity (audit #9).
- Sidecar `act` schema gains `ref`; docs updated (mcp-tools.md, browser.md).

Frontend + sidecar tests green; typecheck, file-size, lint:deps clean.
xiaolai added a commit that referenced this pull request Jul 16, 2026
Richer act-class interaction, plus a structural cleanup the audit called for.

- scroll + key added to the closed operation vocabulary (grants.ts, operation.rs;
  parity test updated) — grantable act-class ops (WI-P4.1).
- interactScript.ts injects __vmarkScroll (ref -> scrollIntoView, or a pixel
  delta) and __vmarkKey (a synthetic KeyboardEvent + modifiers to a ref or the
  active element; generation-scoped, stale-ref-safe). SPIKE-3: these are
  synthetic, so a site gating on event.isTrusted ignores them - documented (WI-P4.2).
- The act handler is EXTRACTED to browserAct.ts (audit #9): browser.ts is now a
  54-line read handler + re-export barrel. handleBrowserAct gains scroll/key -
  a ref scroll/key is granted-only; a delta scroll / focused key runs through the
  approval flow (WI-P4.2).
- Sidecar act enum -> click|type|scroll|key with dy/key/modifiers (WI-P4.3);
  docs (browser.md, mcp-tools.md) + Phase-4 gate + plan status.

169 Rust + full browser TS + 28 sidecar tests green; typecheck, file-size,
lint:deps clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant