Skip to content

v4.3.13

Choose a tag to compare

@aovestdipaperino aovestdipaperino released this 10 May 14:31
· 243 commits to master since this release

Changed

  • Switched the markdown grammar to tree-sitter-grammars/tree-sitter-markdown (block + inline split parsers). The previously-vendored ikatyang/tree-sitter-markdown had been frozen since 2023 and parsed YAML frontmatter as ambiguous markdown, causing GLR fork explosions on otherwise-fine files containing ---\n…\n--- envelopes with deep indentation. A real-world 18 KB resume.md hung the indexer indefinitely. The new grammar treats frontmatter as opaque (minus_metadata) / (plus_metadata) nodes and parses the same file in ~16 ms. Closes #61.

Added

  • Per-file extraction timeout. Every extractor round trip is now wrapped in a watchdog. A file whose extractor doesn't respond in time has its worker subprocess killed and is recorded in SyncResult.skipped_paths with reason "extractor timed out (>Ns)". Worker crashes (the existing failure path) are now also recorded with reason "extractor crashed (...)" instead of disappearing silently. Configurable via extraction_timeout_secs in ~/.tokensave/config.toml (default 60 s). tokensave sync can no longer hang forever on a single malformed file.