Skip to content

fix: re-read files from disk on every compilation - #39

Open
msallin wants to merge 1 commit into
evolvedlight:developfrom
swisspost:fix/reset-file-slots
Open

fix: re-read files from disk on every compilation#39
msallin wants to merge 1 commit into
evolvedlight:developfrom
swisspost:fix/reset-file-slots

Conversation

@msallin

@msallin msallin commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes #28.

Problem

SlotCell::accessed was set on first access and never cleared, so a compiler kept its main file and every import pinned to whatever they contained during the first compilation. Rewriting a .typ on disk and compiling again returned the old output, with no error.

Caching a compiler is what the README recommends, so anyone who follows that advice and also redeploys templates gets silently stale documents.

Change

SystemWorld::reset (previously reset_time) now marks every file slot as not yet accessed before each compilation, the way typst-cli does between watch runs, in addition to clearing the cached date/time.

  • The fingerprint check is untouched: a file whose content has not changed is re-read but not decoded again, so an unchanged document stays cheap to recompile.
  • A SlotCell filled by init_in_memory is pinned. TypstCompiler.FromSource has no file behind <main>, so invalidating it would make the second compilation fail. The exemption is per cell, not per file slot, so a document passed as a string still re-reads its imports.
  • Package files are reset along with everything else. A versioned package cannot legitimately change under its version, but a deployment that vendors its templates as local packages redeploys them exactly like a bare .typ — which is the case this fixes. Same reasoning for retrying a package that failed to resolve.
  • The reset takes the slot mutex instead of reaching past it with Mutex::get_mut. Every first access to a file inserts into that map and can reallocate it, so iterating it unlocked would turn the documented one-compiler-per-thread rule from a stale-read hazard into a dangling iterator.

Consequences worth a decision

Both are documented in the README and the release notes, but they are behaviour changes rather than pure fixes:

Templates must be replaced atomically. A compilation landing halfway through a plain overwrite renders the half-written file. Before, that window was unreachable for a warm compiler; now it is open on every request. Write-temp-then-rename is the fix on the deploying side.

A warm compile now costs one stat and one read per file it touches, and scales with file count. Measured on Windows (release build, warm page cache, 200 compilations of one compiler), per compilation:

.typ files before after
1 64 µs 233 µs
6 471 µs 1.16 ms
21 386 µs 2.45 ms
51 387 µs 5.34 ms

Roughly 99 µs per file per compilation, all of it the forced read — an isolated benchmark of fs::metadata + fs::read + hash128 over the same files gives 97.4 µs. That figure is inflated by on-access AV scanning and would be much lower on Linux, but against the README's 3 ms warm compile it is not nothing for a template split across many files.

Correctness by default seemed the right trade given the issue, but if you would rather have an opt-out, an FFI invalidate_files(compiler) plus a managed InvalidateFiles() would let a server with its own deploy hook pay only when something actually changed — closer to what typst-cli does, since its resets are driven by a watcher rather than by each render. Happy to add it here or as a follow-up.

Unrelated build fix

tests/input_path.rs did not compile on develop: the exported functions became unsafe in #36, after those tests were written in #37, so cargo test failed with seven E0133 errors and the native test step in pack.yml has been red. The calls are now wrapped in unsafe blocks. Worth a look at why that went unnoticed.

Tests

tests/file_reload.rs (8 tests): rewritten main file, rewritten import, rewritten data file read through read(), a file repaired on disk after a failed compilation, a deleted file reported as missing, an in-memory document compiled twice, an in-memory document whose import is rewritten, and an in-memory source shadowing a file it names. Six of the eight fail on develop; the two in-memory ones guard against the reset going too far.

world.rs gained white-box tests for SlotCell itself: that a reset sends the cell back to the file but decodes only what changed, that a file which did not exist yet is picked up, and that an in-memory value is left alone.

Tests.cs covers the end-to-end cases: a template, an import, and a vendored local package rewritten underneath a reused TypstCompiler.

cargo test and dotnet test both pass: 29 native, 59 managed, no warnings.

Follow-ups I did not fold in

  • slot() holds the mutex across read, decode_utf8 and package resolution, so a panic in any of them poisons it and every later call fails with a PoisonError that masks the real error. get_or_init is self-healing after such a panic, so unwrap_or_else(PoisonError::into_inner) is the right recovery — but that is a pre-existing bug in slot(), not something this change introduces.
  • The FFI test boilerplate now exists in three copies across compile.rs, input_path.rs and file_reload.rs; a tests/common/mod.rs would be worth it.
  • README.md recommends caching a compiler in ASP.NET while lib.rs documents that one compiler must not be used from two threads, and TypstCompiler has no synchronisation. The two should be reconciled.

A compiler filled its file slots on first access and never invalidated them, so
the main file and every import stayed pinned to whatever they contained during
the first compilation. Keeping one compiler alive across renders is the
recommended way to use the library, and a template redeployed underneath a
running process went on rendering the retired content with no error to show for
it.

Every compilation now starts by marking the slots as unread, the way typst-cli
does between watch runs. A file whose content has not changed is still
recognised by its hash and is not parsed again, and a document handed over as a
string keeps its content: there is no file behind it to read.

The reset takes the slot mutex rather than reaching past it with get_mut. The
map is grown by every first access to a file, so iterating it unlocked would
turn the documented one-compiler-per-thread rule from a stale-read hazard into a
dangling iterator.

Also wraps the calls in tests/input_path.rs in unsafe blocks. The exported
functions became unsafe after those tests were written, so the test crate no
longer compiled and the native test step failed.
@msallin
msallin force-pushed the fix/reset-file-slots branch from f92ccf1 to 04780a4 Compare September 6, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A reused TypstCompiler never picks up changes to files on disk

1 participant