Defer typecheck driver translation to first typecheck (halve warm boot)#29
Merged
Conversation
typecheck_native.install did ~47ms of work on every boot — the single biggest cost in load_kernel — translating the 16 t-star.kl typecheck driver defuns through the KL->Lua prolog compiler and loadstring'ing the result. A plain boot never typechecks anything, so none of that is needed until the first (shen.typecheck ...) call. Move the parse+translate (the ~32ms CPU cost, plus install_handports) into a memoized ensure_drivers() that typecheck_dispatch calls on the first typecheck. The t-star.kl source READ stays eager: read_kl resolves against the relative P.KLDIR, and a caller can chdir between boot and its first typecheck (run-kernel-tests does exactly this), so a deferred read could fail to find the file — we slurp the string now and defer only the cwd-insensitive work. Harvest and the declare/destroy wrappers also stay eager since they must observe kernel declares issued by initialise(). Warm cached boot drops ~0.10s -> ~0.045s; cold ~0.95s -> ~0.73s. The first typecheck absorbs the deferred ~40ms once. Port specs (453) and the kernel certification suite (35 reports, 100%) stay green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fp6poWsabXMkruseeVihxx
initialise() spends ~19ms running the KL environment/lambda-form/signed- func setup cold (one-shot, so the JIT can't amortize it). Profiling shows it calls `hash` ~800 times — every `put` into the property vector hashes its symbol key — and the compiled-KL hash is expensive per call: explode the key into a character list, map string->n over it, then fold with shen.prodbutzero via KL recursion. Add a native `hash` in install_native_stdlib that folds the key's bytes directly (symbols by name, strings by content; both are byte-based here), reproducing prodbutzero's multiply-then-add-past-1e10 overflow guard and the mod-0->1 rule exactly. Verified identical to the compiled-KL hash over every kernel F-name x 5 bounds plus edge strings (empty, embedded NUL, UTF-8, overflow-triggering lengths): 0 mismatches. Non-symbol/-string values delegate to the original so their printed form is untouched. initialise ~17ms -> ~10ms; warm cached boot ~0.045s -> ~0.037s. Because hash underlies get/put and the whole property system, runtime dict-heavy code (types, source lookup) benefits too. Adds golden-value hash tests pinning the native fold to the kernel semantics. Port specs (459) and the kernel certification suite (35 reports, 100%) stay green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fp6poWsabXMkruseeVihxx
Addresses a review of the lazy-translation change: - Guard the exported entry: M.native_typecheck now calls ensure_drivers() before native_typecheck, so a direct caller (embedder/test) that reaches it before any typecheck dispatch translates the drivers instead of reading nil predicates out of NP and crashing. The internal dispatch path is unchanged (it already calls ensure_drivers and uses the raw local). - Make ensure_drivers commit-at-end: translate into fresh accumulators and set drivers_ready (and the n_ok/n_fail the dispatch consults) only after the full pass AND install_handports() succeed. If R.read_all or an unexpected translator error throws partway, drivers_ready stays false so the next typecheck retries cleanly rather than dispatching over a half-filled NP. Re-running is safe: the driver source is fixed (idempotent NP writes) and the counters are local, not accumulated across attempts. - Add test/typecheck_lazy_spec.lua: asserts drivers are deferred at boot (n_ok == 0), the direct native entry is guarded (no crash, triggers translation), and drivers are translated exactly once. Wired into scripts/run-tests.lua. Full port suite green: 467 pass, 0 fail across 12 specs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
typecheck_native.install did ~47ms of work on every boot — the single
biggest cost in load_kernel — translating the 16 t-star.kl typecheck
driver defuns through the KL->Lua prolog compiler and loadstring'ing
the result. A plain boot never typechecks anything, so none of that is
needed until the first (shen.typecheck ...) call.
Move the parse+translate (the ~32ms CPU cost, plus install_handports)
into a memoized ensure_drivers() that typecheck_dispatch calls on the
first typecheck. The t-star.kl source READ stays eager: read_kl resolves
against the relative P.KLDIR, and a caller can chdir between boot and its
first typecheck (run-kernel-tests does exactly this), so a deferred read
could fail to find the file — we slurp the string now and defer only the
cwd-insensitive work. Harvest and the declare/destroy wrappers also stay
eager since they must observe kernel declares issued by initialise().
Warm cached boot drops ~0.10s -> ~0.045s; cold ~0.95s -> ~0.73s. The
first typecheck absorbs the deferred ~40ms once. Port specs (453) and
the kernel certification suite (35 reports, 100%) stay green.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Fp6poWsabXMkruseeVihxx