Implement cached (§15) - #18
Merged
Merged
Conversation
`cached <expr>` evaluates an expression once and reads the answer back on
every later run. §15 had pinned the cache *key* — the expression treated as a
function, hashed as one — but left "where does the cache actually live" as an
open TODO, and two of the digests it rests on did not exist.
**The key.** A closure now hashes as its code (the AST subtree, structurally,
so reformatting or commenting it changes nothing), its captured `ctx`, and the
values of the free names it uses. `#arg`/`#self` are dynamic lookups that no
closure captures, so `cached` mixes in the stack entries the expression can
reach, bounded statically by the largest N written in it — without that,
`let f func (cached (#arg + 1))` answers `f 10` with `f 1`'s result.
**The layout**, resolving §15's TODO. One entry per key, in ctx.cache's
directory:
<cache>/sha256-<key> a File value, stored as itself
<cache>/sha256-<key>.hb/value.hb anything else, as HashedBuild text
<cache>/sha256-<key>.hb/sha256-<h> each File inside it, by content hash
A file stays a file and a directory stays a directory, so what a build produced
is still something you can open. The text format is HashedBuild's own value
syntax read by a separate literals-only parser, not `import`, so a hand-edited
entry is a parse failure rather than code that runs. Entries commit by rename,
which is also how two runs racing on one key settle it.
**Two digests that had to exist first**, both settled with the language's
owner rather than assumed:
- §3's directory hash, over entries sorted by name. Only Linux can report an
executable bit; WASI and Windows hash every entry as non-executable, so a
tree containing an executable hashes differently there. SPEC.md §3 now says
so, and says why the alternatives were worse. This was the open question
that kept directory hashing unbuilt.
- `ctx.cache`'s, as a tagged constant rather than its path — it is in every
key now, and baking the path in would invalidate a cache that was moved.
Six operations join the fs layer for this (listing by descriptor with
no-follow classification, mkdir, rename, unlink, rmdir, and the exec bit),
implemented on all three targets.
LANGUAGE.md gains a Caching section and loses `cached` and all three
unhashable kinds from "what isn't built yet"; examples/cached.hb is asserted
by the suite like every other example.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A directory entry now hashes as name and content, with no permission input on any target. §3 previously kept a single executable flag — the same reduction git makes (`100644` vs `100755`) — and hashed it where it could be read. Only Linux can read it. WASI's filestat carries no permission bits and Windows has no POSIX execute bit, and neither has a stand-in worth using: an `.exe` extension is a different question, and what MSYS guesses from a shebang is a guess. So hashing it made one tree two values depending on where it was checked out — and not hypothetically. Git sets `core.fileMode=false` on Windows, so the bit round-trips through the repository without ever existing in the working tree; the same commit checks out executable on Linux and not on Windows. This repository has four such files, so `sha256 loadfile "scripts"` would have answered two ways. Git's own way out is to *remember* the bit rather than re-derive it, which a `File` cannot do — it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, this takes the second: two trees differing only in an executable bit are one value, and the platform a build runs on no longer changes what its inputs hash to. The bit is still read and still preserved when `cached` copies a tree, so caching a build output doesn't silently strip it. That is fidelity in the store, not identity in the language, and the comments now say so. Covered by a Linux-only test that sets the bit and asserts the digest is unchanged — Linux-only because elsewhere fs_set_executable_at is a no-op, and the test would pass without establishing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`sha256 <function>` and `sha256 <directory>` both work as of the `cached`
work, and LANGUAGE.md described them in prose — but nothing in examples/
exercised either, and the Hashing section's snippet block still only showed a
string, a file and a Table. A feature nobody outside this repo can run is not
finished.
examples/hashing.hb gains three entries, each asserting a property rather than
a digest so they stay meaningful:
- whitespace and comments are not part of a function's code;
- the values it captures *are* — the property `cached` rests on, since
without it one entry would serve every argument;
- a directory is its own kind of value, not its contents run together.
LANGUAGE.md's snippet block gains the two new forms, and the Function bullet
gains the pair of one-liners above. Every snippet in it was run before it was
written down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#17 landed directory, closure and ctx.cache hashing while this branch was building the same three as scaffolding for `cached`. Its versions win — they are merged, they are better factored, and decisively they handle **cyclic values**, which #16 made constructible and which mine would have recursed on forever. Dropped: hash_directory.odin, my hash_function.odin, my hash.odin changes, my Fs_Entry extension and fs_list_dir_at, my hash tests, and my additions to examples/hashing.hb (three dedicated hashing examples cover it better). cache_store.odin now reads directories through main's fs_list_entries_at, whose getdents walk is better than the /proc/self/fd one it replaces, and refuses a fifo/socket/device the same way §3's hash does. Kept, because main has none of it: - `cached` itself, rebound to value_digest(v, interp) and Hash_Fail. - Four fs write operations — mkdir/rename/unlink/rmdir, on all three backends. #17 added reading; the store needs writing. - The `#arg`/`#self` coverage, now hash_implicit.odin. main's free_names collects identifiers and a uses_ctx flag, but nothing looks at Implicit_Name or Hole — they are dynamic lookups no closure captures, so `let f func (cached (#arg + 1))` would answer `f 10` with 2. **Cyclic values are cacheable**, which the format could not previously express. A Table reached more than once is written `node "N" { … }` at its first occurrence and `ref "N"` after that; the reader creates each Table before reading its entries, exactly as §10's evaluation order does, so a definition always precedes its references and nothing needs patching up. A restored cycle is bisimulation-equal to the stored one, which is what §6 requires. Labels go on merely shared Tables too — not needed for correctness, since §6 compares structurally, but it stops a shared value expanding exponentially on the way out. 230 tests pass; all three targets typecheck. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverses the resolution #17 recorded earlier the same day. A directory entry now hashes as name and content, with no permission input on any target. #17 kept a single executable flag — the same reduction git makes — read where a target can see one and false elsewhere, and cited `core.filemode` as taking the same position. Looking at what git actually *does* rather than what it records reverses that. Git never re-derives the bit on a target that cannot report one: `core.fileMode=false` on Windows means it carries the mode from the index, so a committed `100755` round-trips through the repository without ever existing in the working tree. The same commit is executable on Linux and not on Windows. So hashing the bit made one tree two values depending on where it was checked out — and not hypothetically. This repository has four `100755` files, so `sha256 loadfile "scripts"` would have answered two ways. Remembering the bit instead, as git does, is not available to a `File`: it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, §3 now takes the second. The bit is still read and still restored when `cached` copies a tree, so caching a build output doesn't silently strip it — fidelity in the store, not identity in the language, and the comments say so. hash_linux_test.odin's assertion inverts: it now sets the bit, confirms it actually landed (otherwise the check would hold for the wrong reason), and asserts the digest is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cached (§15), and the hashing it rests oncached (§15)
Caching a value that reaches itself is a user-visible capability and LANGUAGE.md describes it, but nothing in examples/ ran one. cached.hb now caches a self-referential Table and asserts the digest survives the round trip - which is the whole test, since §6 compares cyclic values by bisimulation, so a back-edge that came back as an unfolding of the wrong depth would not compare equal to what was stored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two failures CI caught and this machine cannot, both from the four fs write operations the cache needs. **wasi-threads**: `cached.hb` died with "could not publish cache entry (Access)" under iwasm. preview1 checks a rename against two separate rights — PATH_RENAME_SOURCE on the source descriptor and PATH_RENAME_TARGET on the destination's — and DIR_RIGHTS asked for neither, so every commit-by-rename came back ENOTCAPABLE, which fs.odin folds into a bare .Access. **playground**: the browser refused to instantiate the module at all — `LinkError: "path_rename": function import requires a callable`. docs/wasi.js implements preview1 for the playground, and nothing had needed to remove or rename anything before, so three imports simply weren't there. Added path_rename, path_unlink_file and path_remove_directory, plus the FileSystem `remove`/`rename` they sit on, plus the arity entries a spawned thread marshals them through. `rename` refuses an existing destination, which is what makes it the cache's commit rather than an overwrite. ENOTEMPTY (55) joins the ERRNO table, since an rmdir of a non-empty directory now has a way to say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cached <expr>evaluates an expression once and reads the answer back on everylater run. §15 had pinned the cache key — the expression treated as a
function, hashed as one — but left
> TODO: Where does the cache actually liveopen. This closes it.
The key
mainhashes a closure as its body's shape plus its captures, which is exactlyright for a Function value — but
#arg/#selfare dynamic lookups into theinterpreter's stacks, captured by no closure and invisible to
free_names.Left there,
let f func (cached (#arg + 1))givesf 10the answer2— awrong value, not a stale cache.
cachedmixes in the stack entries theexpression can reach, bounded statically by the largest
Nwritten in it(
hash_implicit.odin). There is a regression test.The layout
In
ctx.cache's directory (--cache-dir, else the XDG/per-user default), oneentry per key:
A file stays a file and a directory stays a directory, so what a build produced
is still something you can open,
diffor copy out. Anything else is writtenas HashedBuild's own value syntax, read back by a literals-only parser rather
than
import— a hand-edited entry is a parse failure, not code that runs.Entries commit by rename, which is also how two runs racing on one key settle
it without locking.
Cyclic values are cacheable. A Table reached more than once is written
node "N" { … }at its first occurrence andref "N"after that:The reader creates each Table before reading its entries — exactly as §10's
evaluation order does — so a definition always precedes its references and
nothing needs patching up afterwards. A restored cycle is bisimulation-equal to
the stored one, which is what §6 requires of it. Labels go on merely shared
Tables too: not needed for correctness, since §6 compares structurally, but it
stops a shared value expanding exponentially on the way out.
Also here
mkdir/rename/unlink/rmdir, on all threebackends. Finish hashing: directories, closures, cycles, and ctx.cache #17 added reading; the store needs writing.
resolution Finish hashing: directories, closures, cycles, and ctx.cache #17 recorded. Finish hashing: directories, closures, cycles, and ctx.cache #17 cited
core.filemodeas agreeing; git actuallydoes the opposite — it never re-derives the bit on a target that cannot report
one, because
core.fileMode=falsecarries the mode from the index, so acommitted
100755round-trips without ever existing in a Windows workingtree. Hashing it made one tree two values depending on where it was checked
out: this repo has four such files, so
sha256 loadfile "scripts"answeredtwo ways. Remembering the bit instead isn't available to a
File— a livehandle with no index beside it. The cache still restores the bit when copying
a tree: fidelity in the store, not identity in the language.
Testing
230 tests pass, repeatedly, on Windows; all three targets typecheck.
examples/cached.hbis asserted by the suite like every other example.Two things I could not run locally, both covered by CI:
wasm-ldon this machine, so the backends aretypechecked for
wasi_wasm32but not linked;hash_linux_test.odin, whose assertion inverts with the exec-bit change:it now sets the bit, confirms it landed, and asserts the digest is unchanged.
🤖 Generated with Claude Code