Finish hashing: directories, closures, cycles, and ctx.cache by janstrakowski · Pull Request #17 · janstrakowski/hashed

janstrakowski · 2026-08-31T09:52:15Z

sha256 now answers for every value. The three kinds LANGUAGE.md listed as
unbuilt each needed a language decision before any code, and each was taken
deliberately rather than inferred from what was easy to implement.

The four decisions

A directory File hashes over its entries per §3 — name plus content hash
and executable bit for a regular file, the child's own directory hash for a
sub-directory, the target string for a symlink (never followed), sorted by
name. The open question was the executable bit, which WASI and Windows cannot
report: unobservable now means not executable, so a tree containing an
executable hashes differently there than on Linux. §3 stays as written, and a
Windows checkout genuinely has no exec bits — the same position git takes with
core.filemode. Reporting one where none can be seen would be inventing data.
An entry that is none of the three shapes fails rather than being skipped.

This is the one digest that does I/O, so it happens at the first demand and
needs ctx.permissions.io at that moment, then is memoised — §3 calls a File
an immutable handle, so its digest must not change under a program. That forced
a small piece of design: values_equal is reached from table_find on the hot
path of every field access and can do no I/O, so the evaluator warms both
operands first (hash_materialize).

A Function is its body's shape plus the values it captures. Free-variable
analysis is what makes that useful: a closure's digest survives an unrelated
binding changing nearby, which is what §15's cached will need — a cache key
that moved whenever a neighbour changed would miss every time. The encoding is
of the program, not its meaning: renaming a local is a different closure, and
alpha-equivalence is not promised.

A cyclic value goes through SCC decomposition: everything outside a cycle
is folded as usual, and nodes inside one get a canonical form — bisimulation
classes, numbered by a deterministic walk from the node itself. This has to
agree with values_equal, which already compares cycles by bisimulation, or
equal values would have two content addresses.

ctx.cache hashes as a bare tag. The only thing that would tell two apart
is the directory it is rooted at, which is exactly the path §9 keeps out of the
language.

Implementation notes

One new fs operation, fs_list_entries_at, on all three backends: a
descriptor's entries with their kind and executable bit. On Linux it lists
through a fresh descriptor so getdents cannot disturb a handle the program is
still using as a .dir.

sha256_tagged now streams rather than concatenating, so hashing a large value
no longer copies it — the digests are unchanged, which the pinned SHA-256 test
vector confirms.

One bug found and fixed while testing: the fold and the graph algorithm ordered
a closure's captures differently (by name vs. by name digest), so the same
closure would have had two digests depending on whether something else in the
value looped. Not observable from inside the language, but it breaks §6's "one
encoding". test_the_two_hash_paths_agree_on_acyclic_values pins it, and was
confirmed to fail when the fix is reverted.

Tests

The second commit exercises the canonical form on the recursive structure it
was designed for — molecules, where a ring is a cycle in the literal sense.
Benzene (six interchangeable carbons, one class), cyclohexane and the Kekulé
drawing (same skeleton, different digests), toluene (a substituent splits the
ring into six positions), the three xylenes (positional isomers all separate),
and naphthalene vs azulene (two interlocking cycles in one component;
naphthalene's five classes come out exactly right).

Two tests pin limitations rather than capabilities, both properties of the
model rather than the encoding: a perfectly symmetric ring cannot count
itself (benzene and cyclopropenyl share a digest — and values_equal agrees,
which is what makes it correct), and reflection is invisible because
bisimulation matches Table entries by key. This is node-identity rather than
graph-identity, which is the question sha256 <atom> actually asks.

Docs

SPEC.md records the four resolutions. LANGUAGE.md loses the hashing entry
from "what isn't built yet" and gains three sections with snippets that were
run before being written down. Three new examples, asserted by the suite.

Verified locally

odin test src (212 passed), odin build src, scripts/build_wasi.sh, and
odin check -target:linux_amd64 — the last because the executable-bit test is
Linux-only, since Linux is the only target that has one. Left to CI: that test
actually running, the threaded WASI build (needs clang), the wasmtime and iwasm
smoke tests, and the playground.

🤖 Generated with Claude Code

`sha256` now answers for every value. The three kinds LANGUAGE.md listed as unbuilt each needed a language decision first, and each was taken: **A directory File** hashes over its entries per SPEC.md §3 - name plus content hash and exec bit for a regular file, the child's own directory hash for a sub-directory, the target string for a symlink, sorted by name. The open question was the exec bit, which WASI and Windows cannot report: unobservable now means *not executable*, so a tree containing an executable hashes differently there than on Linux. §3 stays as written, and a Windows checkout genuinely has no exec bits - the same position git takes with core.filemode. An entry that is none of the three shapes fails rather than being skipped. This is the one digest that does I/O, so it happens at the first demand and needs ctx.permissions.io at that moment, then is memoised - a File is an immutable handle (§3), so its digest cannot change under a program. The evaluator warms both operands before a comparison (hash_materialize), since values_equal is reached from table_find and can do no I/O there. **A Function** is its body's shape plus the values it captures. Free-variable analysis is what makes that useful: a closure's digest survives an unrelated binding changing nearby, which is what §15's `cached` will need. The encoding is of the program, not its meaning - renaming a local is a different closure. Builtins hash as the operation they are, and carry what they were partially applied to. **A cyclic value** goes through SCC decomposition: everything outside a cycle is folded as usual, and nodes inside one get a canonical form of the cycle - bisimulation classes, numbered by a deterministic walk from the node itself. That has to agree with values_equal, which already compares cycles by bisimulation, or equal values would have two content addresses. **ctx.cache** hashes as a bare tag. The only thing that would tell two apart is the directory it is rooted at, which is exactly the path §9 keeps out of the language. One new fs operation, `fs_list_entries_at`, on all three backends: a descriptor's entries with their kind and exec bit. sha256_tagged now streams rather than concatenating, so hashing a large value no longer copies it. SPEC.md records the four resolutions; LANGUAGE.md's "what isn't built yet" loses the hashing paragraph and gains three sections with runnable snippets; three new examples, asserted by the suite. The exec bit is tested on Linux only, because it is the only target that has one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rec_build_test.odin exercises the canonical form on two-node toys. A ring of atoms is a cycle in the literal sense, so molecules push it much harder - and being real structures, they say something about the encoding rather than just covering lines. Five of them, each for a different shape of cycle: * **benzene** - six interchangeable carbons, one bisimulation class. Every carbon hashes alike, which is entry-point independence at its starkest. * **cyclohexane and the Kekule drawing** - the same six-carbon ring with different atoms and bonds. Three rings, three digests: the encoding sees what is in the cycle, not only its shape. * **toluene** - one methyl splits the ring into six positions by distance from it, so the refinement has to find six classes where benzene had one. * **the three xylenes** - same formula, methyls 1,2- / 1,3- / 1,4-. All three separate, and para finds its own two-fold rotation. * **naphthalene and azulene** - both C10H8, a ten-carbon perimeter plus one chord, differing only in where the chord lands. Two interlocking cycles in one component; naphthalene's five classes come out exactly right, and azulene's lopsided fusion correctly has no rotation to find. Two tests pin limitations rather than capabilities, because both turned up while writing the others and both are properties of the *model*, not bugs: * A perfectly symmetric ring cannot count itself. Benzene and the cyclopropenyl cation are the same repeating unit at different periods, so bisimulation calls them one value - and values_equal agrees, which is what makes the shared digest correct rather than a collision. Any substituent recovers the ring size, as the toluene test shows. * Reflection is invisible. m-xylene's two methylated carbons are equivalent to a chemist, and hash apart, because bisimulation matches Table entries by key and a reflection maps `.ring` onto an incoming bond. Listing both neighbours does not help - it just gives reflection two keys to swap. Rotation maps `.ring` onto `.ring`, which is why para works. The root cause of the second is worth recording: a Table's keys must be hashable before its entries can be ordered, so a cyclic value cannot be a key, which rules out the unordered neighbour set a molecule really wants. The digest is therefore of an oriented drawing. Pinned here so a future change to the encoding has to argue with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

#17 landed directory, closure and ctx.cache hashing while this branch was building the same three as scaffolding for `cached`. Its versions win — they are merged, they are better factored, and decisively they handle **cyclic values**, which #16 made constructible and which mine would have recursed on forever. Dropped: hash_directory.odin, my hash_function.odin, my hash.odin changes, my Fs_Entry extension and fs_list_dir_at, my hash tests, and my additions to examples/hashing.hb (three dedicated hashing examples cover it better). cache_store.odin now reads directories through main's fs_list_entries_at, whose getdents walk is better than the /proc/self/fd one it replaces, and refuses a fifo/socket/device the same way §3's hash does. Kept, because main has none of it: - `cached` itself, rebound to value_digest(v, interp) and Hash_Fail. - Four fs write operations — mkdir/rename/unlink/rmdir, on all three backends. #17 added reading; the store needs writing. - The `#arg`/`#self` coverage, now hash_implicit.odin. main's free_names collects identifiers and a uses_ctx flag, but nothing looks at Implicit_Name or Hole — they are dynamic lookups no closure captures, so `let f func (cached (#arg + 1))` would answer `f 10` with 2. **Cyclic values are cacheable**, which the format could not previously express. A Table reached more than once is written `node "N" { … }` at its first occurrence and `ref "N"` after that; the reader creates each Table before reading its entries, exactly as §10's evaluation order does, so a definition always precedes its references and nothing needs patching up. A restored cycle is bisimulation-equal to the stored one, which is what §6 requires. Labels go on merely shared Tables too — not needed for correctness, since §6 compares structurally, but it stops a shared value expanding exponentially on the way out. 230 tests pass; all three targets typecheck. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Reverses the resolution #17 recorded earlier the same day. A directory entry now hashes as name and content, with no permission input on any target. #17 kept a single executable flag — the same reduction git makes — read where a target can see one and false elsewhere, and cited `core.filemode` as taking the same position. Looking at what git actually *does* rather than what it records reverses that. Git never re-derives the bit on a target that cannot report one: `core.fileMode=false` on Windows means it carries the mode from the index, so a committed `100755` round-trips through the repository without ever existing in the working tree. The same commit is executable on Linux and not on Windows. So hashing the bit made one tree two values depending on where it was checked out — and not hypothetically. This repository has four `100755` files, so `sha256 loadfile "scripts"` would have answered two ways. Remembering the bit instead, as git does, is not available to a `File`: it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, §3 now takes the second. The bit is still read and still restored when `cached` copies a tree, so caching a build output doesn't silently strip it — fidelity in the store, not identity in the language, and the comments say so. hash_linux_test.odin's assertion inverts: it now sets the bit, confirms it actually landed (otherwise the check would hold for the wrong reason), and asserts the digest is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Implement `cached` (§15), and the two hashes it needed `cached <expr>` evaluates an expression once and reads the answer back on every later run. §15 had pinned the cache *key* — the expression treated as a function, hashed as one — but left "where does the cache actually live" as an open TODO, and two of the digests it rests on did not exist. **The key.** A closure now hashes as its code (the AST subtree, structurally, so reformatting or commenting it changes nothing), its captured `ctx`, and the values of the free names it uses. `#arg`/`#self` are dynamic lookups that no closure captures, so `cached` mixes in the stack entries the expression can reach, bounded statically by the largest N written in it — without that, `let f func (cached (#arg + 1))` answers `f 10` with `f 1`'s result. **The layout**, resolving §15's TODO. One entry per key, in ctx.cache's directory: <cache>/sha256-<key> a File value, stored as itself <cache>/sha256-<key>.hb/value.hb anything else, as HashedBuild text <cache>/sha256-<key>.hb/sha256-<h> each File inside it, by content hash A file stays a file and a directory stays a directory, so what a build produced is still something you can open. The text format is HashedBuild's own value syntax read by a separate literals-only parser, not `import`, so a hand-edited entry is a parse failure rather than code that runs. Entries commit by rename, which is also how two runs racing on one key settle it. **Two digests that had to exist first**, both settled with the language's owner rather than assumed: - §3's directory hash, over entries sorted by name. Only Linux can report an executable bit; WASI and Windows hash every entry as non-executable, so a tree containing an executable hashes differently there. SPEC.md §3 now says so, and says why the alternatives were worse. This was the open question that kept directory hashing unbuilt. - `ctx.cache`'s, as a tagged constant rather than its path — it is in every key now, and baking the path in would invalidate a cache that was moved. Six operations join the fs layer for this (listing by descriptor with no-follow classification, mkdir, rename, unlink, rmdir, and the exec bit), implemented on all three targets. LANGUAGE.md gains a Caching section and loses `cached` and all three unhashable kinds from "what isn't built yet"; examples/cached.hb is asserted by the suite like every other example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Drop the executable bit from §3's directory hash A directory entry now hashes as name and content, with no permission input on any target. §3 previously kept a single executable flag — the same reduction git makes (`100644` vs `100755`) — and hashed it where it could be read. Only Linux can read it. WASI's filestat carries no permission bits and Windows has no POSIX execute bit, and neither has a stand-in worth using: an `.exe` extension is a different question, and what MSYS guesses from a shebang is a guess. So hashing it made one tree two values depending on where it was checked out — and not hypothetically. Git sets `core.fileMode=false` on Windows, so the bit round-trips through the repository without ever existing in the working tree; the same commit checks out executable on Linux and not on Windows. This repository has four such files, so `sha256 loadfile "scripts"` would have answered two ways. Git's own way out is to *remember* the bit rather than re-derive it, which a `File` cannot do — it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, this takes the second: two trees differing only in an executable bit are one value, and the platform a build runs on no longer changes what its inputs hash to. The bit is still read and still preserved when `cached` copies a tree, so caching a build output doesn't silently strip it. That is fidelity in the store, not identity in the language, and the comments now say so. Covered by a Linux-only test that sets the bit and asserts the digest is unchanged — Linux-only because elsewhere fs_set_executable_at is a no-op, and the test would pass without establishing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Demonstrate function and directory hashing, not just document them `sha256 <function>` and `sha256 <directory>` both work as of the `cached` work, and LANGUAGE.md described them in prose — but nothing in examples/ exercised either, and the Hashing section's snippet block still only showed a string, a file and a Table. A feature nobody outside this repo can run is not finished. examples/hashing.hb gains three entries, each asserting a property rather than a digest so they stay meaningful: - whitespace and comments are not part of a function's code; - the values it captures *are* — the property `cached` rests on, since without it one entry would serve every argument; - a directory is its own kind of value, not its contents run together. LANGUAGE.md's snippet block gains the two new forms, and the Function bullet gains the pair of one-liners above. Every snippet in it was run before it was written down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Drop the executable bit from §3's directory hash Reverses the resolution #17 recorded earlier the same day. A directory entry now hashes as name and content, with no permission input on any target. #17 kept a single executable flag — the same reduction git makes — read where a target can see one and false elsewhere, and cited `core.filemode` as taking the same position. Looking at what git actually *does* rather than what it records reverses that. Git never re-derives the bit on a target that cannot report one: `core.fileMode=false` on Windows means it carries the mode from the index, so a committed `100755` round-trips through the repository without ever existing in the working tree. The same commit is executable on Linux and not on Windows. So hashing the bit made one tree two values depending on where it was checked out — and not hypothetically. This repository has four `100755` files, so `sha256 loadfile "scripts"` would have answered two ways. Remembering the bit instead, as git does, is not available to a `File`: it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, §3 now takes the second. The bit is still read and still restored when `cached` copies a tree, so caching a build output doesn't silently strip it — fidelity in the store, not identity in the language, and the comments say so. hash_linux_test.odin's assertion inverts: it now sets the bit, confirms it actually landed (otherwise the check would hold for the wrong reason), and asserts the digest is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Show a cached cycle in the example, not just in the docs Caching a value that reaches itself is a user-visible capability and LANGUAGE.md describes it, but nothing in examples/ ran one. cached.hb now caches a self-referential Table and asserts the digest survives the round trip - which is the whole test, since §6 compares cyclic values by bisimulation, so a back-edge that came back as an unfolding of the wrong depth would not compare equal to what was stored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make `cached` work on WASI: rename rights, and the playground's shim Two failures CI caught and this machine cannot, both from the four fs write operations the cache needs. **wasi-threads**: `cached.hb` died with "could not publish cache entry (Access)" under iwasm. preview1 checks a rename against two separate rights — PATH_RENAME_SOURCE on the source descriptor and PATH_RENAME_TARGET on the destination's — and DIR_RIGHTS asked for neither, so every commit-by-rename came back ENOTCAPABLE, which fs.odin folds into a bare .Access. **playground**: the browser refused to instantiate the module at all — `LinkError: "path_rename": function import requires a callable`. docs/wasi.js implements preview1 for the playground, and nothing had needed to remove or rename anything before, so three imports simply weren't there. Added path_rename, path_unlink_file and path_remove_directory, plus the FileSystem `remove`/`rename` they sit on, plus the arity entries a spawned thread marshals them through. `rename` refuses an existing destination, which is what makes it the cache's commit rather than an overwrite. ENOTEMPTY (55) joins the ERRNO table, since an rmdir of a non-empty directory now has a way to say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

janstrakowski and others added 2 commits August 31, 2026 09:56

janstrakowski merged commit 5e79088 into main Aug 31, 2026
5 checks passed

janstrakowski deleted the worktree-hashing branch August 31, 2026 11:42

janstrakowski mentioned this pull request Aug 31, 2026

Implement cached (§15) #18

Merged

Navigation Menu

Finish hashing: directories, closures, cycles, and ctx.cache - #17

Finish hashing: directories, closures, cycles, and ctx.cache#17
janstrakowski merged 2 commits into
mainfrom
worktree-hashing

janstrakowski commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

janstrakowski commented Aug 31, 2026

The four decisions

Implementation notes

Tests

Docs

Verified locally

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant