Repository navigation
Design Log
TheMissenoOne edited this page Oct 31, 2026
·
6 revisions
OpenArc's spec was tested by using it: first by writing OpenArc's own compiler as OpenArc specs, then by running it. Each finding below changed the spec ([Fn] tags in SPEC-phase0.md).
| # | Found by | Problem | Resolution |
|---|---|---|---|
| F1 | writing specs | long intents needed .join(), a call, which breaks static specs |
intent: string | string[] |
| F2 | self-hosting | specs' own examples contain marker text in strings, so a line scanner false-alarms | markers are recognised on comment tokens only |
| F3 | running the TS scanner | the raw scanner drops whitespace inside templates and regexes, so a real edit could keep the old hash and be silently overwritten | hash parse-tree leaves, not scanner tokens |
| F4 | tsc on the scaffold | a never-typed placeholder makes later code unreachable, breaking narrowing, so the first compile could never pass |
block placeholder arc.pending(): void
|
| F5 | developer skeletons | new marked regions defaulted to developer, so pending blocks could never be generated |
a placeholder body always means pending
|
| F6 | effectful regions | JSON examples can't pass ports | Fakes |
| F7 | block examples | arrows were treated as opaque, but they capture this/arguments
|
arrows are transparent |
| F8 | running bwrap | asdf shim → wrong Node inside the sandbox; --as too small crashes V8; process.exit() truncated output |
resolved binary + prefix bind; completion line |
| F9 | diagnostic examples | prose messages make examples brittle | codes + structured detail, no message text |
| F10 | cross-file imports | code that imports a not-yet-scaffolded symbol fails tsc | scaffold before validate |
| F11 | prompt template | a generated template would feed model output into its own cache key | policy artifacts are hand-written |
| F12 | core/ |
the compiler can't generate itself before it exists | Bootstrap stages |
| F13 | first real model call | small models return a whole function when asked for a body | the compiler unwraps it |
| F14 | sibling blocks | blocks can only pass their function's examples together | inconclusive per block, joint check before commit |
| — | stage 0 examples |
getLeadingCommentRanges misses same-line comments, so x = 1; // why edits weren't detected |
collect trailing comment ranges too |
| — | first sandboxed .ts
|
Node's WASM type stripper needs ~10 GiB of address space |
--as ≥ 16 GiB (default 32 GiB) |
| — | review | write scope was only implied | structural guards, see Safety Model |
| — | hand-editing a compiled region | with 0 targets, compile returned before STAGE, so the W060 transition was never written: the warning repeated every run and a later spec change couldn't be seen as E061 |
a manifest-only transaction when ownership changed and nothing compiles |
| — | second self-compile | a model wrote import * as ts from "typescript" (TS2595: typescript needs a default import) |
context packages show the project's canonical import line for each package dep; TS2595/1259/2497 escalate context |
| — | second self-compile | Ollama was OOM-killed mid-run, and every remaining region failed with fetch failed
|
the backend retries a connection error once after 10 s; --keep-going stops after 3 consecutive E080s |
| — | second self-compile | with --keep-going, a function whose block failed skipped its joint check, but its other generated blocks were committed anyway; compile.run.parse landed wrong |
a block is committed only after its function passes all its examples; otherwise it's held back (W042) and kept in the cache |
| — | VS Code extension parser |
engine.ts comments that mentioned the marker prefix raised W001 on OpenArc's own source |
reworded; hand-written files are checked with scanMarkers
|
| — | adopting existing code | §10 only covered block extraction |
openarc extract for whole functions, see Adopting Existing Code
|
| E1 | evaluation on three real projects (openarc-eval REPORT.md, 2026-10-31, §1) |
6 of 13 round-1 failures were scope gaps: the type closure printed Side/Muscle though the file didn't import them (TS2304 at L0, L1 and L2), L0 hid the file's imports and same-file helpers (homography couldn't see solve, outputFileName re-implemented sanitize, balance wrote require), and verify allowed only deps while the rules allowed file imports |
L0 shows the file's imports and its implemented same-file values; closure types not in scope are marked and imported by the compiler; rules and verifier accept the file's modules and the marked types; require errors say what to do; default imports via "default"
|
| E2 | evaluation §2 |
fitLine's original code failed its own example (got [0,1,0]; expected [0,1,0]: -0 under isDeepStrictEqual), NaN printed as null, and exact float equality made examples hinge on operation order |
numbers compare within tolerance (default 1e-9 relative), -0 equals 0 unless exact, NaN equals NaN; "$NaN"/"$Infinity"/"$-Infinity"/"$undefined" tokens; faithful printing |
| E3 | evaluation §3, §6 | Ollama ran at temperature 0 with a fixed seed, so a repeated repair prompt returned the same body (balance: byte-identical prompts and bodies); no output cap (two 170 s calls ending in invalid JSON); invalid outputs and failed bodies were not logged or kept |
a byte-identical retry prompt gets temperature 0.4, then 0.7, and a new seed (recorded in provenance.sampling); num_predict = outputReserveTokens; every invalid output is logged with its reason; failed attempts go to .openarc/attempts/<id>/ (last 20) |
| E4 | evaluation §4 |
coverage and resolveChannelId got a stronger example, failed to recompile, and kept their old bodies, which fail it, while graph and inspect still showed compiler
|
display state stale in graph, inspect and watch; OARC-W043 and exit 1 from compile
|
| E5 | evaluation §5 | three regions passed every example and diverged from the original (50/80, 8/40, 1 of 5) |
OARC-W044 for thin examples; openarc check <id> --oracle <module>#<export>
|
| E6 | evaluation §7, §8 |
Date.parse needed clock; "tsc then scaffold" fails once one spec file imports another's region; cache keys used a stale COMPILER_VERSION ("0.2.0" in 0.3.0) |
Date.parse, Date.UTC, new Date(value) and the Date type are pure; init says scaffold, then tsc; COMPILER_VERSION is read from the package version |
| — | watch mode | a continuous compiler can hot-loop a hopeless region, analyse while a model is loaded (the machine had OOM-crashed with two models resident), drop its queue when Ollama dies, and leave .openarc/lock behind when killed |
one worker and no analysis during a compile; failed regions wait for their spec, fingerprint or config to change; pause with backoff and keep the queue; the lock holds a PID and a dead holder's lock is removed (§15, Watch Mode) |
| — | self-compile 3 (selfhost-compile-3.log, 0.6.2) |
compile.run.unlock, .recover and .analyze were cached as verified (examples:1/2 (1 inconclusive)) with bodies that released the lock and re-ran parse through commit, ran a commit, and parsed twice. The only conclusive example, lock held by another run, returns before every block but lock
|
calls allowlist on specs (OARC-E098), tight lists on every compile.run.* block; a coverage probe in the sandbox copy records which blocks passing examples reach, and an unreached block is unverified (OARC-W045) and held back; compile.run gained an end-to-end example that reaches every block |
| — | self-compile 3 |
hash.sha256Tagged failed 3 times with 'crypto' requires effect 'random': the model requested import crypto from "node:crypto", but the static check computed file bindings before merging requested imports |
requested imports count as file bindings; random covers only Math.random and crypto.getRandomValues/randomUUID/randomBytes
|
| — | self-compile 3 |
markers.scanMarkers failed 3 times with E097 for comments such as // matches @openarc: markers
|
such comments are stripped (and logged) before splicing; marker-shaped lines are still rejected |
| — | self-compile 3 |
block.checkRestrictions returned invalid JSON 3 times, cut at num_predict 2048 |
a truncated reply (done_reason/finish_reason length, stop_reason max_tokens) is retried with double the cap; default cap 3072 |
| — | watch runs (0.6.1) | a region that passed but whose commit hit OARC-E096 (the other region of its file was being edited) was remembered as failed and still pending; and committing text.titleCase re-queued the just-compiled text.slugify because the fingerprint hashed the prompt, which lists implemented siblings |
E096 re-queues at the front without a pause (history requeued); fingerprints hash a stable view of the context: sibling signatures, never bodies |
| — | 0.7.0 portability | OpenArc ran only on Linux: the sandbox was bwrap + prlimit, staging used cp -al, the package said os: ["linux"], the lock compared inodes, TypeScript reports Windows paths with /, and CRLF checkouts hashed differently |
tiered sandbox in src/isolation.ts (bwrap, sandbox-exec, node with OARC-W046, opt-in container, explicit unsafe-none); non-bwrap tiers read the project's node_modules in place instead of copying it; the lock falls back to pid + mtime when ino is 0; the engine holds a posix root; CRLF files are held as LF (same hashes on every OS) and written back as CRLF; openarc doctor; CI on Linux, macOS and Windows |
| B1 | benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 1 |
with node_modules a symlink to a shared install (or links leaving the project, as in pnpm layouts), the bwrap stage's copy dangled: ERR_MODULE_NOT_FOUND, and each stderr line became an OARC-E090 example "<unparseable>" that burned every model attempt |
isolation.bwrapReadPaths: links that leave the project are made absolute in the stage and their real paths are bound read-only at the same path, after --tmpfs /tmp (SandboxConfig.extraReadPaths, a new sandbox.buildArgv example). The node and sandbox-exec tiers failed the same way (Node checks <root>/node_modules/<pkg>/package.json under the unresolved path): they now also grant <root>/node_modules as spelled |
| B2 | benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 2 |
type Nested = Array<number | null | undefined | Nested> in F failed the spec itself with TS2589 on the args line, before any body existed, and compile charged it to the model (flatten-array: 0/3 whatever the model wrote) |
Encoded<T> counts nesting depth and stops adding the tokens at 8 levels |
| B3 | benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 3 |
provenance and attempts recorded no token counts, so "implementations per unit of compute" could only be measured with a counting proxy |
tokens: {prompt, completion} per attempt (Ollama prompt_eval_count/eval_count, OpenAI usage, Anthropic usage), totals per region in provenance and the cache, tokens in openarc inspect
|
| B4 | benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 4 |
examples couldn't say "throws": 5 of 11 binary-search and about 8 of 23 wordy tests were unconstrainable |
{ args, throws: true | "substring" } instead of expected (type, harness, failure messages, context package) |
| B5 | benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 6 and the pilot (OpenArc 0/5 against a plain-prompt baseline at 2/5 on qwen3.5:9b) |
every call asked for code as an escaped JSON string under grammar-constrained decoding, while the baseline got free-form markdown: a confound the pilot couldn't separate |
outputFormat: "code" per backend or region: one fenced block, leading imports parsed, no schema sent, then exactly the same guards; in the fingerprint and provenance. Default stays "json" until the ablation is run |
The pattern: every serious bug was found by executing something (tsc, the scanner, the sandbox, the examples), never by re-reading prose. That's why the spec's tables are executable: the ownership table is ownership.detect's examples, and the sandbox command is sandbox.buildArgv's example.