Skip to content

Design Log

TheMissenoOne edited this page Oct 31, 2026 · 6 revisions

Design Log

OpenArc's spec was tested by using it: first by writing OpenArc's own compiler as OpenArc specs, then by running it. Each finding below changed the spec ([Fn] tags in SPEC-phase0.md).

# Found by Problem Resolution
F1 writing specs long intents needed .join(), a call, which breaks static specs intent: string | string[]
F2 self-hosting specs' own examples contain marker text in strings, so a line scanner false-alarms markers are recognised on comment tokens only
F3 running the TS scanner the raw scanner drops whitespace inside templates and regexes, so a real edit could keep the old hash and be silently overwritten hash parse-tree leaves, not scanner tokens
F4 tsc on the scaffold a never-typed placeholder makes later code unreachable, breaking narrowing, so the first compile could never pass block placeholder arc.pending(): void
F5 developer skeletons new marked regions defaulted to developer, so pending blocks could never be generated a placeholder body always means pending
F6 effectful regions JSON examples can't pass ports Fakes
F7 block examples arrows were treated as opaque, but they capture this/arguments arrows are transparent
F8 running bwrap asdf shim → wrong Node inside the sandbox; --as too small crashes V8; process.exit() truncated output resolved binary + prefix bind; completion line
F9 diagnostic examples prose messages make examples brittle codes + structured detail, no message text
F10 cross-file imports code that imports a not-yet-scaffolded symbol fails tsc scaffold before validate
F11 prompt template a generated template would feed model output into its own cache key policy artifacts are hand-written
F12 core/ the compiler can't generate itself before it exists Bootstrap stages
F13 first real model call small models return a whole function when asked for a body the compiler unwraps it
F14 sibling blocks blocks can only pass their function's examples together inconclusive per block, joint check before commit
— stage 0 examples getLeadingCommentRanges misses same-line comments, so x = 1; // why edits weren't detected collect trailing comment ranges too
— first sandboxed .ts Node's WASM type stripper needs ~10 GiB of address space --as ≥ 16 GiB (default 32 GiB)
— review write scope was only implied structural guards, see Safety Model
— hand-editing a compiled region with 0 targets, compile returned before STAGE, so the W060 transition was never written: the warning repeated every run and a later spec change couldn't be seen as E061 a manifest-only transaction when ownership changed and nothing compiles
— second self-compile a model wrote import * as ts from "typescript" (TS2595: typescript needs a default import) context packages show the project's canonical import line for each package dep; TS2595/1259/2497 escalate context
— second self-compile Ollama was OOM-killed mid-run, and every remaining region failed with fetch failed the backend retries a connection error once after 10 s; --keep-going stops after 3 consecutive E080s
— second self-compile with --keep-going, a function whose block failed skipped its joint check, but its other generated blocks were committed anyway; compile.run.parse landed wrong a block is committed only after its function passes all its examples; otherwise it's held back (W042) and kept in the cache
— VS Code extension parser engine.ts comments that mentioned the marker prefix raised W001 on OpenArc's own source reworded; hand-written files are checked with scanMarkers
— adopting existing code §10 only covered block extraction openarc extract for whole functions, see Adopting Existing Code
E1 evaluation on three real projects (openarc-eval REPORT.md, 2026-10-31, §1) 6 of 13 round-1 failures were scope gaps: the type closure printed Side/Muscle though the file didn't import them (TS2304 at L0, L1 and L2), L0 hid the file's imports and same-file helpers (homography couldn't see solve, outputFileName re-implemented sanitize, balance wrote require), and verify allowed only deps while the rules allowed file imports L0 shows the file's imports and its implemented same-file values; closure types not in scope are marked and imported by the compiler; rules and verifier accept the file's modules and the marked types; require errors say what to do; default imports via "default"
E2 evaluation §2 fitLine's original code failed its own example (got [0,1,0]; expected [0,1,0]: -0 under isDeepStrictEqual), NaN printed as null, and exact float equality made examples hinge on operation order numbers compare within tolerance (default 1e-9 relative), -0 equals 0 unless exact, NaN equals NaN; "$NaN"/"$Infinity"/"$-Infinity"/"$undefined" tokens; faithful printing
E3 evaluation §3, §6 Ollama ran at temperature 0 with a fixed seed, so a repeated repair prompt returned the same body (balance: byte-identical prompts and bodies); no output cap (two 170 s calls ending in invalid JSON); invalid outputs and failed bodies were not logged or kept a byte-identical retry prompt gets temperature 0.4, then 0.7, and a new seed (recorded in provenance.sampling); num_predict = outputReserveTokens; every invalid output is logged with its reason; failed attempts go to .openarc/attempts/<id>/ (last 20)
E4 evaluation §4 coverage and resolveChannelId got a stronger example, failed to recompile, and kept their old bodies, which fail it, while graph and inspect still showed compiler display state stale in graph, inspect and watch; OARC-W043 and exit 1 from compile
E5 evaluation §5 three regions passed every example and diverged from the original (50/80, 8/40, 1 of 5) OARC-W044 for thin examples; openarc check <id> --oracle <module>#<export>
E6 evaluation §7, §8 Date.parse needed clock; "tsc then scaffold" fails once one spec file imports another's region; cache keys used a stale COMPILER_VERSION ("0.2.0" in 0.3.0) Date.parse, Date.UTC, new Date(value) and the Date type are pure; init says scaffold, then tsc; COMPILER_VERSION is read from the package version
— watch mode a continuous compiler can hot-loop a hopeless region, analyse while a model is loaded (the machine had OOM-crashed with two models resident), drop its queue when Ollama dies, and leave .openarc/lock behind when killed one worker and no analysis during a compile; failed regions wait for their spec, fingerprint or config to change; pause with backoff and keep the queue; the lock holds a PID and a dead holder's lock is removed (§15, Watch Mode)
— self-compile 3 (selfhost-compile-3.log, 0.6.2) compile.run.unlock, .recover and .analyze were cached as verified (examples:1/2 (1 inconclusive)) with bodies that released the lock and re-ran parse through commit, ran a commit, and parsed twice. The only conclusive example, lock held by another run, returns before every block but lock calls allowlist on specs (OARC-E098), tight lists on every compile.run.* block; a coverage probe in the sandbox copy records which blocks passing examples reach, and an unreached block is unverified (OARC-W045) and held back; compile.run gained an end-to-end example that reaches every block
— self-compile 3 hash.sha256Tagged failed 3 times with 'crypto' requires effect 'random': the model requested import crypto from "node:crypto", but the static check computed file bindings before merging requested imports requested imports count as file bindings; random covers only Math.random and crypto.getRandomValues/randomUUID/randomBytes
— self-compile 3 markers.scanMarkers failed 3 times with E097 for comments such as // matches @openarc: markers such comments are stripped (and logged) before splicing; marker-shaped lines are still rejected
— self-compile 3 block.checkRestrictions returned invalid JSON 3 times, cut at num_predict 2048 a truncated reply (done_reason/finish_reason length, stop_reason max_tokens) is retried with double the cap; default cap 3072
— watch runs (0.6.1) a region that passed but whose commit hit OARC-E096 (the other region of its file was being edited) was remembered as failed and still pending; and committing text.titleCase re-queued the just-compiled text.slugify because the fingerprint hashed the prompt, which lists implemented siblings E096 re-queues at the front without a pause (history requeued); fingerprints hash a stable view of the context: sibling signatures, never bodies
— 0.7.0 portability OpenArc ran only on Linux: the sandbox was bwrap + prlimit, staging used cp -al, the package said os: ["linux"], the lock compared inodes, TypeScript reports Windows paths with /, and CRLF checkouts hashed differently tiered sandbox in src/isolation.ts (bwrap, sandbox-exec, node with OARC-W046, opt-in container, explicit unsafe-none); non-bwrap tiers read the project's node_modules in place instead of copying it; the lock falls back to pid + mtime when ino is 0; the engine holds a posix root; CRLF files are held as LF (same hashes on every OS) and written back as CRLF; openarc doctor; CI on Linux, macOS and Windows
B1 benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 1 with node_modules a symlink to a shared install (or links leaving the project, as in pnpm layouts), the bwrap stage's copy dangled: ERR_MODULE_NOT_FOUND, and each stderr line became an OARC-E090 example "<unparseable>" that burned every model attempt isolation.bwrapReadPaths: links that leave the project are made absolute in the stage and their real paths are bound read-only at the same path, after --tmpfs /tmp (SandboxConfig.extraReadPaths, a new sandbox.buildArgv example). The node and sandbox-exec tiers failed the same way (Node checks <root>/node_modules/<pkg>/package.json under the unresolved path): they now also grant <root>/node_modules as spelled
B2 benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 2 type Nested = Array<number | null | undefined | Nested> in F failed the spec itself with TS2589 on the args line, before any body existed, and compile charged it to the model (flatten-array: 0/3 whatever the model wrote) Encoded<T> counts nesting depth and stops adding the tokens at 8 levels
B3 benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 3 provenance and attempts recorded no token counts, so "implementations per unit of compute" could only be measured with a counting proxy tokens: {prompt, completion} per attempt (Ollama prompt_eval_count/eval_count, OpenAI usage, Anthropic usage), totals per region in provenance and the cache, tokens in openarc inspect
B4 benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 4 examples couldn't say "throws": 5 of 11 binary-search and about 8 of 23 wordy tests were unconstrainable { args, throws: true | "substring" } instead of expected (type, harness, failure messages, context package)
B5 benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 6 and the pilot (OpenArc 0/5 against a plain-prompt baseline at 2/5 on qwen3.5:9b) every call asked for code as an escaped JSON string under grammar-constrained decoding, while the baseline got free-form markdown: a confound the pilot couldn't separate outputFormat: "code" per backend or region: one fenced block, leading imports parsed, no schema sent, then exactly the same guards; in the fingerprint and provenance. Default stays "json" until the ablation is run

The pattern: every serious bug was found by executing something (tsc, the scanner, the sandbox, the examples), never by re-reading prose. That's why the spec's tables are executable: the ownership table is ownership.detect's examples, and the sandbox command is sandbox.buildArgv's example.

Clone this wiki locally