Design Log

Design Log

OpenArc's spec was tested by using it: first by writing OpenArc's own compiler as OpenArc specs, then by running it. Each finding below changed the spec ([Fn] tags in SPEC-phase0.md).

#	Found by	Problem	Resolution
F1	writing specs	long intents needed .join(), a call, which breaks static specs	intent: string | string[]
F2	self-hosting	specs' own examples contain marker text in strings, so a line scanner false-alarms	markers are recognised on comment tokens only
F3	running the TS scanner	the raw scanner drops whitespace inside templates and regexes, so a real edit could keep the old hash and be silently overwritten	hash parse-tree leaves, not scanner tokens
F4	tsc on the scaffold	a never-typed placeholder makes later code unreachable, breaking narrowing, so the first compile could never pass	block placeholder arc.pending(): void
F5	developer skeletons	new marked regions defaulted to developer, so pending blocks could never be generated	a placeholder body always means pending
F6	effectful regions	JSON examples can't pass ports	Fakes
F7	block examples	arrows were treated as opaque, but they capture this/arguments	arrows are transparent
F8	running bwrap	asdf shim → wrong Node inside the sandbox; --as too small crashes V8; process.exit() truncated output	resolved binary + prefix bind; completion line
F9	diagnostic examples	prose messages make examples brittle	codes + structured detail, no message text
F10	cross-file imports	code that imports a not-yet-scaffolded symbol fails tsc	scaffold before validate
F11	prompt template	a generated template would feed model output into its own cache key	policy artifacts are hand-written
F12	core/	the compiler can't generate itself before it exists	Bootstrap stages
F13	first real model call	small models return a whole function when asked for a body	the compiler unwraps it
F14	sibling blocks	blocks can only pass their function's examples together	inconclusive per block, joint check before commit
—	stage 0 examples	getLeadingCommentRanges misses same-line comments, so x = 1; // why edits weren't detected	collect trailing comment ranges too
—	first sandboxed .ts	Node's WASM type stripper needs ~10 GiB of address space	--as ≥ 16 GiB (default 32 GiB)
—	review	write scope was only implied	structural guards, see Safety Model
—	hand-editing a compiled region	with 0 targets, compile returned before STAGE, so the W060 transition was never written: the warning repeated every run and a later spec change couldn't be seen as E061	a manifest-only transaction when ownership changed and nothing compiles
—	second self-compile	a model wrote import * as ts from "typescript" (TS2595: typescript needs a default import)	context packages show the project's canonical import line for each package dep; TS2595/1259/2497 escalate context
—	second self-compile	Ollama was OOM-killed mid-run, and every remaining region failed with fetch failed	the backend retries a connection error once after 10 s; --keep-going stops after 3 consecutive E080s
—	second self-compile	with --keep-going, a function whose block failed skipped its joint check, but its other generated blocks were committed anyway; compile.run.parse landed wrong	a block is committed only after its function passes all its examples; otherwise it's held back (W042) and kept in the cache
—	VS Code extension parser	engine.ts comments that mentioned the marker prefix raised W001 on OpenArc's own source	reworded; hand-written files are checked with scanMarkers
—	adopting existing code	§10 only covered block extraction	openarc extract for whole functions, see Adopting Existing Code
E1	evaluation on three real projects (openarc-eval REPORT.md, 2026-10-31, §1)	6 of 13 round-1 failures were scope gaps: the type closure printed Side/Muscle though the file didn't import them (TS2304 at L0, L1 and L2), L0 hid the file's imports and same-file helpers (homography couldn't see solve, outputFileName re-implemented sanitize, balance wrote require), and verify allowed only deps while the rules allowed file imports	L0 shows the file's imports and its implemented same-file values; closure types not in scope are marked and imported by the compiler; rules and verifier accept the file's modules and the marked types; require errors say what to do; default imports via "default"
E2	evaluation §2	fitLine's original code failed its own example (got [0,1,0]; expected [0,1,0]: -0 under isDeepStrictEqual), NaN printed as null, and exact float equality made examples hinge on operation order	numbers compare within tolerance (default 1e-9 relative), -0 equals 0 unless exact, NaN equals NaN; "$NaN"/"$Infinity"/"$-Infinity"/"$undefined" tokens; faithful printing
E3	evaluation §3, §6	Ollama ran at temperature 0 with a fixed seed, so a repeated repair prompt returned the same body (balance: byte-identical prompts and bodies); no output cap (two 170 s calls ending in invalid JSON); invalid outputs and failed bodies were not logged or kept	a byte-identical retry prompt gets temperature 0.4, then 0.7, and a new seed (recorded in provenance.sampling); num_predict = outputReserveTokens; every invalid output is logged with its reason; failed attempts go to .openarc/attempts/<id>/ (last 20)
E4	evaluation §4	coverage and resolveChannelId got a stronger example, failed to recompile, and kept their old bodies, which fail it, while graph and inspect still showed compiler	display state stale in graph, inspect and watch; OARC-W043 and exit 1 from compile
E5	evaluation §5	three regions passed every example and diverged from the original (50/80, 8/40, 1 of 5)	OARC-W044 for thin examples; openarc check <id> --oracle <module>#<export>
E6	evaluation §7, §8	Date.parse needed clock; "tsc then scaffold" fails once one spec file imports another's region; cache keys used a stale COMPILER_VERSION ("0.2.0" in 0.3.0)	Date.parse, Date.UTC, new Date(value) and the Date type are pure; init says scaffold, then tsc; COMPILER_VERSION is read from the package version
—	watch mode	a continuous compiler can hot-loop a hopeless region, analyse while a model is loaded (the machine had OOM-crashed with two models resident), drop its queue when Ollama dies, and leave .openarc/lock behind when killed	one worker and no analysis during a compile; failed regions wait for their spec, fingerprint or config to change; pause with backoff and keep the queue; the lock holds a PID and a dead holder's lock is removed (§15, Watch Mode)
—	self-compile 3 (selfhost-compile-3.log, 0.6.2)	compile.run.unlock, .recover and .analyze were cached as verified (examples:1/2 (1 inconclusive)) with bodies that released the lock and re-ran parse through commit, ran a commit, and parsed twice. The only conclusive example, lock held by another run, returns before every block but lock	calls allowlist on specs (OARC-E098), tight lists on every compile.run.* block; a coverage probe in the sandbox copy records which blocks passing examples reach, and an unreached block is unverified (OARC-W045) and held back; compile.run gained an end-to-end example that reaches every block
—	self-compile 3	hash.sha256Tagged failed 3 times with 'crypto' requires effect 'random': the model requested import crypto from "node:crypto", but the static check computed file bindings before merging requested imports	requested imports count as file bindings; random covers only Math.random and crypto.getRandomValues/randomUUID/randomBytes
—	self-compile 3	markers.scanMarkers failed 3 times with E097 for comments such as // matches @openarc: markers	such comments are stripped (and logged) before splicing; marker-shaped lines are still rejected
—	self-compile 3	block.checkRestrictions returned invalid JSON 3 times, cut at num_predict 2048	a truncated reply (done_reason/finish_reason length, stop_reason max_tokens) is retried with double the cap; default cap 3072
—	watch runs (0.6.1)	a region that passed but whose commit hit OARC-E096 (the other region of its file was being edited) was remembered as failed and still pending; and committing text.titleCase re-queued the just-compiled text.slugify because the fingerprint hashed the prompt, which lists implemented siblings	E096 re-queues at the front without a pause (history requeued); fingerprints hash a stable view of the context: sibling signatures, never bodies
—	0.7.0 portability	OpenArc ran only on Linux: the sandbox was bwrap + prlimit, staging used cp -al, the package said os: ["linux"], the lock compared inodes, TypeScript reports Windows paths with /, and CRLF checkouts hashed differently	tiered sandbox in src/isolation.ts (bwrap, sandbox-exec, node with OARC-W046, opt-in container, explicit unsafe-none); non-bwrap tiers read the project's node_modules in place instead of copying it; the lock falls back to pid + mtime when ino is 0; the engine holds a posix root; CRLF files are held as LF (same hashes on every OS) and written back as CRLF; openarc doctor; CI on Linux, macOS and Windows
B1	benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 1	with node_modules a symlink to a shared install (or links leaving the project, as in pnpm layouts), the bwrap stage's copy dangled: ERR_MODULE_NOT_FOUND, and each stderr line became an OARC-E090 example "<unparseable>" that burned every model attempt	isolation.bwrapReadPaths: links that leave the project are made absolute in the stage and their real paths are bound read-only at the same path, after --tmpfs /tmp (SandboxConfig.extraReadPaths, a new sandbox.buildArgv example). The node and sandbox-exec tiers failed the same way (Node checks <root>/node_modules/<pkg>/package.json under the unresolved path): they now also grant <root>/node_modules as spelled
B2	benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 2	type Nested = Array<number | null | undefined | Nested> in F failed the spec itself with TS2589 on the args line, before any body existed, and compile charged it to the model (flatten-array: 0/3 whatever the model wrote)	Encoded<T> counts nesting depth and stops adding the tokens at 8 levels
B3	benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 3	provenance and attempts recorded no token counts, so "implementations per unit of compute" could only be measured with a counting proxy	tokens: {prompt, completion} per attempt (Ollama prompt_eval_count/eval_count, OpenAI usage, Anthropic usage), totals per region in provenance and the cache, tokens in openarc inspect
B4	benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 4	examples couldn't say "throws": 5 of 11 binary-search and about 8 of 23 wordy tests were unconstrainable	{ args, throws: true | "substring" } instead of expected (type, harness, failure messages, context package)
B5	benchmark 2026-10-31 (openarc-wt-bench bench/results/2026-10-31-untuned-ollama-*/summary.md), finding 6 and the pilot (OpenArc 0/5 against a plain-prompt baseline at 2/5 on qwen3.5:9b)	every call asked for code as an escaped JSON string under grammar-constrained decoding, while the baseline got free-form markdown: a confound the pilot couldn't separate	outputFormat: "code" per backend or region: one fenced block, leading imports parsed, no schema sent, then exactly the same guards; in the fingerprint and provenance. Default stays "json" until the ablation is run
—	0.8.0 structured specs (docs/design/structured-specs.md)	Everything a developer knew about how a function works had to be prose in intent, which isn't verified. compile.run's block intents were code inside strings, retyped by a model; three such blocks were cached as verified while doing the wrong thing. There was no way to say "all auth strategies obey these rules" or "a form with these fields wired to this action"	sketches (arc.hole<T>() in defineRegion({ sketch }), containment OARC-E099, per-hole coverage, OARC-E100), concepts (defineConcept, laws as examples, waive), TSX UI specs (a static JSX reader, a DOM harness in the sandbox with the project's jsdom). The prototype found that only containment catches a "helpful" whole-body edit that hides user enumeration: it passes tsc. The palette (auth, load balancer, p2p, login page) is built from them; its laws catch a plain === password compare that every behavioural example passes
—	0.8.0 palette build	a region whose body started with a blank line lost export when the engine rewrote it: the exported check read only the first body line	the check reads the first non-blank line (regression in tests/benchfix.check.ts)
—	0.9.0 code-first specs (docs/design/code-first-specs.md, issue #13)	A region was a configuration object: prose intent in a string, JSON examples, and the function as a separate export const between markers. A class member couldn't be a region (OARC-E033), so stateful code was bent into pure functions with their state as an argument, and JSON examples couldn't construct an object and then make a sequence of calls	implicit regions: a pending body (return arc.todo();) in a function, method, getter or arrow is a region, its signature the contract and its JSDoc the intent; markers inside the body at the first commit, so hashing, the splice guard and ownership apply unchanged; class members verified jointly; arc.test, arc.test.hidden, arc.law with a static plan and a registry check in the sandbox; fakes by type; @arc tags, arc.configure and defineRegion by id with OARC-E013; OARC-E022, OARC-E064, OARC-W023
—	0.9.0 build	a classic scaffold placeholder, export const f: F = (n) => arc.todo("id") between markers, has exactly the shape of a pending code-first arrow, and was claimed twice (OARC-E011 in seven checks)	a declaration inside a marker pair is never code-first
—	0.9.0 build	@arc calls this.net.send made this.backends.find(…) an OARC-E098: Backend[] is Array, which is an interface in the lib	only fields typed with a project interface are restricted roots
—	0.9.0 build	a test registered in a loop, outside the static plan, was caught (test-registry) but then retried by the model three times, though no body can fix it	a registry mismatch stops the region at once, like OARC-E100
—	0.9.0 build	openarc init writing a starter into src/ would add a region to every script that runs init and then writes its own src/ (seven checks do)	the starter is written at a terminal or with --example
—	0.9.0 scope review	A bundled palette and a built-in UI vocabulary would have to grow into a component library, which nobody asked for, while what users need is the machinery to write their own	the bundled palette (auth, load balancer, p2p, login page, tools/palette.ts, examples/palette-demo) and the open-arc/ui vocabulary are deprecated and frozen: they keep working and shipping, each export carries @deprecated. Concepts, laws, variants, fixtures, waive, sketches, code-first classes and tests, arc.law and the TSX mechanism stay first-class; Palette has a worked cache concept, type-checked and run by tests/own-palette.check.ts
—	issue #19, ablations (qwen2.5-coder 7B)	retries at temperature 0 repeated the body (14 of 24); in 33 of 54 failed attempts helpers came before a whole declaration of the function, a TS2322; Rules and the Examples legend were sent in full for regions with no effects or imports	every retry varies sampling (0.4, then 0.7) and a second repeated reply stops the region without verifying it; helpers + the own declaration are unwrapped; a closed function region gets two short Rules lines and only the example conventions it uses (0.9.1)
—	issue #19, efficiency rerun (0.9.1)	still about 550 tokens and 2 s per task above the plain prompt at equal accepted-and-correct: no unit passed on attempt 2 or 3, 16 of 27 retries were repeats, every region was its own request with the instructions and types again, and resampling did as well as repairing	the first repeat stops the region; an example or test failure is resampled with a short note instead of the body; a file's independent function regions share one first request, verified and committed one by one (compilation.batch, 0.9.2)

The pattern: every serious bug was found by executing something (tsc, the scanner, the sandbox, the examples), never by re-reading prose. That's why the spec's tables are executable: the ownership table is ownership.detect's examples, and the sandbox command is sandbox.buildArgv's example.