Spec and first two build plans for a shared factorio-oracle CLI - #84
Merged
Conversation
Four repos ask the real game questions, and each wrote the same plumbing. Measured: about 1,320 lines of near-identical wiring across 23 copies, before counting FactorioTools' own share. 14 of 18 blueprint-editor probes hardcode a factorio_version that silently skips the mod when it is wrong, only 1 of 18 checks the binary exists, and none of the three repos has a timeout. The design is a Rust CLI, JSON in and JSON out. It owns the plumbing and not the questions, because a probe compares the game against the consumer's own reimplementation - so analysis has to stay in the consumer's language. That also rules out a shared probe framework in any single language. Records what reading the repos changed: five run modes rather than one, a per-mode success predicate, the runner returning a work directory instead of a dump, Lua passed in opaque, and reading factorio-data at a tag rather than checking one out in a clone three repos share. Spec lives here because this repo has the specs convention and the oracle repo does not exist yet. Move it there when it does. Refs #82, #83.
Eleven TDD tasks from an empty directory to a working 'run' subcommand: crate skeleton with Renovate in the first commit, version parsing, install layout resolution, discovery, the probe spec types, Lua literal injection, mod scaffolding, per-mode argument vectors, the per-mode success predicate, the spawn boundary with a timeout, and the wiring plus a fake game. Records a conflict plan 2 has to handle: the spec wants a byte-for-byte reproduction of the committed fixture, but #83 shows that fixture's directions come from reading a documentation index as if it were a value. The port has to reproduce the bug first, then fix it as a visible diff. Refs #82, #83.
Contamination reporting is now on by default, and the mechanism changed: script.on_init takes exactly one handler, so a prelude using it would be silently replaced by the consumer's own - which 17 of 18 blueprint-editor probes register. A self-cancelling on_nth_tick collides with nothing. Adds the f32 round-trip rule the map repo asked for, with its evidence: two candidate kernels shared an identical worst absolute error of 2.682e-7 and differed by 42 exact matches of 512, so an error bound could not tell them apart. Recorded as a test in a new task rather than a comment, because a capture that loses precision still looks fine. --instrument-mod is confirmed real in the 2.1.14 binary and gets a spike before plan 2. Records MapWebUI#234 as the first customer.
The map seed can reach the game twice: inside the map-gen settings JSON and as --map-gen-seed. MapWebUI sets both from the same variable, so they have never disagreed - and nothing establishes which one the game honours. Collapsing to one channel would be the bad kind of hazard. If the flag beat the file, a CLI writing only the file would generate a different map from the same request, with nothing erroring. That repo already paid for a seed-provenance mistake once: a correct field scored 0.5% overlap against the wrong convention where the right one scored 99.9%. Taking one seed field and writing both channels makes the design correct whichever wins, and makes disagreement structurally impossible. Also softens the --map-gen-settings claim from 'required' to 'always passed', which is what is actually established. Nobody has run --create without it.
FactorioMapWebUI ran the probe: a mod reading back game.surfaces[1].map_gen_settings.seed, four argument vectors, file seed 111111 against flag seed 222222. --map-gen-settings is NOT required. Two arms generated a map, loaded the mod and produced a dump with no settings file at all. 'Always passed' was habit in the consumer repos, not a rule of the game, so the CLI now passes it only when a caller supplies one. --map-gen-seed OVERRIDES the file's seed: file 111111 plus flag 222222 gives a surface seed of 222222, and a file-only arm rules out the file being ignored in general. So writing both channels from one field is now the only shape that cannot bite - a tool writing only the file would be silently overridden by a caller's flag. Also records what that measurement found in MapWebUI itself: its settings file's seed field has never done anything, because the harness always passes the flag too. No fixture is wrong, but mapGenOverrides silently discards a seed passed through it. Tracked on that repo's #232.
Ran a four-file probe mod on 2.1.14 logging from every stage. --instrument-mod is real and gives earlier hooks: instrument-data.lua at 0.045s against data.lua at 0.129s, and instrument-control.lua toplevel before control.lua toplevel. Without the flag neither instrument file loads at all. But instrument-control.lua's script.on_init handler NEVER fired, because control.lua registered one too and the later registration replaced it. So Instrument Mode hands a probe an earlier hook whose event registration the consumer then silently destroys. Not adopting it. That also turns the single-handler rule from an inference into a measurement, which matters because the whole contamination design rested on it. The better answer came out of the same run: helpers.write_file works at plain control.lua toplevel with no event, and script.active_mods is populated there and includes the probe's own mod. So the prelude is now one line with no event registration and no collision surface, replacing the self-cancelling on_nth_tick. A test asserts the absence of on_init, on_nth_tick and on_event so it cannot regress. Also points the new repo at FactoryGameFan, following today's move. Refs #82.
Two corrections from the FactorioMapWebUI session. The toplevel write is for metadata, not sampling. game.surfaces[1] does not exist at control-stage toplevel, so anything calling calculate_tile_properties or get_tile still needs on_init. Those registrations are load-bearing rather than habit, and the two requirements compose: the prelude registers nothing, a sampling probe registers exactly one on_init, and nothing contests the slot. The spec now says not to read the finding as 'probes should stop using on_init', because that is the wrong lesson the earlier wording invited. Instrument Mode is actively dangerous rather than merely unhelpful: a tool built on it works on every probe that does not register on_init and fails silently on every probe that does. Keeping the entry despite not adopting the feature, since a measured destruction of the handler is a stronger claim than an inference from the docs.
The same document already said, under Repo setup, that all six repos moved to FactoryGameFan. Open question 1 still called it undecided. Record the answer rather than deleting the question. GitHub documents what happens to a fork on deletion and on detachment, but never on transfer, and that gap was the stated blocker. A transfer rewrites a cross-repo pull request's head to the new owner and keeps the pull request - verified on PR #10, which now reads head: FactoryGameFan:main and is still closed with both comments intact. It was closed before the transfer, so this covers the closed case only. Also note the two things that bite afterwards: the transfer API is asynchronous, and old wormeyman URLs 301 correctly, so stale references survive without complaining. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jij2RRpAh3qNARvhxM3PUP
Build-order steps 1 to 3 are built, and running them against a real 2.1.14
install contradicted this document three times.
The sentinel is on stdout. Factorio writes nothing to stderr at all: a
control-stage error("DUMPED-OK"), a data-stage error and an unknown command
line flag all printed to stdout with stderr at zero bytes. sentinelSeen was
false on every real run, which is worse than not reporting it, because a
create run keys success off the dump existing and the sentinel is the only
thing that catches a probe which wrote its dump and then died.
Omission in mod-list.json means enabled. The game rewrites the file and adds
back every bundled mod it does not mention. A file naming only base came
back naming five, and all five loaded. A control arm settled both halves at
once: three mods listed as disabled stayed out, while recycler, unmentioned
in the same file, was added and loaded. So an empty mod directory keeps out
user mods and nothing else.
Both defects were invisible to sixty unit tests, because the fake game
encoded the same wrong belief as the code. That is the part worth carrying
into the remaining plans, so the testing section now says it outright.
Two claims survived and are now measured: --dump-data does honour write-data
for script-output, which closes an open item, and --create still needs no
settings file.
The plan's task bodies are left alone, since they record what was planned. A
corrections section names the three tasks that carry a wrong premise, and
flags for plan 2 that the byte-for-byte fixture comparison only holds
against the default mod set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jij2RRpAh3qNARvhxM3PUP
Covers build-order step 4, which is the gate on whether the shared tool can replace tools/capture-factorio-oracle.sh. Written after spiking the risky parts rather than guessing at them, so four things the plan would otherwise have had to hedge on are settled: The capture half is already done. A dump-data run through the new tool produced a 28 MB data-raw-dump.json in 2.9 seconds, and feeding that to the existing Python trimmer reproduced the committed fixture byte for byte. Only the trim half is left. The port is achievable. A sixty-line Rust spike over the real dump produced entities and modules blocks byte identical to the fixture. The float formatting the spec called the trap is not one here: serde_json's printer emits shortest round-trip, same as Python's repr. But serde_json's number PARSER is one ULP out on Factorio's long decimal expansions, in both directions, while Rust's own f64::from_str agrees with CPython bit for bit. Round-tripping the whole 25 MB dump found 9,744 differing lines, none in a field the fixture keeps. So it is latent, and cheap to fix now. Task 4 does. captureInfo.loadedMods cannot come from the active-mods prelude, because the fixture lists core and script.active_mods never reports it. Task 7 reads it off the game's stdout instead. The #83 conflict plan 1 flagged is handled by ordering: port the order bug first, prove the port with the acceptance test, then fix it as its own change. A create probe already showed the game agrees with order today, so that fix provably changes no bytes - and an acceptance test asserts it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jij2RRpAh3qNARvhxM3PUP
wormeyman
changed the title
The table said four sources and listed data/changelog.txt. The capture reads three, and there is no changelog handling anywhere in tools/. The changelog is a research source a human reads; it had been written into the table as though it were automated. Records that FactoryGameFan/factorio-oracle now reproduces this fixture byte for byte, with an acceptance test that proves it against a real 2.1.14 install. The script stays: the agreed migration rule is new probes only. Two things that tool measured, written down where they matter. An empty mod directory keeps out user mods and nothing else - Factorio rewrites mod-list.json and re-enables every bundled mod the file omits, so the base-only list this script writes still loads all six. And the fixture's directions come from `order`, which is a documentation index rather than the runtime value, so a note now sits beside the runtime-api.json row instead of that row implying it publishes numbers. Issue #83. --help printed a hardcoded line range, so growing the header truncated it mid-sentence. It now prints the whole leading comment block. Verified: --check still reports the fixture up to date against 2.1.14, and --help reaches the Usage section again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jij2RRpAh3qNARvhxM3PUP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design docs only. No code here, and nothing in
tools/orsrc/changes.Four repos now run Factorio headless to ask it questions, and each wrote the same
plumbing from scratch. This branch writes down the shared Rust CLI that replaces
that plumbing, plus the first two of four build plans. Tracking issue: #82.
The tool itself now exists at FactoryGameFan/factorio-oracle.
Plan 1 is built and CI is green there.
What is in here
docs/superpowers/specs/2026-08-16-shared-factorio-oracle-design.md- the design.Commands, the five run modes, the probe spec, the output contract, determinism
rules, provenance, testing, repo setup, prior art, build order, and a decision log.
docs/superpowers/plans/2026-08-16-factorio-oracle-runner-core.md- plan 1,build-order steps 1 to 3: the repo skeleton,
installs list, andrun. Built.docs/superpowers/plans/2026-08-17-factorio-oracle-trimmer.md- plan 2, thetrimmer and the byte-for-byte acceptance test. Written, not started.
Three things the first real run disproved
Plan 1 was built, then run against a real 2.1.14 install. Three claims in this
design turned out to be wrong, and all three had passed sixty unit tests.
error("DUMPED-OK"), a data-stage error, and an unknown command line flag allprint to stdout and leave stderr at zero bytes. The spec said stderr, so
sentinelSeenwas false on every real run.mod-list.jsonmeans enabled. The game rewrites the file andadds back every bundled mod it does not mention. A file naming only
basecameback naming five, and all five loaded. An explicit
enabled: falseis honoured,proven in one run where three named-disabled mods stayed out while
recycler,unmentioned in the same file, was added and loaded.
Two of the three were invisible to the unit tests by construction: the fake game
encoded the same wrong belief as the code. The testing section now says so
outright, and the integration test that caught them is no longer optional.
What plan 2 rests on, measured rather than assumed
The risky parts were spiked before the plan was written.
capture-factorio-oracle.shis already replaced. Adump-datarun through the new tool produced a 28 MB dump in 2.9 seconds, andfeeding that to the existing Python trimmer reproduced the committed fixture
byte for byte.
entitiesandmodulesblocks byte identical to the fixture, so the port is achievable.
serde_json's number parser is one ULP out on Factorio's long decimalexpansions, in both directions, while Rust's own
f64::from_stragrees withCPython bit for bit. Round-tripping the whole 25 MB dump found 9,744 differing
lines, none in a field the fixture keeps. Latent, and cheap to fix now.
captureInfo.loadedModscannot come from the active-mods prelude, because thefixture lists
coreandscript.active_modsnever reports it.A bug found on the way
runtime-api.jsondoes not contain the values ofdefines. Across all 1,554entries the only keys are
name,orderanddescription.trim-factorio-oracle.py:150readsorderas if it were the value, which iscorrect today only because Factorio declares directions clockwise from
north = 0. Tracked as #83.A
createprobe has now read the real table out of the running game:north=0, east=4, south=8, west=12. It matches, so fixing #83 provably changes nofixture bytes. Plan 2 handles it by ordering - port the bug, prove the port, then
fix it as its own reviewable change with a test asserting the output did not move.
Still to come
Plans 3 and 4:
provenance checkwith theunknownratchet, andrefsplus thethree knowledge documents.