Add a swarmforge package and move the YAML parser into it - #33
Merged
Conversation
scripts/ holds flat modules that load each other by file path, and the one genuinely shared primitive -- the YAML subset parser -- lives in anvil/, the directory whose whole purpose is "files COPY'd into the image". So the host launcher imports out of the build context to borrow it, and nothing can be type-checked, linted for import errors, or mocked through ordinary means. Add an empty package at the repo root to hold that shared code. Both sides can reach it: the launcher imports it from the checkout, and the image copies it in. pyproject.toml exists so editors, type checkers and linters can resolve the imports; it is not a deployment mechanism, and its dependency list is empty and meant to stay that way. The harness image installs no third-party Python and the launcher runs on whatever python3 the host provides, so stdlib-only is a constraint, not a preference.
The parser is not part of agent translation. It reads the nested-map and flat-list subset that agent frontmatter and tong definitions both use, and it only lives in the translator because that is where it was first needed -- which is why the host launcher reaches into the image's build context to get at it. Move parse_map and its helpers to swarmforge/yamlite.py verbatim and import them back. It is a leaf module: it imports nothing else from the package, so either side of the container boundary can depend on it without dragging the rest along.
The launcher loaded anvil/translate_agents.py by file path purely to borrow parse_map, so a host-side module depended on a container-side one and paid for it with a spec_from_file_location block. Import swarmforge.yamlite instead. The launcher runs out of a checkout rather than an installed distribution, so it still has to put the repo root on sys.path first -- but that is one line in one place, and it prepends so the checkout's own package wins over any stale copy installed on the host.
The container-side translator now imports swarmforge.yamlite, and the tests load it straight off disk, so the suite only passed because make happened to run from the repo root and put it on sys.path. Say it outright with PYTHONPATH instead of relying on the working directory -- which is also how the image supplies the same import root.
The translator now imports the shared package, and a build whose context is anvil/ cannot see it -- COPY has no way to reach a sibling directory. Build with an explicit `-f anvil/Dockerfile` from the root instead and copy the package in beside the scripts that import it, at the import root the entrypoint hands the translator. A root context would otherwise sweep in ollama state, checkouts and editor droppings, so .dockerignore admits only the two directories the Dockerfile reads from; everything else would slow the build and bust its cache for unrelated reasons. The build recipes had no test coverage at all, which is uncomfortable now that the Dockerfile and the context have to agree. Two go in: one on the argv the recipes assemble, one that stages the image's layout and translates an agent through it with the checkout off the path, so a translator that only imports because someone ran it from the repo does not pass.
scripts/test_translate_agents.py loads anvil/translate_agents.py off disk, and that module now imports the shared package. Run under the make target it works, because the target puts the repo root on PYTHONPATH; run the way its own docstring says to -- and the way every other test file in scripts/ still can -- it died with ModuleNotFoundError. Stand in for the image's PYTHONPATH in the test's loader preamble, the same one line tongs.py uses. No test body changes. Two more, found alongside: run_anvil.py described its path loading as "the same way tongs.py loads translate_agents.py", which is exactly the block this branch deleted. And the Dockerfile's copy destination and the entrypoint's import root are two strings in two files with nothing tying them together -- a mismatch does not fail the build, it just stops translating agents at runtime, so assert they agree.
The PYTHONPATH check matched the string anywhere in the entrypoint, so moving it off the translator invocation onto some other line left the test green. Anchor it to the invocation. The Dockerfile reader also dropped any COPY carrying a flag, and the lookup that followed then failed with a bare KeyError. Skip flag words and say which COPY line is missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
scripts/tongs.pyis a host-side launcher, and it loadsanvil/translate_agents.pyby file path — a module whose whole directory exists to beCOPY'd into the harness image. It does that to borrow one function,parse_map. The layering runs backwards, and the price is aspec_from_file_locationblock that nothing can type-check, lint for import errors, or mock through ordinary means.The parser is the genuinely shared piece. It reads the nested-map / flat-list subset that agent frontmatter and tong definitions both use, and it only lives in the translator because that is where it was first needed.
What changed
swarmforge/package at the repo root for code shared across the container boundary, with apyproject.tomlso editors and linters can resolve the imports.dependenciesis empty and meant to stay that way: the image installs no third-party Python and the launcher runs on whateverpython3the host has.parse_mapand its helpers move verbatim toswarmforge/yamlite.py— a leaf module that imports nothing else, so either side of the boundary can depend on it alone. The translator imports them back; the launcher imports them directly and drops its path load.-f anvil/Dockerfile. A build whose context isanvil/cannot see the package —COPYhas no way to reach a sibling directory. A.dockerignoreadmits only the two directories the Dockerfile reads from, so a root context does not sweep in ollama state or checkouts.PYTHONPATHrather than relying on make's working directory, andscripts/test_translate_agents.pystands in for that itself so it still runs standalone the way its docstring says.Behavior
None intended. The moved functions are byte-identical to their originals, the translator's remaining code is untouched apart from the import, and inside the image every path is where it was — only the
COPYsources gained ananvil/prefix.The launcher's
docker runargv is unchanged, enforced by the passthrough tests inscripts/test_run_anvil.pyand the recipe-driven tests inscripts/test_run_agent_container.py; both are untouched and green. The suite went from 473 tests to 478.New coverage
The build recipes had no tests at all, which is uncomfortable now that the Dockerfile and the build context have to agree.
scripts/test_image_layout.pyadds three things: the argv the recipes assemble, a check that the Dockerfile's copy destination and the entrypoint's import root still name the same directory, and a run of the translator against the image's directory shape with the checkout deliberately off the path — so a translator that only imports because someone ran it from the repo does not pass.Not verified here
make build_opencodeand a no-tongsmake run_opencodewere not run: docker is not available in the environment this was written in. The tests above cover the wiring, but the real build is worth doing before merge.