The gap
seamless verify stands up a stack and runs a Playwright matrix against it, but that stack is built
from verify/docker-compose.verify.yml and the harness's own adapter apps. Nothing ever executes
what seamless init writes to disk. The generated artifacts are asserted only as strings, by unit
tests that compare them to expected substrings.
That is not a theoretical hole. #164 pinned postgres:18 while leaving the generated volume mount at
/var/lib/postgresql/data, which PostgreSQL 18+ ignores: the scaffolded database restart-looped and
never came up. It passed Build and unit test, passed verify / verify, was reviewed, and merged.
The conformance job missed it because the harness's postgres service mounts no volume at all, so it
exercised the new in-image default rather than the broken mount. It was found only when the same bump
was attempted in seamless-templates and someone ran docker compose up by hand (#165).
Everything init generates is in this blind spot:
docker-compose.yml (src/generators/docker/docker.ts) — services, images, mounts, healthchecks,
published ports, service dependencies
auth/.env (src/generators/auth/auth.ts) — the auth server's environment, including generated
secrets
- each template's
.env (applyTemplateEnv) — the resolved {{placeholder}} contract
seamless.config.json (src/generators/config/config.ts)
A unit test proves the file says the right thing. It cannot prove Docker accepts it, that the
services reach each other, or that a volume is mounted where the image actually stores data.
Proposal: a scaffold smoke job
Add a CI job that treats the scaffold as the artifact under test:
seamless init smoke --local --yes --email=... into a temp dir (this is exactly what --yes
was added for, so no pty or expect script is needed).
docker compose up -d --wait in the generated project.
- Assert the stack is actually working, not merely started.
docker compose down -v.
Step 3 is the part worth getting right. "Container started" is too weak: with the wrong volume mount
the database starts and serves queries, it just writes to the container layer instead of the
volume. The check that catches it is write a row, recreate the container, read it back. Worth
doing for the database specifically, since silent non-persistence is the failure mode a naive smoke
test misses.
seamless check cannot be the assertion as it stands
The obvious move is to reuse seamless check as step 3, and it does check the right things
(structure, Docker, compose, containers, health). But it always exits 0:
$ seamless check # in an empty directory
✖ seamless.config.json not found
→ Run: seamless init
exit=0
$ seamless check # scaffolded project, stack down
✖ API not reachable
✖ Auth not reachable
✖ Console not reachable
Check complete.
exit=0
So it cannot gate CI, and anyone scripting against it today gets no signal either. That is arguably
its own bug: check is the command you would reach for in a health-check script. Either give it a
non-zero exit when a check fails (possibly behind --strict to avoid breaking anyone parsing the
current behaviour), or have the smoke job assert independently.
Cost and practicalities
- Needs Docker. CI already provisions it for
verify / verify.
- The generated compose publishes fixed ports (5432, 5312, 3000, 5173, 5174) on loopback, which
collide with the verify stack. Run the jobs on separate runners, or serialize them.
- Building the scaffolded
api and web services means an npm install per run. If that is too
slow, a first cut could bring up only the services with healthchecks (db, auth) and still catch
the entire class of bug that motivated this.
--admin=none --auth=docker keeps the first version small.
Acceptance
- CI scaffolds a project and brings its generated compose up on every PR.
- The database is asserted to persist across a container recreate, not merely to start.
- A deliberate regression (reverting the mount path in
docker.ts to /var/lib/postgresql/data)
fails the job. Worth confirming once when the job is written, since a smoke test that cannot fail
is worse than none.
- Decide the
seamless check exit-code question, here or in a linked issue.
The gap
seamless verifystands up a stack and runs a Playwright matrix against it, but that stack is builtfrom
verify/docker-compose.verify.ymland the harness's own adapter apps. Nothing ever executeswhat
seamless initwrites to disk. The generated artifacts are asserted only as strings, by unittests that compare them to expected substrings.
That is not a theoretical hole. #164 pinned
postgres:18while leaving the generated volume mount at/var/lib/postgresql/data, which PostgreSQL 18+ ignores: the scaffolded database restart-looped andnever came up. It passed
Build and unit test, passedverify / verify, was reviewed, and merged.The conformance job missed it because the harness's postgres service mounts no volume at all, so it
exercised the new in-image default rather than the broken mount. It was found only when the same bump
was attempted in
seamless-templatesand someone randocker compose upby hand (#165).Everything
initgenerates is in this blind spot:docker-compose.yml(src/generators/docker/docker.ts) — services, images, mounts, healthchecks,published ports, service dependencies
auth/.env(src/generators/auth/auth.ts) — the auth server's environment, including generatedsecrets
.env(applyTemplateEnv) — the resolved{{placeholder}}contractseamless.config.json(src/generators/config/config.ts)A unit test proves the file says the right thing. It cannot prove Docker accepts it, that the
services reach each other, or that a volume is mounted where the image actually stores data.
Proposal: a scaffold smoke job
Add a CI job that treats the scaffold as the artifact under test:
seamless init smoke --local --yes --email=...into a temp dir (this is exactly what--yeswas added for, so no pty or expect script is needed).
docker compose up -d --waitin the generated project.docker compose down -v.Step 3 is the part worth getting right. "Container started" is too weak: with the wrong volume mount
the database starts and serves queries, it just writes to the container layer instead of the
volume. The check that catches it is write a row, recreate the container, read it back. Worth
doing for the database specifically, since silent non-persistence is the failure mode a naive smoke
test misses.
seamless checkcannot be the assertion as it standsThe obvious move is to reuse
seamless checkas step 3, and it does check the right things(structure, Docker, compose, containers, health). But it always exits 0:
So it cannot gate CI, and anyone scripting against it today gets no signal either. That is arguably
its own bug:
checkis the command you would reach for in a health-check script. Either give it anon-zero exit when a check fails (possibly behind
--strictto avoid breaking anyone parsing thecurrent behaviour), or have the smoke job assert independently.
Cost and practicalities
verify / verify.collide with the verify stack. Run the jobs on separate runners, or serialize them.
apiandwebservices means annpm installper run. If that is tooslow, a first cut could bring up only the services with healthchecks (
db,auth) and still catchthe entire class of bug that motivated this.
--admin=none --auth=dockerkeeps the first version small.Acceptance
docker.tsto/var/lib/postgresql/data)fails the job. Worth confirming once when the job is written, since a smoke test that cannot fail
is worse than none.
seamless checkexit-code question, here or in a linked issue.