Skip to content

test: nothing executes what init generates, so a broken scaffold passes CI #166

Description

@Bccorb

The gap

seamless verify stands up a stack and runs a Playwright matrix against it, but that stack is built
from verify/docker-compose.verify.yml and the harness's own adapter apps. Nothing ever executes
what seamless init writes to disk.
The generated artifacts are asserted only as strings, by unit
tests that compare them to expected substrings.

That is not a theoretical hole. #164 pinned postgres:18 while leaving the generated volume mount at
/var/lib/postgresql/data, which PostgreSQL 18+ ignores: the scaffolded database restart-looped and
never came up. It passed Build and unit test, passed verify / verify, was reviewed, and merged.
The conformance job missed it because the harness's postgres service mounts no volume at all, so it
exercised the new in-image default rather than the broken mount. It was found only when the same bump
was attempted in seamless-templates and someone ran docker compose up by hand (#165).

Everything init generates is in this blind spot:

  • docker-compose.yml (src/generators/docker/docker.ts) — services, images, mounts, healthchecks,
    published ports, service dependencies
  • auth/.env (src/generators/auth/auth.ts) — the auth server's environment, including generated
    secrets
  • each template's .env (applyTemplateEnv) — the resolved {{placeholder}} contract
  • seamless.config.json (src/generators/config/config.ts)

A unit test proves the file says the right thing. It cannot prove Docker accepts it, that the
services reach each other, or that a volume is mounted where the image actually stores data.

Proposal: a scaffold smoke job

Add a CI job that treats the scaffold as the artifact under test:

  1. seamless init smoke --local --yes --email=... into a temp dir (this is exactly what --yes
    was added for, so no pty or expect script is needed).
  2. docker compose up -d --wait in the generated project.
  3. Assert the stack is actually working, not merely started.
  4. docker compose down -v.

Step 3 is the part worth getting right. "Container started" is too weak: with the wrong volume mount
the database starts and serves queries, it just writes to the container layer instead of the
volume. The check that catches it is write a row, recreate the container, read it back. Worth
doing for the database specifically, since silent non-persistence is the failure mode a naive smoke
test misses.

seamless check cannot be the assertion as it stands

The obvious move is to reuse seamless check as step 3, and it does check the right things
(structure, Docker, compose, containers, health). But it always exits 0:

$ seamless check          # in an empty directory
✖ seamless.config.json not found
→ Run: seamless init
exit=0

$ seamless check          # scaffolded project, stack down
✖ API not reachable
✖ Auth not reachable
✖ Console not reachable
Check complete.
exit=0

So it cannot gate CI, and anyone scripting against it today gets no signal either. That is arguably
its own bug: check is the command you would reach for in a health-check script. Either give it a
non-zero exit when a check fails (possibly behind --strict to avoid breaking anyone parsing the
current behaviour), or have the smoke job assert independently.

Cost and practicalities

  • Needs Docker. CI already provisions it for verify / verify.
  • The generated compose publishes fixed ports (5432, 5312, 3000, 5173, 5174) on loopback, which
    collide with the verify stack. Run the jobs on separate runners, or serialize them.
  • Building the scaffolded api and web services means an npm install per run. If that is too
    slow, a first cut could bring up only the services with healthchecks (db, auth) and still catch
    the entire class of bug that motivated this.
  • --admin=none --auth=docker keeps the first version small.

Acceptance

  • CI scaffolds a project and brings its generated compose up on every PR.
  • The database is asserted to persist across a container recreate, not merely to start.
  • A deliberate regression (reverting the mount path in docker.ts to /var/lib/postgresql/data)
    fails the job. Worth confirming once when the job is written, since a smoke test that cannot fail
    is worse than none.
  • Decide the seamless check exit-code question, here or in a linked issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions