Skip to content

benchmark: clear another entry's compose stack before starting one, and say why a stack failed - #1189

Merged
MDA2AV merged 1 commit into
mainfrom
fix/gateway-stale-stack-ports
Aug 16, 2026
Merged

benchmark: clear another entry's compose stack before starting one, and say why a stack failed#1189
MDA2AV merged 1 commit into
mainfrom
fix/gateway-stale-stack-ports

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 16, 2026

Copy link
Copy Markdown
Owner

production-stack failed on #1182 with nothing in the log but an exit code:

Container httparena-fulmine-production-stack-authsvc-1 Error dependency authsvc failed to start
dependency failed to start: container ...-authsvc-1 exited (101)
[FAIL] gateway compose up failed

What it was

101 is a Rust panic. I built frameworks/_shared/authsvc and ran it two ways:

clean host         → running,   "authsvc listening on 0.0.0.0:9090",  /_health 200
9090 already held  → exit 101,  "bind 0.0.0.0:9090: Address in use (os error 98)"

Byte-for-byte the same failure. Nothing about it is fulmine's — that PR is a one-line package.json bump, and authsvc is the shared sidecar fulmine, aspnet-minimal_nginx and sark-production all build.

Every gateway and production stack runs network_mode: host on the same fixed ports — edge 8443, authsvc 9090, server 8080 — so no two can coexist. gateway_up already runs down --remove-orphans, but scoped to this framework's own compose project, so a stack left behind by another entry, or by a run killed between profiles, survives it. The only broader sweep is the one benchmark.sh does once at startup.

The fix

_gateway_clear_stale() removes running containers belonging to any other httparena-* compose project just before up, naming what it removed:

[warn] another httparena compose stack is still up; removing it so this one can bind its ports
  stale: httparena-thirdfw-gateway-64-authsvc-1
  stale: httparena-otherfw-production-stack-authsvc-1

It matches on the compose project label, so the harness's own docker run sidecars — httparena-postgres, httparena-redis — carry no such label and are left alone. The listing is split on | rather than whitespace: an unlabelled container prints an empty field, and with whitespace splitting its name would shift into the label's position and match the httparena- test by accident.

And why it took a rebuild to diagnose

compose reports the exit code and nothing else, so the reason never reached the run log. _gateway_dump_logs() prints each container of the failed stack through the existing dump_container_logs(), so the next one names itself:

─── httparena-...-authsvc-1 — status=exited exit=101 oom=false error=
─── httparena-...-authsvc-1 — last 120 log lines ───
  | bind 0.0.0.0:9090: Address in use (os error 98)

Verified

Against real containers, not by inspection: two stale stacks from other projects are found, named and removed while an unlabelled sidecar survives; and a stack whose service exits 101 has its stderr printed.

Not changed

authsvc itself. Refusing to start when its port is taken is correct behaviour — the bug was the port still being taken. Making it retry or bind elsewhere would hide exactly the collision this fixes.

🤖 Generated with Claude Code

…nd say why a stack failed

production-stack failed on #1182 with nothing in the log but an exit code:

    Container httparena-fulmine-production-stack-authsvc-1 Error dependency authsvc failed to start
    dependency failed to start: container ...-authsvc-1 exited (101)
    [FAIL] gateway compose up failed

101 is a Rust panic. Building frameworks/_shared/authsvc and running it twice
reproduces it exactly: on a clean host it logs "authsvc listening on
0.0.0.0:9090" and answers /_health 200; with 9090 already held it exits 101 with
"bind 0.0.0.0:9090: Address in use (os error 98)".

Nothing about that is fulmine's. authsvc is the shared sidecar every
production-stack entry builds, and every gateway and production stack runs
network_mode: host on the same fixed ports - edge 8443, authsvc 9090, server
8080 - so no two can coexist. gateway_up already runs `down --remove-orphans`,
but scoped to this framework's own compose project, so a stack left behind by
another entry, or by a run killed between profiles, survives it. The only
broader sweep is the one benchmark.sh does once at startup.

_gateway_clear_stale() now removes running containers belonging to any other
httparena-* compose project just before `up`, naming what it removed. It
matches on the compose project label, so the harness's own `docker run`
sidecars - httparena-postgres, httparena-redis - carry no such label and are
untouched. The listing is split on `|` rather than whitespace: an unlabelled
container prints an empty field, and with whitespace splitting its name would
shift into the label's position and match the httparena- test by accident.

The second half is why this took a rebuild to diagnose at all. compose reports
the exit code and nothing else, so the reason never reached the run log.
_gateway_dump_logs() prints each container of the failed stack through the
existing dump_container_logs(), so the next failure names itself:

    ─── httparena-...-authsvc-1 — status=exited exit=101 oom=false error=
    ─── httparena-...-authsvc-1 — last 120 log lines ───
      | bind 0.0.0.0:9090: Address in use (os error 98)

Verified against real containers: two stale stacks from other projects are
found, named and removed while an unlabelled sidecar survives, and a stack
whose service exits 101 has its stderr printed.

Leaves authsvc alone. Refusing to start when its port is taken is correct; the
bug was the port still being taken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant