preview: the container NEVER sleeps - node is PID 1 and ignores SIGTERM (the "~8.5min idle tail" was a metrics backfill artifact)

[wormeyman]

The title and the premise below were REFUTED - corrected 2026-08-05 (901e832, #155)

There is no idle tail. The container never slept at all. Node runs as PID 1 under the
Dockerfile's exec-form ENTRYPOINT, Linux gives PID 1 no default signal dispositions, and
@cloudflare/containers stops an idle instance with SIGTERM and never escalates to
SIGKILL - so every stop request was silently discarded. The instance only ever went away
when a deploy replaced the placement.

The "~8.5 minutes" was a metrics backfill artifact. containersUsageAdaptiveGroups
backfills, and that figure came from reading the 5-minute buckets ~14 minutes after a test
render and treating the missing tail as sleep. Re-read once settled, the same window has
every bucket present and the placement stayed allocated 29.3 hours straight - 352 of
352 buckets, no gaps - on a total of 5 worker requests. Wait at least an hour before
reading absence as sleep, and confirm with placementId continuity, never bucket presence.

Also refuted: the response-drain fix (2026-07-29) did not stop this on its own. Billing
shows 100% allocation every full day through 2026-08-03, including the five days after it
deployed. Keep the drain guard - the hazard is real - but do not credit it with the bill.

Neither fix is deployed as of 2026-08-05. See #168.

The original report follows unchanged.

---

#116 fixed the container staying awake forever. It now sleeps - but ~25x later than configured. Costs almost nothing at current traffic; worth understanding before traffic grows.

Measured

One POST /preview fired against production on 2026-08-03 (worker version 2e67470f, container app a039bbb1-c8a4-4ecb-b6f5-bb3a1209a093), with a fresh seed to miss the R2 cache:

fired    22:39:01Z
returned 22:39:11Z   status=200  10.59s  350,848B image/png

Billing buckets from containersUsageAdaptiveGroups (a 1 GiB instance awake for a whole 5-minute bucket = 300 GiB-seconds):

bucket	GiB-seconds
22:35Z	35
22:40Z	455
22:45Z	170
22:50Z	absent - asleep

Awake roughly 22:39:00 -> 22:47:50, about 8.5 minutes, for a 10.6-second render. sleepAfter is "20s" in preview-service/worker/src/container.ts. Total billed: 660 GiB-seconds for one render.

Two things this same test confirmed, both good, both worth recording here because they were open questions:

• basic (1 GiB) is genuinely live and sufficient. The disk-to-memory ratio reads 4.0 (1 GiB / 4 GB); every pre-deploy sample read 2.0 (standard-1, 4 GiB / 8 GB). A full 1024px Nauvis render completes inside it, cold start included.
• Sleeping works at all. Before this test, 21:40Z -> 22:35Z shows no buckets whatsoever - a full hour of zero allocation. That is what fix(preview): stop the container from being billed 24/7 #116 bought.

Why this is probably NOT the bug #116 fixed

The inflight-counter leak (isActivityExpired() returning false while inflightRequests > 0) makes a container sleep never, not late. A leaked counter has no timeout to expire. So this is a different mechanism.

The likely candidate is scheduling granularity in @cloudflare/containers' own alarm loop rather than anything in this repo. In dist/lib/container.js the loop computes minTime = Math.min(minTimeFromSchedules, minTime, this.sleepAfterMs) where minTime is already clamped to a coarse interval (the comment there reads Math.min(3m or maxTime, sleepTimeout)), then re-arms via setAlarm. If the effective floor is minutes, a 20s sleepAfter cannot be honoured no matter what it is set to. Unverified - it needs someone to actually trace the alarm scheduling rather than infer it from a comment.

Cost, so nobody over-reacts

At ~7 renders/day: 660 GiB-s x 7 x 30 / 3600 = 38.5 GiB-hours/month against 25 included in Workers Paid, so about $0.12/month. Against the ~$28/month #116 fixed, this is noise.

It stops being noise if traffic grows: the tail is a fixed ~8.5 minutes of 1 GiB per wake, so cost scales with the number of cold wakes, not with render work. At ~50 renders/day it is ~275 GiB-hours/month, or roughly $2.25. Requests that arrive inside an existing awake window are free, so bursty traffic is cheap and sparse traffic is not.

What would settle it

1. Fire two renders ~30s apart and confirm the second does not extend the tail (proves the timer is not being renewed by something in this repo's code path).
2. Set sleepAfter to something large, e.g. "10m", and check whether the tail changes at all. If the tail stays ~8.5 minutes regardless, the setting is inert and the floor is upstream - that is the decisive experiment and it is cheap.
3. Only then consider filing upstream at cloudflare/containers, with these numbers.

Do not "fix" this by lowering sleepAfter further. If step 2 shows the value is inert, changing it is theatre.

Reproducing the measurement

curl -s -X POST https://fmw-preview.wormeyman.workers.dev/preview \
  -H 'content-type: application/json' \
  -d '{"mapGenSettings":{"width":0,"height":0,"seed":<FRESH>},"planet":"nauvis","seed":<FRESH>,"size":1024}' \
  -o out.png -w 'status=%{http_code} time=%{time_total}s size=%{size_download}B\n'

Use a seed you have not used before, or R2 serves a cache hit and the container never wakes. Omitting the Origin header is deliberate and passes - the worker's check is if (origin && origin !== env.ALLOWED_ORIGIN).

Then read the 5-minute buckets from containersUsageAdaptiveGroups. Note that wrangler containers instances <ID> is not a reliable check here: it reported state: running with an 80-minute-old created timestamp during an hour when billing showed zero allocation. It describes the placement, not whether you are paying.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AB7J1qK6kSBnJJmgDqMst

[wormeyman]

Root cause found, and it is in this repo - not upstream (#154)

This issue's candidate was scheduling granularity in @cloudflare/containers' alarm loop, flagged UNVERIFIED with a note to trace it rather than infer it from a comment. Traced. That candidate is refuted, and the real cause is ours.

The alarm clamp is not it

minTime starts at Date.now() + 3 * 60 * 1000 and sleepAfterMs is also an absolute timestamp, so

minTime = Math.min(minTimeFromSchedules, minTime, this.sleepAfterMs);

picks the earlier of the two - a 20s sleepAfter beats the 3m ceiling. On expiry the loop calls onActivityExpired() then renewActivityTimeout(), i.e. it re-checks every 20s. There is no minutes-long floor. The // Math.min(3m or maxTime, sleepTimeout) comment reads like one and is not.

The cause

At the SDK's own sleepAfter declaration:

// The signal sent to the container by default is a SIGTERM.
// The container won't get a SIGKILL if this threshold is triggered.

onActivityExpired() calls stop(), which sends SIGTERM and never escalates. Our Dockerfile is ENTRYPOINT ["node", "/app/server.mjs"] - exec form, no shell, no init - so node runs as PID 1, and Linux gives PID 1 no default signal dispositions. server.mjs registered no handler, so the SIGTERM was silently discarded. The SDK never stopped the container at all; it lingered until the platform reclaimed it. That is the ~8.5 minutes.

Fixed in #154

SIGTERM/SIGINT handler, server.close() then exit(0) - close rather than exit outright so an in-flight render finishes streaming, since the Worker must drain the body for its own inflight counter to decrement.

Still open: production confirmation

Not measured yet. Closing this properly needs a container rebuild + preview:deploy, one render with a fresh seed, then a re-read of the containersUsageAdaptiveGroups buckets to confirm the tail collapses from ~8.5min toward 20s.

Step 2 of "What would settle it" is now unnecessary if this diagnosis is right - setting sleepAfter to "10m" tested whether the value was inert, and the trace above already shows it is honoured. It remains the fallback if the tail survives the fix.

Leaving this open until the deploy confirms it.

[wormeyman]

The tail is not 8.5 minutes. There is no tail - it never sleeps.

The ~8.5 minute figure in the issue body is an artifact of when the dataset was read, not of the container's behaviour. containersUsageAdaptiveGroups backfills, and that read happened ~14 minutes after the render, at which point 22:50Z had not landed yet. It was recorded as absent - asleep. It is there now, and so is everything after it.

The same placement is still awake 29.3 hours later

The render fired at 2026-08-03T22:39:01Z and woke placement a20877a2 (first bucket 22:35:00Z). Walking every five-minute bucket from there to 2026-08-05T03:50:00Z:

buckets present	352 of 352
gaps > 5 min	none
distinct placementId	1 (a20877a2)
distinct instanceId	1 (b72db0c9…)
allocated memory	103,575 GiB-seconds over 29.33 h
awake fraction, 1 GiB instance	0.981

And from workersInvocationsAdaptive, scriptName = fmw-preview, same window:

2026-08-03T22:00:00Z  success  req=1
2026-08-04T16:00:00Z  success  req=2
2026-08-05T03:00:00Z  success  req=2
                               ----
                               5 requests

Five requests over 29 hours cannot hold an instance open against sleepAfter = "20s". So the corrected reading of the issue body's own experiment is: one render woke the container and it has not slept since.

That is exactly what the #154 diagnosis predicts, and more strongly than "a long tail" did. The SDK sends SIGTERM and never escalates; node is PID 1 and discarded it; so the SDK never stops the container at all and the instance only goes away when a deploy replaces the placement. The diagnosis is corroborated, not weakened - the symptom is bigger than the issue claimed.

The 21:40Z -> 22:35Z hour of zero allocation the issue body credited to #116 ("sleeping works at all") is the same misreading from the other side: that gap is bounded by the 2026-08-03T21:22:46Z deploy tearing down placement 2d2b3db4, and it ends when the test render started a new one. It is a deploy, not a sleep.

The bill never stopped, either

Daily allocated memory for applicationId = a039bbb1, against the 96.0 GiB-hours a 4 GiB instance bills for a full day:

2026-07-13  95.6      2026-07-25  95.6      2026-07-31  95.7
2026-07-14  96.2      2026-07-26  97.0      2026-08-01  96.3
2026-07-15  98.2      2026-07-27  95.3      2026-08-02  99.0
2026-07-16  32.2      2026-07-28  96.1      2026-08-03  89.1  <- downsize 21:22Z
2026-07-19   2.3      2026-07-29  87.0      2026-08-04  23.6  <- = 24.0 at 1 GiB
2026-07-20  96.3      2026-07-30  11.6
2026-07-21  96.0
2026-07-22  96.8

100% of every full day, including the five days after the drain fix deployed on 2026-07-29. 2026-07-16 to 2026-07-19 is the last time this thing was genuinely idle; from 2026-07-20 it has been awake continuously. So the ~$28/month attributed to #116 was still being paid, and the 2026-08-03 downsize to basic cut it 4x rather than ending it: **$7/month is still going out for an instance serving ~7 requests/day.** The drain guard is still correct and should stay, but it was not the load-bearing cause.

(Ratio check, in the dataset's native bytes: 1.86 before 21:22Z and 3.73 after - standard-1 and basic respectively. Those are the same numbers CLAUDE.md records as 2.0 and 4.0 in dashboard GiB/GB units.)

The fix is on main and is NOT deployed

worker deployments   latest = 2e67470f, 2026-08-03T21:22:46Z
container app        version 3, updated_at 2026-08-03T21:24:01.382Z
                     image sha256:16205edadadd02568e19af0019de1aa13cf93b17840f94d6c7aed8dc952466a7
343df53 (SIGTERM handler)  committed 2026-08-05T01:52:20Z

The running image predates the handler by ~4.5 hours. Nothing measured above says anything about whether the fix works - it is not in production. This issue stays open.

Both container response paths in worker/src/index.ts already drain (.text() on the 502 path, .arrayBuffer() on the success path), so there is no second undrained path contributing.

What remains

1. pnpm run preview:deploy (needs Docker) to ship the handler.
2. One render with a fresh seed.
3. Wait at least an hour, then re-walk the five-minute buckets and check placementId continuity. The pass condition is a placement that appears for one or two buckets and then stops - not merely "the last bucket is missing", which is what produced the 8.5-minute number in the first place.

Tree state, unchanged by this: pnpm run verify exits 0 - 1588 app tests passed / 3 skipped, 13 worker, 4 container including exits promptly on SIGTERM.

#155 corrects the two CLAUDE.md claims this contradicts.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QZvNS2H4cbaZk46ybA7hbj

[wormeyman]

Verdict: INCONCLUSIVE on the idle tail - and the deploy has taken the preview service down

The fix has still never been exercised, for a reason nobody could have predicted from the plan: the new image does not start. Details below, worst news first.

🔴 Production is currently broken

Four renders, fresh unused seeds, spread over seven minutes - all 502:

05:55:49 -> 05:56:11   502  22.0s   seed 887766554
05:56:50 -> 05:57:10   502  20.7s   seed 887766555
06:00:37 -> 06:00:59   502  21.8s   seed 887766556
06:02:34 -> 06:02:56   502  22.0s   seed 887766557

wrangler tail on worker version 4d74b8ce:

container render failed (500): Failed to start container:
  There has been an internal error connecting to the port

That is @cloudflare/containers' TIMEOUT_TO_GET_PORTS_MS = 20_000 expiring - the container never listens on 8080 within 20 s. Billing confirms placements being born and dying: 9a0594d3 bills 10 GiB-seconds in the 05:57 minute and 50 in 06:00, i.e. lifetimes of seconds.

This is not a code defect in server.mjs. Run locally with PORT explicitly unset, it behaves correctly:

$ env -u PORT node server.mjs
preview container listening on 8080
(POST /render with a bad body -> 400)
SIGTERM received, closing server

The SDK does not inject PORT either (envVars defaults to {} and is only sent when non-empty), and the deployed app config has environment_variables: [] - so process.env.PORT is undefined in the container and Number(undefined ?? 8080) is 8080. The PORT override from #154 is exonerated.

So the failure is in the image or its startup, not the source. Prime suspects, none confirmed: the freshly built sha256:89a6723c... layer set, or a first-pull that cannot reach port-ready inside the SDK's 20 s window. Someone with deploy rights needs to look at this - the service renders nothing right now.

Why the quiet window proved nothing

The measurement plan assumed the 04:18-04:21 renders ran on the new image. They did not.

placements active since 04:00Z	1 - a20877a2, the pre-deploy one
container app v4 (sha256:89a6723c) updated_at	04:19:15Z
render #1 fired	04:18:38Z - before the app update landed
a20877a2 last allocation	04:33Z

a20877a2 had been awake since 2026-08-03T22:35Z. It survived the ~04:17 deploy and served both renders - the per-minute CPU confirms the work happened there (04:19 5.62s, 04:20 5.10s, 04:21 9.28s, 04:23 3.75s against an idle baseline near 1s). It then stopped at ~04:33, 16 minutes after the deploy, on an image that ignores SIGTERM by construction. That shutdown is the rollout evicting the old instance, not sleepAfter.

Everything after 04:34 is a container nobody woke. Zero allocation on zero traffic is also what the pre-fix code does - see 2026-07-16 → 07-19, genuinely idle for three days. The 85 quiet minutes are consistent with a working fix and with no fix at all, so they discriminate nothing.

Second arm: ONE placement, so the cold-start argument is void

Asked whether the two ~04:19 and ~04:20 renders show as two placements. They do not - one continuous placement, a20877a2, every single minute from 04:15 through 04:31 with no break:

04:15 04:16 04:17 04:18 04:19 04:20 04:21 04:22 04:23
04:24 04:25 04:26 04:27 04:28 04:29 04:30 04:31       (04:32 absent, 04:33 last)

Eric's render queued behind the first on one 0.25-vCPU instance. It never cold-started, so it says nothing about a stop-and-restart.

Worth adding: 27.3 s was never evidence of a cold start. The render in this issue's own body took 10.59 s including a cold start. 27.3 s is slower than a cold start, which is what contention on a shared quarter-core looks like - the two requests overlapped.

The one genuinely new fact, and it is good news

From the same tail, ~20 s after a request:

Activity expired, signalling container to stop

So in production, on the real deployed worker, sleepAfter = "20s" is honoured and the stop path does fire on schedule. That confirms the half of #154's trace that concerned the alarm loop. What remains unproven is the other half - whether the container now obeys the SIGTERM it is sent - because no container has stayed up long enough to be asked.

Where this leaves the issue

Not closing it. Nothing measured tests the fix.

To settle it, in order:

1. Fix the startup failure. Until a container serves a request, nothing else can be measured.
2. One render with a fresh seed that returns 200.
3. Go quiet, wait ~15 min, then walk the buckets: the pass condition is a placementId that appears for one or two buckets and stops - never "the newest bucket is empty", which is the backfill artifact that manufactured the phantom 8.5-minute tail in the first place.

Standing cost note: with the old placement gone since 04:33 and the new one unable to start, current container spend is near zero - but by outage, not by fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QZvNS2H4cbaZk46ybA7hbj

[wormeyman]

Root cause of the 502s found: the base image sets PORT=34197

Following up on the INCONCLUSIVE verdict above - the reason no container would start is in this repo, and it is one line.

#154 made the listen port overridable so test/shutdown.test.mjs could bind a free port:

const PORT = Number(process.env.PORT ?? 8080);

Our base image is factoriotools/factorio, and its image config sets PORT=34197 - Factorio's own UDP game port. Read directly from the registry config blob for the pinned digest:

ENV: PATH=... PORT=34197 RCON_PORT=27015 SAVES=/factorio/saves ...

So in production that resolves to 34197. The server binds the wrong port, the runtime waits its 20s TIMEOUT_TO_GET_PORTS_MS for something on 8080, and reports "There has been an internal error connecting to the port". Every render 502s after ~21s - reproduced at 05:55, 05:56, 06:00, 06:02 and 06:05 UTC on five fresh seeds.

Why the earlier local check exonerated it wrongly. That check ran env -u PORT node server.mjs, with PORT explicitly unset - the one condition under which an inherited-variable bug cannot appear. A test that clears the variable it is defending against proves nothing.

Fix: #159

Renames the override to FMW_CONTAINER_PORT; a project-prefixed name cannot collide with a base image's environment. 8080 stays the default, matching the Dockerfile EXPOSE and the Worker's defaultPort.

New test/port.test.mjs sets PORT=34197 in both cases. It is not vacuous - restoring process.env.PORT makes both tests fail (pass 0, fail 2), verified by planting the regression.

This issue stays open

#159 does not restore service by itself - it needs pnpm run preview:deploy, and the deploy path is what is currently broken.

Once a render returns 200 again, the actual #120 measurement can run for the first time: one render, quiet, wait ~15 minutes, walk the buckets. Pass = a placementId appearing for a bucket or two and then stopping. Never "the newest bucket is empty" - that backfill artifact is what produced the phantom 8.5-minute tail.

Note also that the 85-minute quiet window measured earlier discriminates nothing: it was a container nobody woke, and zero allocation on zero traffic is exactly what the pre-fix code does too.