example/billing: the kind functional tests lose their PostgreSQL port-forward when a case drops its database WITH (FORCE)

[Zaba505]

What happens

The Billing walkthrough (kind) job fails intermittently, on story PRs, in the merge queue and on main, e.g. run 39378019962. Outside Renovate's runs it failed in 18 of 162 runs since 2026-10-19, about 1 in 9. All 18 have the same signature:

• queries in flight fail with unexpected EOF;
• a case's cleanup DROP DATABASE … WITH (FORCE) fails with conn closed;
• every later connection to PostgreSQL fails with dial tcp 127.0.0.1:<port>: connect: connection refused, so the rest of TestBilling_passes_the_shared_cases_as_deployed, TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack and TestGrafana_alerts_on_the_billing_job_and_has_its_dashboard fail too.

connection refused on the forwarded port means the kubectl port-forward service/postgres that deploy/kind/test.sh starts has exited. The PostgreSQL pod itself is still Running with no restarts.

Where only one case was running, the first thing to fail is always the DROP DATABASE … WITH (FORCE) that comes right after the case closes its own connection to that database:

• cases.go:254, in newDatabase's cleanups (the conn.Close cleanup runs first, then the drop): TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack in run 39402915366 and TestGrafana_… in run 39412690207;
• functional_test.go:720, in failingAnInvoice's deferred calls (again conn.Close first, then the drop): TestGrafana_… in run 39257723132 and run 39411536440.

In the parallel shared cases the forward dies at some case's drop, and the cases still running or yet to start fail after it.

Why

This hasn't been triaged. What follows comes from the logs and a reproduction outside humus.

• pgx's Close sends Terminate and closes its socket without waiting for the server. Through the port-forward, that Terminate still has to cross kubectl, the API server, the kubelet and containerd before it reaches the pod. The admin connection's DROP DATABASE … WITH (FORCE), sent right after, can terminate the closed connection's backend first. PostgreSQL then resets the connection inside the pod.
• On CI's node (containerd 2.3.4) that reset comes back on the connection's error stream, and client-go closes the whole port-forward session on it. In my runs on a 1.29 node (containerd 1.7) it didn't. kubectl exits with error: lost connection to pod, taking every other forwarded connection with it. This is upstream's kubernetes/kubernetes#111825 (open). kubernetes/kubernetes#143050 (open) would stop one connection's error from closing the session.

Reproduction. I deployed postgres:18.0 on kind and ran a small pgx v5.11.0 program through kubectl port-forward service/postgres :5432. Each of 8 workers loops: connect as admin, CREATE DATABASE, connect to it, create a table and insert a row, Close that connection, then DROP DATABASE … WITH (FORCE) on the admin connection. That is the same order as the cases' cleanups.

Cluster	kubectl	Before the drop	Result
kind v0.33.0, kindest/node:v1.37.0 (containerd 2.3.4, as in CI)	v1.37.1	nothing	the forward died in 8 of 8 runs, each within seconds
same	v1.37.1	wait until pg_stat_activity shows no backend on the database	3 of 3 runs passed (2,400 cycles)
same, 1 worker	v1.37.1	nothing	3 of 3 runs passed (600 cycles)
kind v0.22.0, kindest/node:v1.29.2 (containerd 1.7)	v1.29.14 and v1.37.1	nothing	3 of 3 runs passed (2,600 cycles)

So the race needs some load. CI supplies it: the shared cases run in parallel under -race on a 4-CPU runner, next to the LGTM stack. kubectl's log on each death:

"An error occurred forwarding" err="error forwarding port 5432 to pod …, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-…\": writeto tcp4 127.0.0.1:46060->127.0.0.1:5432: read tcp4 127.0.0.1:46060->127.0.0.1:5432: read: connection reset by peer"
error: lost connection to pod

The program's drop then failed with conn closed, as in CI.

Expected

The kind functional tests pass whenever the deployment works. A case dropping its own database doesn't take down the port-forward the other cases and tests share.

References

• run 39378019962, the run that prompted this
• kubernetes/kubernetes#111825 and kubernetes/kubernetes#143050
• jackc/pgx#2584, on aborted closes surfacing as connection reset by peer at a proxy

[Zaba505]

Triage

Verdict: A reproducible bug in the billing walkthrough's kind tests, not in a humus package. A case's cleanup drops its database WITH (FORCE) right after it closes its connection to it. Through kubectl port-forward on a containerd 2.x node, that race resets one connection, and the reset ends the whole forward. It isn't only CI's: the walkthrough's pages tell readers to run test.sh on kind, and today's kind runs the same node. #1178 is filed to fix it.

Is this a misuse of humus?

No. No end user's code is involved. The behaviour belongs to the walkthrough's own test harness: the shared cases' newDatabase, failingAnInvoice, and deploy/kind/test.sh, which starts the one kubectl port-forward service/postgres that every test shares. No humus package is on the path. The tests reach PostgreSQL with pgx, through kubectl.

The cause is upstream, in containerd 2.x and client-go (see Why). That doesn't make it a misuse. The harness is humus's, and it's what puts the drop right behind the close. A reader can't defend against it either: they run test.sh as deploying.md and testing.md say.

Is it a feature request?

No. It breaks these promises:

• #949: "Every case creates a database of its own, so the cases run in parallel, and against a shared staging server." Its revision 1 runs "The shared cases, deployed" "in the "Billing walkthrough (kind)" job, which isn't required, and against an end user's staging".
• story(example): run the billing job's shared cases against kind #987's acceptance criterion: "Every shared case, from story(example): run the billing job's shared cases against PostgreSQL and WireMock in containers #983 and story(example): rehearse failures in the billing job's shared cases #984, passes against kind, in the "Billing walkthrough (kind)" job."

A fix changes no humus API. The example module isn't released, and the fix is in its tests and scripts.

Is it reproducible?

Yes. At d4eadda, on kind v0.33.0 (kindest/node:v1.37.0, containerd v2.3.4) with kubectl v1.37.1, postgres:18.0, pgx v5.11.0 and Go 1.27.0, on rootless Podman 5.8.9. Every run was under -race.

The new tests are in the functional package. Each starts its own kubectl port-forward service/postgres, as the databases walkthrough's functional tests do, so it can log kubectl's output. Through that forward, workers repeat what a shared case does to PostgreSQL: newDatabase, an insert, then newDatabase's cleanups in their order. Each test asserts that every cycle completes and that a new connection succeeds afterwards.

Test	Node	Runs failed	Cycles completed	Drops that terminated a backend
TestPostgres_forward_outlives_cases_dropping_their_databases, 8 workers × 100	v1.37.0, containerd v2.3.4	4 of 4	0, 40, 0 and 16 of 800	1 per run, and it killed the forward
TestPostgres_forward_outlives_one_case_at_a_time_dropping_its_databases, 1 worker × 300	v1.37.0, containerd v2.3.4	0 of 4	300 of 300 each	0
the 8-worker test	v1.29.2 (kind v0.22.0), containerd v1.7.13	0 of 4	800 of 800 each	20 to 28 per run
test.sh as it is: the existing functional tests	v1.37.0, containerd v2.3.4	0 of 4	n/a	0

8 workers on containerd 2.3.4: the forward dies at the first drop that terminates a backend. Run 1:

1. A worker closes its connection to its database, then drops the database WITH (FORCE) on its admin connection.
2. The close's Terminate hasn't reached PostgreSQL yet, so the drop terminates that backend: 14:05:51.127 UTC [1640] FATAL: terminating connection due to administrator command.
3. 3 ms later, containerd's forward of that connection reads a reset from PostgreSQL: E1020 14:05:51.130198 … "Unhandled Error" err="error forwarding port 5432 to pod …, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-…\": writeto tcp4 127.0.0.1:34624->127.0.0.1:5432: read tcp4 127.0.0.1:34624->127.0.0.1:5432: read: connection reset by peer".
4. kubectl reads that on the connection's error stream and closes the session: portforward.go:549] "An error occurred forwarding" err="error forwarding port 5432 to pod …: read: connection reset by peer", then error: lost connection to pod.
5. Every other connection through the forward ends with it. The other workers' drops fail with conn closed (8 in this run), a connect in flight fails with failed to receive message: unexpected EOF, and the test's last connect fails with dial tcp 127.0.0.1:33649: connect: connection refused. The PostgreSQL pod stays Running, with no restarts.

That is the report's signature from CI, step for step. The other three runs went the same way, with steps 2 and 3 within 3 ms of each other each time.

1 worker: nothing to race. With no other load, every Terminate reached PostgreSQL before its drop. No drop terminated a backend in 1,200 cycles.

containerd 1.7.13: the same reset, and the forward survives. The drops terminated 20 to 28 backends per run, and containerd logged the resets, such as httpstream.go:282] "forwarding port" err="failed to execute portforward in network namespace …: read tcp4 127.0.0.1:51112->127.0.0.1:5432: read: connection reset by peer". kubectl logged nothing, and every cycle completed.

test.sh as it is passed. On a 32-CPU machine, no case's drop terminated a backend in 4 runs, so the race never happened. CI's 4-CPU runner is busier: the cases run under -race, next to the LGTM stack. I count the same 18 failed runs with this signature since 2026-10-19 outside Renovate's, and none in the other walkthroughs' kind jobs.

The report's account holds. One precision about its containerd 1.7 runs: the reset happens there too. containerd 1.7.13 logs it and keeps it off the error stream. Why is below.

Why

1. pgx v5.11.0's Close sends Terminate and closes its socket without waiting for the server (pgconn.go L757-L765). I got that from pgx's code.
2. Through the forward, the Terminate goes from kubectl through the API server and containerd's streaming server to PostgreSQL. The drop takes the same way on another connection. Under load, the drop can get there first. The backend is then still alive, and the drop terminates it. Reproduced: PostgreSQL's FATAL lines, which never appeared with one worker.
3. The terminated backend's connection is reset, not closed. I reasoned that from TCP: a socket closed with the Terminate still unread, or written to after the backend closed it, answers with a reset. The reset itself is reproduced: containerd reads connection reset by peer within 3 ms of the FATAL.
4. containerd v2.3.4's portForward returns that error. The copy from kubectl ends first, and the reset comes within the second it waits for the other direction (sandbox_portforward_linux.go L101-L126). Its streaming server, k8s.io/cri-streaming v0.36.3, writes every error from portForward to the connection's error stream (httpstream.go L253-L262). I got that from containerd's code. The message is reproduced: kubectl logs it word for word.
5. client-go v1.37.1, which kubectl uses, closes the whole session when an error stream carries a message (portforward.go L548-L550). ForwardPorts then returns lost connection to pod (L360-L365), and kubectl exits. That is port-forward closed on specific postgres command kubernetes/kubernetes#111825, still open. Reproduced: kubectl's log.
6. containerd v1.7.13 had its own copy of the streaming server. It logs ECONNRESET and EPIPE and resets that connection's streams without writing to the error stream (httpstream.go L269-L284), citing port-forward closed on specific postgres command kubernetes/kubernetes#111825. containerd 2.3.4's cri-streaming has no such case. That's why the v1.29.2 node survived. I got that from containerd's code, and its httpstream.go:282 line is in the reproduction's log.

What else it reaches:

• Readers. The walkthrough's pages tell readers to run test.sh on kind, and kind v0.33.0's default node is the one I reproduced on. A reader whose machine is busy hits the same failure. I reasoned that from the above; test.sh passed here.
• An end user's staging. Proposal: job walkthrough #949 runs the deployed cases against an end user's staging. Through kubectl port-forward to a containerd 2.x node, they are exposed the same way. Reasoned only.
• The other forwards. test.sh forwards WireMock, Tempo, Loki, Mimir and Grafana with kubectl too. A reset from any of those pods would end that forward the same way. Nothing in the tests resets those connections, and no CI failure shows it. Reasoned only.
• The databases walkthrough. It forwards PostgreSQL with its own kubectl port-forward, and its cases drop databases WITH (FORCE) through it (dropRetained). It is exposed only when a connection to that database closes just before the drop. I didn't test it, and no failed run of its kind job since 2026-10-19 shows the signature.

#1178 is filed to fix it:

• a case dropping its database no longer ends the other cases' and tests' connections, and the three tests pass whenever the deployment works;
• the fix holds on containerd 2.x, where one reset ends the session;
• tests in functional_test.go, from these reproductions, show 8 parallel workers completing every cycle, in the kind job;
• the walkthrough's pages still match test.sh and the cases.

[Zaba505]

Fixed by #1179 (#1179), merged as 443a9ae.

A case's cleanup no longer drops its database while its own connection is still closing. cases.NewDatabase now closes the case's connection and waits until pg_stat_activity no longer lists that backend. Only then does it run DROP DATABASE … WITH (FORCE). The drop therefore never terminates a backend reached through kubectl port-forward, PostgreSQL never resets a forwarded connection, and a containerd 2.x node has no reset that could end the forward. failingAnInvoice now calls cases.NewDatabase instead of keeping its own copy of the lifecycle, so it gets the same fix.

TestPostgres_stays_reachable_while_cases_drop_their_databases, in functional_test.go, is the triage's 8-worker reproduction. It runs through the same forward the cases use: 8 parallel workers do 100 database lifecycles each, then a new connection has to succeed. On kind v0.33.0 (kindest/node:v1.37.0, containerd 2.3.4) it failed in 3 runs out of 3 without the fix. With the fix it passed in 4 runs out of 4, and PostgreSQL logged no terminations across 3,200 drops. It runs in the "Billing walkthrough (kind)" job, which passed on the pull request.