Stop the podman suite tripping sshd's per-source penalty - #301
Merged
Conversation
The 303 that _get_session raises for a cookieless request was invisible in the OpenAPI schema, so the generated docs claimed 44 endpoints could never redirect. Static analysis flagged roughly 36 of them. Adds AUTH_REDIRECT_RESPONSES and composes the two existing response dicts off it, so the four routes that already documented 400 and 403 inherit the 303 without restating its wording. Two tests carry the contract, because either one alone is insufficient. The sweep walks each route's dependency graph, the same graph that causes the 303, and asserts the annotation is present; deriving the route set from the cause rather than a hand-written list is what makes a future route added without the annotation fail. That only proves the annotation exists, not that it is true, so a second test pins the constant to the status _get_session actually raises: switching the raise to 401 now fails loudly instead of leaving 44 stale annotations green. The two websocket routes are left alone; responses= has no meaning for a handshake.
Ruff flagged UP006 and UP007 on the new AUTH_REDIRECT_RESPONSES. The two dicts below it carry the same legacy typing style and were only unreported because they are outside this diff, so all three move together; leaving one modern constant wedged between two old ones would be worse than either style applied consistently. Dict and Union are now unused and drop out of the typing import. Optional stays: the four remaining Optional annotations are untouched Form defaults on unrelated route signatures, and converting them here would be churn beyond this change.
The `podman` job failed on the last two or three tests of an otherwise green run, in asyncssh with `ConnectionLost`, on PR #295, PR #297 and on main. It looked PR-specific because each branch lost a different tail test, but the cause is shared and the sshd journal names it: sshd[189]: drop connection #0 from [172.18.0.1]:44264 on [172.18.0.3]:22 penalty: connections without attempting authentication OpenSSH 9.8 added PerSourcePenalties and Fedora ships it enabled. sshd charges a source address one penalty second per TCP connection that closes without attempting authentication, and starts dropping connections outright once the accrued total passes the default 15s floor. The suite generated 34 of those in 45 seconds. `podman_target` probes the port with a bare `socket.create_connection`, to skip rather than fail when no host is up, and at function scope that ran once per test. Every client sits behind the compose bridge and so shares one source address, which makes the whole run a single offender. Two changes, both wanted. The probe becomes session-scoped, which is all it was ever meant to be, and the test image turns the penalty off, because a rate limiter a full run can still reach is one that will fire again the next time the suite grows. The sshd drop-in already asserts its own effect through `sshd -G`, so the new directive is verified the same way and a base image that drops the option will fail the build rather than quietly restore the flake. Verified on fedora:43 with openssh-server, where `sshd -G` reports `persourcepenalties no` under the drop-in and `noauth:1 ... min:15` without it, and against the real host: 36 passed, 4 skipped, in 35s.
Contributor
Up to standards ✅🟢 Issues
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
The
GitHub / podmanjob has been failing on the last two or three tests of anotherwise green run, in asyncssh with
ConnectionLost. It looked PR-specific,because each branch lost a different tail test and the branches bumped
unrelated things, but PR #295, PR #297 and
main(run 30828236096) all failedthe same way within twenty minutes of each other.
The sshd journal from inside the host container names the cause outright:
Why
OpenSSH 9.8 added
PerSourcePenaltiesand Fedora ships it enabled. sshd chargesa source address one penalty second per TCP connection that closes without
attempting authentication, and starts dropping connections outright once the
accrued total passes the default 15s floor:
podman_targetprobes the port with a baresocket.create_connection, so adeveloper with no host up gets a skip rather than a wall of red. At function
scope that ran once per test. Every client sits behind the compose bridge and so
shares one source address, 172.18.0.1, which makes the whole run a single
offender: 34 unauthenticated connections in 45 seconds, comfortably past the
floor, and the tests still running when it tripped are the ones that died.
That also explains the shape of it. Always the tail of the run, never the start;
a different test each time, because the crossing point moves with timing; and
intermittent on
main, where a fast run squeaked under the threshold.The fix
Two changes, both wanted.
tests/podman/conftest.py:podman_targetbecomes session-scoped, cuttingthe probes from roughly forty per run to one. That is all the probe was ever
meant to be.
Dockerfile.podman-host:PerSourcePenalties noin the sshd drop-in, becausea rate limiter that a full run can still reach is one that will fire again the
next time the suite grows.
The drop-in already verifies its own effect through
sshd -G, so the newdirective is asserted the same way.
sshd -Gexits non-zero on an unknownkeyword, so a base image bump that drops the option fails the build rather than
quietly restoring the flake.
The app itself is untouched. It authenticates every connection it opens and so
never accrued these penalties; only the test probe did. No production sshd
config changes.
Verification
Built against the real base image (
fedora:43+openssh-server):Removing the drop-in in the same container confirms the default was
noauth:1againstmin:15.Full suite against the real host: 36 passed, 4 skipped, 1102 deselected in
35.13s. The 4 skips are the
tests/e2e/test_podman_e2e.pyones, which needthe app container.
Follow-up
#295 and #297 will not go green on their own. Once this lands they each need a
rebase (
@dependabot rebase) to pick it up.