Skip to content

v2.30.0 — input the operator did not write

Choose a tag to compare

View all tags
@sean-e-dietrich sean-e-dietrich released this 17 Sep 16:13
· 50 commits to 2.x since this release
v2.30.0
b2b5389
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v2.30.0

Input the operator did not write.

This release was planned as metrics and a tarpit. It is mostly something else, and the something else arrived by asking questions rather than by planning: five of the seven items are places where a value nobody typed became a decision the firewall enforced, with nothing bounding it.

What they have in common is not the mechanism. It is that each failed in a way that looked like the firewall working — which is why none of them had been noticed. A feed containing 0.0.0.0/0 refuses every visitor while --lint reports the configuration clean. A visitor's browser posts the lifetime of its own challenge pass, and thirty-one years is a valid answer. A rule source is fetched over HTTPS and used, and HTTPS says nothing about whether it is the file the publisher meant to publish.

Four of those five came out of two questions: triaging #120, and "do we allow a cookie time limit to get set for the challenge?"

PHP 8.1–8.5 supported. There are behaviour changes on upgrade — three of them, below.

What changes on upgrade

  • A challenge rule asking for a pass longer than an hour now grants an hour. challenge.ttl is new, defaults to 3600, and is a ceiling as well as a default. Raise it if a rule legitimately wants longer. firewall-check --lint reports any rule this affects, so you can find out before you deploy rather than from a visitor.
  • A rule source larger than 32 MiB now fails instead of being read. upstream.max_size is new and finite by default, and it applies to the decompressed body too. max_size: 0 opts out.
  • A rule source whose every entry fails its validate: now fails the source, rather than quietly contributing nothing. With on_error: last_known_good — the default — the previous list keeps serving.

And one new warning on configurations that change nothing: firewall-doctor now counts remote rule sources fetched without a checksum or signature. One line for all of them, not one per source.

Everything else is opt-in and inert until configured: challenge.revocable, challenge.passes_valid_from, checksum:, signature:, max_entries, response: tarpit, and the metrics exporters.

Fixed

  • A rule source containing 0.0.0.0/0 blocked every visitor, and --lint called the config clean (#364)

    ConfigLinter already refuses that value when you write it, and bin/firewall-block refuses it from the CLI. A source was the one route by which it reached a block decision unexamined — and linting deliberately does not fetch, so the entry it would object to was never there to see.

    The realistic cause is ordinary rather than adversarial: an empty line parsed as a prefix, a placeholder shipped by mistake, a CSV column read by the wrong index.

    before:  BLOCKED  GET /     blocked by from-a-feed     ← every visitor
         ✓ 1 rule inspected · 0 errors, 0 warnings     ← and lint says fine

after:   ALLOWED  GET /                                 ← catch-all refused
         BLOCKED  GET /     blocked by from-a-feed      ← the rest of the feed survives

    Any /0 prefix in either family, *, and the full-span range form. /1 is not included — half the internet is somebody's decision; this is for the value that cannot be one. The entry is dropped and the rest of the source kept, logged at error rather than warning, because every other dropped entry is one rule that will not fire and this one would have been every rule firing on every request.

    A source that genuinely means it declares allow_catch_all: true. The refusal cannot be conditioned on the rule's response:, because a plugin is never told which bucket it is in.

  • A challenge pass lasted as long as the client asked for (#367)

    The lifetime that signs a pass token arrives in the interstitial's POST body. max(0, …) put a floor under it and nothing put a ceiling on it:

    requested ttl: 999999999
token exp:     2058-05-25  (31.7 years)

    Solve one challenge, post a large ttl, and the pass short-circuits every response: challenge rule that provider serves for three decades. The token is HMAC-signed, so it is valid — the signature covers the lifetime the client asked for. For a math provider, the documented starting point and scriptable, that is cheap to obtain.

    challenge.ttl binds configuration as well as the POST, and it has to: the submission arrives at challenge.path, not at the protected URL, so verification cannot tell one rule's lifetime from another's. A ceiling that only bound the POST would be a ceiling on nothing.

    The clamp lives in one place, so the number rendered into the form, signed into the token, carried on ChallengeSolved, and re-rendered after a rejected solution are the same number. That last one was a real hole: the rejected path echoed the posted value back, handing an over-long request straight to the next submission.

  • Bounded a rule source by size and entry count, not only by time (#366)

    upstream.timeout bounded how long a fetch may take and max_redirects bounded where it may go. Nothing bounded how much came back — and for most of these formats the decoded form is larger than the body, with both held at once. #120: "a list that grows to 4 GB should fail, not OOM the refresh job."

    The read itself is bounded, one byte past the ceiling, so an oversized body is detectable without ever being held. A local file is checked by stat before it is opened at all.

    It applies after decompression too, which was raised in review and mattered:

    gzip body on the wire:   104,776 bytes
  passes max_size: 32M?  YES
after gzdecode():        54,000,000 bytes
amplification:           515x

    That is ordinary repetitive list data, not a crafted bomb. A ceiling that stopped at the wire would be a ceiling with compression: gzip as a documented bypass — and .gz on a URL infers compression without anybody choosing it.

    Both ceilings refuse rather than truncate. Half a block list is a list whose meaning nobody knows: it loads, matches fewer things than it should, and looks exactly like a list that is simply shorter this week.

    The interesting failure turned out not to be the big one. A CDN or captive portal answering with an HTML error page is small, decodes as txt without complaint, and produces a few dozen lines that are not addresses — every one dropped by validate:, leaving the source contributing nothing and the rule silently matching nothing. A source that decoded to something and validated down to nothing now fails, so last_known_good keeps the list that was working an hour ago.

Added

  • Verify a rule source against a published checksum or signature (#365)

    sources:
  - name: abusive-ips
    upstream: https://example.org/v1/abusive-ips.txt
    checksum: sha256            # sidecar at <upstream>.sha256

    HTTPS authenticates the host and protects the transport. It says nothing about a repository that was compromised, a CDN object that was replaced, or a publisher who pushed the wrong file. #120 called this "the part that decides whether this is safe", and it is the part that did not ship with rule sources in 2.18.0.

    Two tiers, and the gap between them is the point. A sidecar fetched from the same host raises the bar without clearing it — anyone who can replace the list can replace the sidecar. It is the cheap tier, and it is exactly why signature: with a pinned ed25519 key is the one that means something.

    Opt-in, and staying that way. Most published lists in this ecosystem ship no sidecar at all, so a source declaring neither key keeps working unchanged.

    The loader's existing body_hash is not this, and the word sha256 appearing there is what made it look handled: it compares this fetch against the previous one so an unchanged body can skip the decode. Verification therefore runs before that short-circuit, and the verification settings join the source fingerprint — entries cached before a source was verified came from bytes nothing checked.

    Needs ext-sodium for signature: only, and a source declaring one on a host without it fails rather than running unverified.

  • Withdraw a challenge pass without rotating the secret for everybody (#368)

    A pass token is stateless and signed, so until now the only lever was rotating challenge.secret — which re-challenges every legitimate visitor holding a pass in order to withdraw one.

    $ bin/firewall-challenge firewall.yml --inspect=eyJpcCI6…
Pass token:
  address:  203.0.113.9
  provider: math
  expires:  2026-09-17 13:04:11 UTC
  nonce:    af2f100884fa35c3281b6d1db727b8fd

$ bin/firewall-challenge firewall.yml --revoke-nonce=af2f1008… --reason="abusing the pass"

    Two levers, both off by default, because the stateless property is the point of the design:

    Costs Reaches
    challenge.passes_valid_from Nothing. One config value, no storage Every pass issued before a moment
    challenge.revocable One storage read, on an otherwise-valid pass One pass, by nonce

    The cutoff compares against a new iat claim, and a pass minted before that claim existed is withdrawn by any cutoff — which is the right way round rather than a compromise: the passes most worth withdrawing this way are the ones issued before the firewall was fixed.

    bin/firewall-challenge is new, because revocation by nonce is an API nobody can reach if there is no way to get a nonce. The Challenge solution accepted log line now carries pass_nonce and pass_expires, so you grep it by address rather than retrieving a token out of somebody's browser.

    Revoking is not blocking. Blocking refuses a client; this withdraws an exemption a client earned.

  • StatsD and Prometheus exporters over the decision events (#222)

    $listener = new DecisionMetricsListener(new StatsdRecorder('127.0.0.1', 8125));

foreach (DecisionMetricsListener::eventClasses() as $event) {
    $dispatcher->addListener($event, $listener);
}
    firewall_requests_total{decision,rule,enforced}
firewall_challenges_total{provider,outcome}
firewall_challenge_failures_total{provider,reason}
firewall_tarpit_in_flight

    The socket is twenty lines; deciding what the series are is the work, so that two installations produce the same ones. Issued and solved are counted separately on purpose — ChallengeSolved only fires for the visitors who came back, so one number cannot tell "working perfectly" from "catching only bots that never retry".

    Every label is bounded, and nothing derived from a request is one — not the address, not the path. The obvious first draft labels by client IP, and that is the draft that takes a Prometheus server down: one series per address, forever, on the component that exists to be hit by addresses nobody expected. Both the listener and the recorder cap anyway and bucket the excess, because a series that stops incrementing looks like traffic that stopped.

    StatsdRecorder is UDP only, and the missing TCP mode is a refusal: a connect to a host that is up but not answering waits, so a metrics box having a bad afternoon becomes a slow site. An unreachable agent is recorded once to getDegradedBackends() and then not retried.

    PrometheusRecorder's documentation leads with when not to use it. Under PHP-FPM a counter is gone before anything can scrape it — that is the execution model, not a limitation here — so StatsD is the honest answer on FPM.

    The dashboard is deliberately not here. A Grafana file or a CMS admin screen changes on its own cadence and nothing on the request path should carry it.

  • A tarpit response, and the concurrency gauge that makes it safe (#329)

    Split out of #203, which asked for four response actions. Three shipped in 2.26.0; this one did not, because shipping it with them would have shipped a self-DoS.

    tarpit:
  max_concurrent: 5     # holds allowed at once, across every tarpit rule
  max_seconds: 30

plugins:
  - plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
    response: tarpit
    metadata: { name: slow-the-scrapers, tarpit_seconds: 5 }
    config: ["user_agent@contains:python-requests"]

    A tarpit holds a php-fpm worker for its duration, and that worker is a resource an attacker can consume on purpose. With pm.max_children = 20 and a ten-second hold, twenty requests take the site down — and the rule that did it looks like it is working. So the delay is the easy part and the counting is the point.

    ConcurrencyGaugeInterface is the missing primitive: StorageInterface has no atomic increment, and RateLimitStorageInterface counts a rate, which is not the same question. It is optional, in the shape QueryableStorageInterface established, so nothing about the existing contract moves and no custom backend breaks.

    Backend Counts Scope
    RedisStorage INCR/DECR The fleet
    FileStorage Exclusive lock This host
    DatabaseStorage, InMemoryStorage No —

    Per-host is the right scope rather than a compromise: what a tarpit consumes is workers, and workers are per-host.

    A backend that cannot count atomically does not implement the interface, and a tarpit rule against one is a startup failure — loud, rather than running uncapped and looking like it is working. Under attack the tarpit stops tarpitting: a claim over the cap is given back and the request is served normally. That is the failure the site survives.

    The documentation says plainly that a CDN or nginx limit_req does this better, without holding a PHP worker, and that the case for doing it here is rule expressiveness rather than mechanism.

Also filed, not fixed here

Three things found while reviewing this release's own work, each kept out of it rather than rushed in:

  • #375 — block-list records keep the visitor's cookies and headers verbatim, including the host application's session cookie. Predates this release; sharpened by #368, because a pass is now worth something to whoever holds it.
  • #376 — a challenged POST is silently discarded: the visitor solves the interstitial and lands on a GET of the same URL with their submission gone.
  • #379 — every Monolog HTTP log handler makes a blocking HTTPS round trip inside the request, which is the hazard #222 explicitly refused for metrics.

Upgrading

$ composer require kanopi/firewall:^2.30
$ vendor/bin/firewall-check config/firewall.yml --lint
$ vendor/bin/firewall-doctor config/firewall.yml

--lint is worth running on this upgrade in particular: it is what reports a challenge rule affected by the new challenge.ttl ceiling, and a tarpit rule that would hold more workers than your pool can spare.

Full changelog: v2.29.0...v2.30.0

Assets 2
Loading