Skip to content

v2.11.0

Choose a tag to compare

@sean-e-dietrich sean-e-dietrich released this 30 Jul 23:09
24f08a3

A performance and tooling release. The UserAgent plugin was recompiling a 1.7 MB regex corpus in every PHP process — ~618 ms, paid again by every php-fpm worker on its first request. That is now cached and scoped, taking a typical evaluation to ~0.5–3 ms.

Alongside it: a CLI that answers "would this request be blocked, and by what?", storage you can search and prune, and a new automated: rule that catches the scanner traffic bot:true has been quietly letting through.

No configuration changes are required and no existing rule changes meaning. One new runtime dependency — see Upgrade notes.

PHP 8.1–8.5 supported.

Added

  • bin/firewall-check (#106, fixes #105). Ask whether a given request would be blocked, against a config file, without a running site.

    $ vendor/bin/firewall-check --config=firewall.yml --ip=203.0.113.5 --url=/wp-admin/
    BLOCKED  GET /wp-admin/
      client            203.0.113.5
      blocked by        IP Address
      status            400
      storage           throwaway (in-memory)

    Takes --ip, --url, --method, --header (repeatable), --body, and --config (repeatable, merged as Firewall::create() merges them). --json for scripting; exit codes are the verdict — 0 allowed, 1 blocked, 2 challenged, 64 usage, 70 internal — so it composes in CI.

    --explain lists every plugin that evaluated with its timing, and then the plugins an earlier match short-circuited past. That second list is usually the answer to "why wasn't this caught?".

    Three behaviours are load-bearing enough to state here:

    • A check cannot ban anyone. Blocking is not a read-only event — it writes to storage, records an offense and applies blocking_escalation. Pointed at a production config, a naive checker would ban the address it was asked about. Storage is replaced with a throwaway by default; --live-storage opts back in and warns.

    • Mode is forced to exception. Firewall::evaluate() returns early under any other mode when PHP_SAPI is cli, which would report every request as allowed. A checker that always says "nothing is blocked" is worse than no checker, so this is not optional.

    • Attribution comes from the firewall's own decision log, not from re-running each plugin. A second pass would repeat every side effect, including any outbound reputation lookup.

  • Searchable, prunable storage (#104, fixes #26). StorageInterface was keyed access only, which left two operational questions unanswerable: who is currently blocked? and how do I lift a block that should not have been applied? The only recourse was reset().

    if ($storage instanceof QueryableStorageInterface) {
        foreach ($storage->find('203.0.113.0/24') as $address => $record) { … }
        $storage->deleteMatching(['203.0.113.5', '198.51.100.0/24']);
    }

    Both take a single address or a CIDR range, IPv4 or IPv6. All three shipped storages implement it; FileStorage inherits from InMemoryStorage.

    A new QueryableStorageInterface rather than additions to StorageInterface, for two reasons. Not every backend can enumerate its own keys — docs/guides/custom-storage.md uses Memcached as its worked example, and Memcached cannot list keys at all; mandatory enumeration would oblige it to return an empty set that reads as "nothing is blocked". And storage.type accepts any class implementing StorageInterface, so adding methods there would fatal every existing custom storage on upgrade.

    Three behaviours worth knowing: a malformed pattern matches nothing rather than everything, and an out-of-range prefix such as /33 is rejected rather than silently clamped to a single host; find() hides expired records while deleteMatching() still removes them; and offense history is cleared alongside the block, or blocking_escalation would re-escalate a just-un-blocked address on its next request.

  • automated: user-agent rule (#112, refs #109). bot:true is backed by device-detector's curated database, which does not classify sqlmap, nikto, curl, python-requests or Go-http-client as bots. If you wrote bot:true expecting scanners to be stopped, they have been getting through.

    config:
      - "automated:true"

    automated:true is the union of that database and a broader crawler list. It is an ordinary rule variable, so it composes with the rest of the syntax:

    - type: AND
      rules:
        - "automated:true"
        - "!client.name@contains:StatusCake"   # except your own monitoring

    bot: is unchanged. The wider list deliberately counts generic HTTP client libraries as automated, so redefining bot:true would start blocking a partner integration or mobile app built on python-requests — on a rule written long ago and never touched. A regression test pins all nine reference agents against today's bot: behaviour.

    bot.name, bot.category and bot.producer still come from the curated database only; the wider list yields a matched pattern rather than an identity.

Performance

  • The UserAgent regex corpus is cached (#110, fixes #107). matomo/device-detector compiles 20 YAML files totalling 1.7 MB on the first parse in each PHP process, and nothing ever called setCache(). Measured through bin/firewall-check:

    618 ms   first request (populating the cache)
     23 ms   every subsequent process
    618 ms   with caching disabled, every time
    

    Under php-fpm each worker paid that on its first request and again after every pm.max_requests recycle.

    On by default, writing to KANOPI_FIREWALL_CACHE_DIR when defined and otherwise the system temp directory — the convention AbuseIpdb already uses. metadata.cache accepts false to disable, a dir, an ['adaptor', 'args'] pair matching the shape CacheRateLimitStorage already takes, or an already-constructed PSR-6 pool.

    Nothing here can take a site down: an unusable adaptor falls back to the default and warns, and an unwritable directory logs and continues uncached.

    Worth correcting an assumption from the issue: this is not a scanner problem. Bots are the cheap case, because parsing returns early once one is identified — Googlebot cost ~48 ms while an ordinary iPhone Safari agent cost ~596 ms, since brand and model detection walks the largest part of the corpus. It hit real traffic hardest.

  • Only the needed parse phases run (#111, fixes #108). Detection has four phases — bot, OS, client, device — and all four ran regardless of what the rules asked for. The rules are known at construction, so the depth is now derived once:

    Deepest variable in your rules Per request
    bot, automated ~0.5 ms
    os.* ~0.9 ms
    client.* ~2.8 ms
    device.type, brand, model ~8.4 ms

    Nothing to configure. Two properties are deliberate: phases are cumulative rather than individually selectable, because device detection reads the OS and client results to infer a type and running it without them would produce a wrong answer rather than a slower one; and bot detection always runs, because parsing stops early once a bot is identified and skipping it would let bots reach phases they do not reach today.

    An unrecognised variable or rule shape falls back to running every phase, so the worst case is a lost optimisation rather than a rule that quietly stops matching.

Fixes

  • presets/README.md described config.yml wrongly (#103, fixes #102). The table called it "the composed preset that pulls in the recommended set". It is malicious-urls.yml + wordpress.yml, and it deliberately excludes malicious-requests.yml — which the same table calls "the recommended starting point" six rows earlier. Anyone who read the table and included config.yml on that basis got URL-pattern blocking while believing they had the recommended configuration.

Upgrade notes

  • composer update kanopi/firewall pulls one new runtime dependency, jaybizzle/crawler-detect — 156 KB, MIT, no dependencies of its own beyond PHP. It backs the automated: rule and is inert unless you use it.

  • Nothing else is required, and no existing rule changes meaning. bot: matches exactly what it always did, StorageInterface is untouched, and detection results are identical with caching and phase-scoping in place — asserted against a full parse across seven user agents and every field the plugin can read.

  • The user-agent cache writes to disk by default. It goes to KANOPI_FIREWALL_CACHE_DIR when that constant is defined, otherwise a kanopi-firewall-device-detector directory inside the system temp directory. Set metadata.cache: false on the plugin to opt out; detection is unchanged either way, only the speed differs. There is nothing to clear on upgrade — device-detector keys its entries by its own version, so a package bump produces new keys and the stale ones age out.

  • If you rely on bot:true to stop scanners, add automated:true. sqlmap and nikto are not in device-detector's bot database and never have been. Roll it out somewhere you can watch first if you have API consumers: the wider list counts curl, python-requests and Go-http-client as automated, which is usually right for a firewall and occasionally wrong for a partner integration.

  • bin/firewall-check appears in vendor/bin for consumers installing this package. It reads config and never writes to your storage unless asked with --live-storage.

Internal

  • A working performance harness (#100, fixes #7). The previous tests/Performance suite had never run and could not have: BenchmarkRunner.php was a 0-byte file, the CircleCI job invoked a script that did not exist, that job was never referenced from config.yml, and its executor combined cimg/php with docker-php-ext-install and sudo service nginx start. The load generator was single-process and serialised its own batches.

    Replaced with nginx → php-fpm → firewall driven by k6, 14 scenarios each against a freshly recreated container so no warm-up leaks between measurements. baseline (no firewall) and bootstrap (firewall, zero plugins) are reference points, which is what separates config-parsing cost from plugin cost. A PHP auto_prepend_file rewrites REMOTE_ADDR from a header so one generator can present thousands of distinct clients; it is inert unless FIREWALL_PERF=1.

    It reports rather than gates — CI runners are shared hardware and an absolute latency threshold would flake more often than it would catch a regression. composer perf, perf:quick, perf:validate, perf:down. The much cheaper scenario validation runs on every PR.

    This is the instrument that found both of the performance issues fixed above.

  • composer check:request wraps bin/firewall-check for local use.

  • Suite: 1,012 unit tests and 49 integration tests pass. PHPCS, PHPStan at level max and Rector are clean, and bin/firewall-check is clean at level max despite sitting outside the src/-scoped gates. The 2 reported deprecations are a pre-existing ReflectionProperty::setValue() call in LoggingFactoryTest, unrelated to this release.