How It Works

How it works

Patterns is a pipeline, not a runtime component. Nothing from this project sits in the request path: it produces configuration files, and your proxy does the blocking.

The pipeline

coreruleset/coreruleset  ──daily──▶  owasp2json.py  ──▶  owasp_rules.json
                                                              │
                    ┌──────────────┬──────────────┬───────────┴──┐
                    ▼              ▼              ▼              ▼
              json2nginx.py  json2apache.py  json2traefik.py  json2haproxy.py
                    │              │              │              │
                    ▼              ▼              ▼              ▼
             nginx_waf.zip  apache_waf.zip  traefik_waf.zip  haproxy_waf.zip
                              published as a GitHub Release

owasp2json.py pulls the upstream Core Rule Set and flattens it into one JSON intermediate. Each converter reads that intermediate and emits the native syntax of its target: map and if for Nginx, SecRule for Apache ModSecurity, middleware TOML for Traefik, ACL files for HAProxy. badbots.py builds the User-Agent blocklists alongside them, from three public lists: nginx-ultimate-bad-bot-blocker, Crawler-Detect and referrer-spam-blacklist.

The consequence worth knowing: a bad rule reaches every backend at once, because they all descend from the same JSON. That is why several fixes in the issue tracker touch owasp2json.py rather than a single converter.

Which CRS version you are actually getting

This is worth checking rather than assuming, because two different mechanisms produce two different answers. Tracked as #43.

owasp2json.py selects the upstream tag with --ref, which is a prefix, not an exact tag. fetch_latest_tag() lists the CRS tags, keeps those whose ref starts with the prefix, sorts them as strings in reverse, and takes the first. The default prefix is v4.0, and the tags matching it upstream are:

refs/tags/v4.0.0
refs/tags/v4.0.0-rc1
refs/tags/v4.0.0-rc2

Reverse string ordering puts v4.0.0-rc2 first, because it is longer than v4.0.0 and shares its prefix. So a default run converts a release candidate, not the current Core Rule Set.

The string sort is worth noting on its own: it is not semver-aware, so v4.9.0 sorts above v4.29.0.

Meanwhile the release notes published by the daily workflow do not read that value at all. They query the CRS latest release separately:

CRS_REF=$(curl -sfL https://api.github.com/repos/coreruleset/coreruleset/releases/latest | jq -r '.tag_name')

and print whatever comes back. The version named in a release body is therefore the newest CRS release on that day, which is not necessarily the one the archives were built from.

owasp_rules.json records a _provenance block, but its source_ref currently reads latest rather than a resolved tag, so it does not settle the question either.

To build against a known version, pass it yourself:

python owasp2json.py --ref v4.29.0
python json2nginx.py

An exact tag is still treated as a prefix, but a full tag matches only itself. To check what a given intermediate contains, compare its categories against the rule filenames at the ref you expect: the category is the last dash-separated segment of the CRS filename, so RESPONSE-956-DATA-LEAKAGES-RUBY.conf becomes RUBY, and a category present upstream but absent from your JSON means you built from an older ref.

Running it yourself

pip install -r requirements.txt

python owasp2json.py     # fetch upstream CRS into owasp_rules.json
python json2nginx.py     # then whichever backend you need
python badbots.py        # bad-bot blocklists

Output lands in waf_patterns/<platform>/.

The four json2*.py converters are independent and idempotent, and take no command-line flags: they read INPUT_FILE (default owasp_rules.json) and OUTPUT_DIR from the environment. owasp2json.py is the exception and takes --output, --ref and --dry-run. Both it and badbots.py read GITHUB_TOKEN when present, which only raises the API rate limit; neither needs it to run. The full reference is in the API docs.

Adding a platform means one more converter against the same JSON. The existing four are between 189 and 259 lines, so it is a file rather than a fork.

What the automation does

Workflow	When	What it does
update_patterns.yml	daily at 00:00 UTC, plus manual	Re-fetches CRS, regenerates every backend, commits any change, republishes the latest release
test_nginx.yml	push to main and pull requests	Validates the generated Nginx rules against a live container
test_apache_docker.yml	push to main and pull requests	Validates the Apache rules against ModSecurity in Docker
docs.yml	push to main touching docs/	Builds and deploys the documentation site

Everything runs on GitHub-hosted runners. Because the daily job pushes its regenerated files to main, the two validation workflows run against each daily rebuild as well as against pull requests.

Three practical implications:

• The archives are rebuilt daily under the same latest tag. The tag you downloaded last month is not the tag you download today. Record the SHA-256 from the release body of the build you deployed.
• A local patch to a generated file under waf_patterns/ is overwritten by the next build. Fixes belong in the converter that produced it.
• The daily job commits only when the output changed. A long run of unchanged files means the pipeline is resolving to the same upstream ref, not that upstream has stopped moving. No file under waf_patterns/ has changed since 2025-02-28, which is the symptom described in #43. The archives still change checksum daily because zip embeds timestamps.