Krawl 3.0.0

The biggest release yet. Detection now runs on YARA rules: over 3,100 bundled rules, each with its own fake reply. The dashboard has been rebuilt tab by tab, and the docs are now a site at blessedrebus.github.io/Krawl.

A 2.4 database upgrades in place, but a few things changed underneath. Read the breaking changes before you deploy.

⚠️ Breaking changes

Important

The full checklist is in Upgrading from 2.4 to 3.0. Back up your database first: see Backups.

wordlists.json is gone. Detection uses the YARA rules in rules/, and every fake page and attack reply is a file listed in responses/manifest.yaml. A mounted wordlists.json is ignored, and Krawl logs one warning at startup. Move any values you customised into config.yaml (see Moving from wordlists.json). In Helm, the wordlists: values and their ConfigMap are removed; set the same keys under config:.

Retention has its own retention: section. database.retention_days was a single window for everything. Now each kind of data has its own:

retention:
  non-suspicious: 30d        # clean logs, clean IPs and their category history
  recheck-suspicious: 30d    # suspicious logs re-checked against current rules
  re-evaluate-interval: 5d   # re-score IPs whose analysis is older than this
  uninteresting: 7d          # new: random-path crawler GETs
  overall: null              # new: optional hard cap on all data

retention_days and KRAWL_DATABASE_RETENTION_DAYS still set the first two windows when those are absent, but both are deprecated. (#341)

Cleanup moved off startup. Purging ignored IPs, and IPv6 rows when ipv6.purge_existing is on, now runs in the nightly db-cleanup task at 03:00. Booting no longer scans the tables, so restarts are fast. Run the cleanup sooner from Settings → Maintenance.

Container and chart defaults changed:

Pod memory: Helm pods now request 128Mi with a 512Mi limit, up from 64Mi / 256Mi. YARA and the new background tasks need the headroom.
Pod user: pods run as UID 1000, with fsGroup: 1000 so existing volumes are re-owned on first start.
uvicorn flags moved from the image CMD into entrypoint.sh, which appends them to any command that starts with uvicorn. If you override command:, drop the flags and tune them with the new KRAWL_UVICORN_* variables instead.

YARA detection

The regex patterns are replaced by a YARA engine with 3,173 bundled rules. They cover:

generic web attacks: SQL injection, XSS, path traversal, command injection, SSRF, SSTI, deserialization and webshells;
exposure probes and IoT botnets;
the exploit requests of over 2,500 CVEs.

Every rule carries its attack type, severity, confidence, MITRE ATT&CK mapping, CVE and references, so a detection explains itself. Rules match exploit syntax rather than keywords: a parameter named host= or the word curl is not an attack. Each near miss is proven by a test corpus of benign requests. (#349, #350, #352)

Every detected attack gets a fake reply.

Reply type: a rule's response_type picks a reply from responses/manifest.yaml.
Variant: the string that matched picks the variant. GET /download?file=../../etc/shadow returns a fake shadow file, while GET /static/../../ returns a fake directory listing.
Examples: SQL injection gets a database error, and requests to the cloud metadata address get fake credentials.
Replace them: the bundled bodies are the same in every deployment, so swap in your own to avoid fingerprinting.

Each request field is scanned on its own: the path, the query, the body, the user agent and every header, both raw and URL-decoded. A rule knows which field it is looking at (scan_target), so a cookie such as a=1; id=5 does not trip a command-injection pattern meant for the query.

The new Rules tab lists every loaded rule with its metadata, fake reply and match counts. You can upload your own .yar files from the dashboard, without a restart. Each upload is checked before it goes live:

it must compile, with no YARA warnings;
import and include are refused;
the metadata is validated;
the rule must scan a 64 KB body within the time budget.

Uploads are stored in the database, so they survive restarts and image upgrades and reach every replica.

The raw request viewer highlights what matched, and per-CVE attack types are labelled by their CVE ID. (#370)

Dashboard

The whole dashboard was reworked:

Layout: a new header with centered tabs and summary tiles.
Search palette: opened with Ctrl K.
Consistent display: readable attack names, with consistent numbers and dates.
Severity colours: every attack type keeps one colour everywhere, taken from its rule's severity band.
IP links: every IP is a keyboard-reachable link that you can deep-link to.
Honest states: loading, empty and error states say what is going on. (#336, #353, #354, #355)

Overview leads with what changed. A "Last 24 hours" panel lists the attackers active right now. The live map and the real-time Recent Suspicious Activity feed update as traffic arrives, and the summary tiles count active attackers and current time-bans. (#330)

Attacks has a shared period window.

Period: 1D, 7D or 30D, stepping back through history.
Attack Trends in 1-hour, 4-hour or 1-day buckets.
Filters by rule and severity, plus Only/Hide text filters that narrow every panel at once.
Attackers ranked within the period.
Shareable state: the view is kept in the URL, so you can share a link, and it survives reloads.

Threats tracks campaigns.

Grouping: payloads are grouped by TLSH fuzzy hash, with a Campaign activity heatmap.
By target: campaigns can also be grouped by the path they hit.
Tables: compact, filterable Campaigns, Captured files and Assets tables. (#333, #370)

New: Advanced search. Find any stored request by path, host, IP, referer, user agent, method, raw request text or HTTP header. (#334)

IP Insight was redesigned as a scoreboard.

Totals for the IP.
Behaviour radar: the signals that decided its category, with each category's score.
Ranked attack types.
Request filters and timing metrics.
Always-visible ban and track actions, each stating what it will do. (#365)

Targeted Domains gained filters and a link map of the domains and subdomains attackers aim at (#331). Settings shows the running configuration, with secrets hidden.

Data and retention

Uninteresting requests are pruned early.

The problem: most stored requests are crawlers following Krawl's own random trap links, often millions of them.
The fix: these are marked uninteresting when written and deleted after retention.uninteresting (7 days) through a partial index, with no per-IP reads.
No change to categories: each IP keeps a count of its pruned requests, so its category does not shift.
No lost evidence: anything with a detection, an upload or a honeypot hit is never pruned. (#364)

Stored logs are re-checked against current rules after retention.recheck-suspicious, and IPs are re-scored every retention.re-evaluate-interval. A rule you add today also applies to recent history.

More of the analysis is configurable:

the scoring weights, under analyzer.scoring_weights;
the batch size of every scheduled task, under tasks: (#329).

Background backfills enrich 2.4 data after the upgrade. They work in batches over the following minutes or hours:

targeted domains and request assets;
captured files, with better attachment detection (#328);
campaign hashes;
the inbound Referer, to rebuild which bait led where (#322).

Integrations

CrowdSec export. fwtype=crowdsec emits JSON decisions for cscli decisions import, with a ready-made script in plugins/crowdsec/. Thanks to @mgkvibez (#359, #362).

Federated banlists. An instance can publish its banlist on a public path (banlist.export_path) and merge lists from other instances or any plain-text IP list (banlist.sources).

New settings:

server and proxy headers;
credential form fields;
error codes;
map tiles;
the YARA rules and responses paths.

Every setting can be overridden with a KRAWL_* variable.

Infrastructure

Docker image:

Smaller: a multi-stage build, so the runtime image ships no compiler.
Less memory: MALLOC_ARENA_MAX=2.
Health: a TCP HEALTHCHECK.
Workers: one uvicorn worker per container, with limits tunable through KRAWL_UVICORN_*. Scale with containers or pods, not workers.

Helm chart:

Security: pods run as UID 1000, with their own ServiceAccount and no API token mounted.
Probes: startup, readiness and liveness probes are all configurable.
Clean shutdown: a preStop delay and a configurable grace period.
Scheduling: topologySpreadConstraints.
Upgrades: pods roll on every upgrade.
Networking: configurable session affinity.
Secrets: existingSecret support for the PostgreSQL credentials.
Startup: an init container that waits for Redis.
Safety: the chart refuses to render standalone mode with more than one replica.

PostgreSQL and Redis:

PostgreSQL is tuned for the write path, in both the chart and Docker Compose: shared buffers, WAL settings and synchronous_commit=off.
It gets 256MB of shared memory for parallel queries, and its liveness probe is relaxed.
Redis is capped at 192MB with volatile-lru eviction. (#363)

Indexes are built without blocking. Large PostgreSQL indexes are built online with CREATE INDEX CONCURRENTLY by the build-online-indexes task, so an upgrade never locks writes. Unused and redundant indexes are dropped. (#363, #367)

The dashboard is faster on large databases:

warmed caches for the default views;
keyset paging on the map;
estimated totals on large tables;
one shared host rollup across the domain panels;
Targeted Domains served from hourly totals in Redis;
a 60-second ceiling on any dashboard read.

Prometheus per-category IP counts are computed every 5 minutes instead of on every scrape.

CI:

the banlist update workflow runs as krawl-bot with proper labels (#323, #325);
the docs site builds and deploys from the self-hosted runner (#373);
the test suite runs in about 60 seconds, down from about 130.

Fixes

Multiple replicas: new ip_stats rows are created race-free.
Missing User-Agent: no longer crashes the dashboard.
Missing locations: they show N/A instead of NoneNone.
IP Insight: it no longer sends a NUL character to PostgreSQL. (#366)
Threats tab (scalable mode): the campaigns panel no longer crashes on cached rows. (#371)
Panels that load together: none are dropped any more. Captured files on the Threats tab could stay blank because of this. (#374)
Request assets: URLs found in User-Agents are no longer recorded as assets.
Campaign activity: it counts requests, not detections.
KRAWL_ERROR_CODES: parsed as numbers.
KRAWL_BANLIST_SOURCES: parsed as a list; it used to fetch each character as a URL.
Settings → Configuration: credential_password_fields is no longer hidden as a secret, and scoring_weights shows its values instead of [object Object].
AI page generation: it handles case variants of a path and scanner probes, and produces a unique page per path.
Deception templates: fixed. (#351)
Accessibility: attack badges keep 4.5:1 text contrast, and filter pills and chart slots work from the keyboard.

Docs and tools

The docs are now a site: blessedrebus.github.io/Krawl (#373, #374).

Config builder: every setting as a form field, with its help text and KRAWL_* variable. Copy the result as config.yaml, environment variables or Helm values. It's generated from the code, so it always matches the release.
Rule builder: write a YARA rule, check it against Krawl's rule contract, and test it on a raw HTTP request, including which fake reply Krawl would send.
API reference: every Dashboard API endpoint, from the OpenAPI schema.
Updated guides: configuration, upgrading, retention, detection, the dashboard and the architecture, with every screenshot retaken from 3.0.0.

⬆️ Upgrading

Schema changes and backfills run automatically at startup, so there is no manual migration. Back up first, and read the upgrade guide.

Docker Compose. The shipped compose files track latest, so pulling is the upgrade. Replace any wordlists.json mount with the rules/ and responses/ mounts from the new compose files, or drop it.

docker compose -f docker/docker-compose.standalone.yaml pull
docker compose -f docker/docker-compose.standalone.yaml up -d
# scalable: same two commands against docker-compose.scalable.yaml

Docker run. Your data lives in the volume, not the container.

docker pull ghcr.io/blessedrebus/krawl:3.0.0
docker stop krawl && docker rm krawl
docker run -d -p 5000:5000 -v krawl-data:/app/data \
  --name krawl ghcr.io/blessedrebus/krawl:3.0.0

Helm. The chart and the image are both 3.0.0. Move any wordlists: values under config: first.

helm upgrade krawl oci://ghcr.io/blessedrebus/krawl-chart --version 3.0.0 -f values.yaml

From source. yara-python is a new dependency, with prebuilt wheels for Linux and macOS. The uvicorn flags are shorter now.

git pull && pip install -r requirements.txt
uvicorn app:app --host 0.0.0.0 --port 5000 --app-dir src --no-server-header

What's Changed

tuned performances and fixed db and ui bugs by @BlessedRebuS in #320
Tuned referer and improved attachement detection by @BlessedRebuS in #322
Feat: improved banlist workflow and fix trigger paths by @Lore09 in #323
feat(banlist): update pull request author and committer to krawl-bot, add labels and review comments by @Lore09 in #325
Community banlist update by @krawl-bot[bot] in #326
feat(banlist): remove reviewers from pull request creation step by @Lore09 in #327
Bug/fix file attach by @BlessedRebuS in #328
Feat: add configurable task batch size by @Lore09 in #329
Feat/add live map by @BlessedRebuS in #330
Added filters and js map for domains and subdomains by @BlessedRebuS in #331
Community banlist update by @krawl-bot[bot] in #332
Feat: improved threats UI by @Lore09 in #333
Added advanced search capability by @BlessedRebuS in #334
Community banlist update by @krawl-bot[bot] in #335
Community banlist update by @krawl-bot[bot] in #337
Feat: UI refactor and performance upgrade by @Lore09 in #336
Fix branch conflicts by @Lore09 in #340
Feat: non-suspicious request crunch and retention config by @Lore09 in #341
Feat: tune crunching and UI minor improvements by @Lore09 in #342
Feat tune crunching and UI by @Lore09 in #343
perf(crunch): skip single-request IPs, crunch most recently idle first by @Lore09 in #344
Community banlist update by @krawl-bot[bot] in #345
Community banlist update by @krawl-bot[bot] in #348
Added yara rule engine, replaced the old regex-based pattern by @BlessedRebuS in #349
Tuned yara rules by @BlessedRebuS in #350
Fix deception templates by @Lore09 in #351
Tuned Yara rules by @BlessedRebuS in #352
Feat: improving db handling by @Lore09 in #353
Feat faster UI experience by @Lore09 in #354
Community banlist update by @krawl-bot[bot] in #356
Feat improving UI experience by @Lore09 in #355
feat(export): crowdsec decisions format (#148) by @mgkvibez in #359
fix(export): crowdsec decisions must be JSON for cscli import by @mgkvibez in #362
Feat: db tuning by @Lore09 in #363
feat: replace access-log crunching with pruning of uninteresting requests by @Lore09 in #364
Feat: ip insight redesign by @Lore09 in #365
fix(ip-insight): stop sending a NUL character to PostgreSQL by @Lore09 in #366
Community banlist update by @krawl-bot[bot] in #368
Feat db old stuff cleanup by @Lore09 in #367
feat(threats): group campaigns by target, highlight YARA matches by @Lore09 in #370
fix(threats): campaigns panel crashed on cached rows in scalable mode by @Lore09 in #371
ci: publish docs to GitHub Pages by @Lore09 in #373
Release/3.0.0 by @Lore09 in #372
docs: 3.0 screenshots, interactive config and rule builders, two dashboard fixes by @Lore09 in #374
docs(export): CrowdSec decisions are JSON by @Lore09 in #375
ci: skip the regression tests on docs-only PRs by @Lore09 in #376
Enhance security policy with reporting guidelines by @BlessedRebuS in #378
Release 3.0.0 by @Lore09 in #379

New Contributors

@krawl-bot[bot] made their first contribution in #326
@mgkvibez made their first contribution in #359

Full Changelog: v2.4.0...v3.0.0

Navigation Menu

Repository navigation

v3.0.0

Choose a tag to compare

Sorry, something went wrong.