Skip to content

v2.2.0

Latest
Latest

Choose a tag to compare

View all tags
@Lore09 Lore09 released this 23 Jun 22:34
· 2 commits to main since this release
v2.2.0
This tag was signed with the committer’s verified signature.
Lore09 Lorenzo Venerandi
GPG key ID: 62F2D0D542B76B54
Verified
Learn about vigilant mode.
6b5b2c9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Krawl 2.2.0

Highlights

This release brings a 1200+ page deception library built from real-world attacks, first-class Prometheus metrics & monitoring, custom HTML templating, bulk import/export of deception pages, and a large round of Kubernetes/Helm hardening for scalable deployments.

New Features

1200+ Deception Pages from Real-World Attacks (#224)

by @BlessedRebuS

Krawl now ships with 1289 HTML deception templates under src/templates/deception/, generated from real-world attacks captured across many Krawl honeypot instances. When an attacker probes a known vulnerable endpoint, Krawl serves a realistic page instead of a generic response, capturing their activity.

Category Count Examples
Spring Actuator / Heapdump ~256 /actuator/env, /actuator/heapdump, /actuator/logfile
.env & Secret Files ~214 /.env, /admin/.env, /backend/.env.backup
Configuration Files ~150 /config.php, /config.json, /application.yml
WordPress ~106 /wp-admin/, /wp-json/wp/v2/users, /wp-config.php
Login / Admin / Auth ~84 /admin/login, /adminer.php, /auth/signin
Database Backups & SQL ~84 /backup.sql.gz, /database.sql, /dump.tar.gz
Archive Downloads ~65 /api.zip, /site.tar.gz, /src.rar
Cloud Services (AWS/GCP/Azure/Firebase) ~55 /.aws/credentials, /gcp-key.json, /firebase-config.json
API / Swagger / REST ~49 /api-docs, /swagger-ui, /openapi.json
Credentials & Secrets ~47 /credentials.json, /private-key.json, /secrets.yml
CI/CD (Jenkins, GitLab, CircleCI) ~23 /jenkins/.env, /gitlab/.env
Debug / Profiler ~25 /debugbar, /_profiler/open
...and more PHP Info, Git Leaks, Crypto Bots, Livewire, PHP Shells, Sitemaps, Framework-specific

Templates auto-import on startup, .html extensions are stripped for cleaner URLs, and any path already present in the database is skipped. The community can now open PRs to contribute their own deception templates and expand Krawl's coverage.

Bulk Import / Export of Deception Pages (#197, #198)

by @BlessedRebuS

Deception pages can now be exported and imported in bulk, making it easy to back up, migrate, or share page libraries between Krawl instances. Upload and download file parsing was reworked, page limits for bulk uploads were removed, and the dashboard was updated to drive the new workflow. Kubernetes manifests were also updated to support bulk import.

Custom HTML Templating (#201)

by @BlessedRebuS

Krawl can now serve a fully custom deception page via a user-provided HTML template.

  • New custom_template_path config field / KRAWL_CUSTOM_TEMPLATE_PATH env var.
  • Docker Compose and Kubernetes manifests mount the template at /templates/custom_page.html.
  • Helm support via a disabled-by-default customTemplate feature backed by a ConfigMap.
  • Removed redundant Kubernetes manifests.

Self-Hosted LLMs on Kubernetes (#202, #217, #218)

by @BlessedRebuS

Building on the 2.1.0 self-hosted LLM support, local inference engines (llama.cpp and Ollama) can now be deployed directly on Kubernetes via the Helm chart.

  • Added Helm values for both Ollama and llama.cpp, with a unified krawl-llm service name.
  • Tuned llama.cpp Helm template and fixed template syntax.
  • Extended Helm chart documentation with full LLM setup instructions.

Credential Download as ZIP (#223)

by @BlessedRebuS

Captured credentials can now be downloaded directly from the dashboard as a ZIP archive containing usernames.txt and passwords.txt (de-duplicated) — ready to feed into wordlist analysis or threat intelligence pipelines.

Metrics & Monitoring

Prometheus Metrics (#189, #196, #204)

by @carnivuth, @Lore09

Krawl now exposes operational and security metrics in Prometheus format.

  • New metrics section in config.yaml (metrics.enabled) to toggle exposure on or off.
  • A /metrics endpoint served under the secret dashboard path (returns 404 when disabled).
  • All metric definitions centralized in a dedicated src/metrics.py module, with background tasks (analyze_ips.py, dashboard_warmup.py) refreshing metrics as part of their normal execution.
  • A custom KrawlMetricsCollector exposes cumulative values as Prometheus counters and recomputes clients_total live at scrape time, eliminating drift and negative values.
  • Batch counter reads (get_many) and an optimized get_all in metrics_counters.py reduce Redis round-trips for scalable deployments.

ServiceMonitor for Prometheus Operator (#204)

by @Lore09

  • New serviceMonitor section in helm/values.yaml and a servicemonitor.yaml template.
  • The ServiceMonitor is only created when metrics are enabled and a fixed dashboard secret path is set, with clear guidance and warnings in NOTES.txt for misconfigurations.

A Grafana dashboard and metrics/monitoring documentation are included to get started quickly.

Bug Fixes

Private/Reserved IPs No Longer Banned or Tracked (#219)

by @Lore09

Private, local, and reserved IP addresses (e.g. Kubernetes health-check sources) were being tracked and could be banned. A new is_local_or_private_ip utility is now used in BanCheckMiddleware, record_access, and increment_page_visit to skip ban and tracking logic entirely for these addresses.

Broken Healthcheck (#220)

by @Lore09

  • Added a dedicated /healthz endpoint under the secret dashboard prefix, exempt from ban and deception tracking.
  • The Helm deployment now uses the secret dashboard path for HTTP probes (falling back to a TCP probe when no secret path is set); the all-in-one manifest uses a TCP startup probe.
  • Health probe requests are suppressed from access logs to reduce noise.

Exponential Backoff Overflow (#221)

by @Lore09

The exponential ban multiplier for repeat offenders could overflow the database integer column and crash with "integer out of range". A new MAX_BAN_EXPONENT constant and _ban_multiplier_for helper cap the multiplier at 2**10, enforced everywhere the multiplier is set.

Helm Service Selector Collision (#216)

by @BlessedRebuS

The Krawl service selector was matching both Krawl and Ollama pods (both used krawl.selectorLabels). An app.kubernetes.io/component: krawl label now distinguishes them.

Boolean Migration Defaults (#203)

by @Lore09

Migration functions were updated to use proper boolean defaults.

Improvements

Database Manager Refactor & Query Optimization (#208)

by @Lore09

  • Introduced modular repository classes (CredentialRepo, GeneratedPageRepo, and others) for cleaner database access, with a re-exported public API (db.credentials.count(), db.ip_stats..., etc.).
  • Added indexes on the ip_stats table (last_seen, first_seen, reputation_score) to speed up paginated and sorted queries.

Dashboard & AI Generation Improvements (#200, #222, #223)

by @BlessedRebuS, @Lore09

  • Removed duplicate AI-generated pages and added generated pages to the search bar.
  • Collapsible search-results summary with a cleaner expand/collapse interaction; clicking an IP cell collapses results.
  • Direct export download URL with a copy-to-clipboard button (for external systems like OPNsense).
  • "Inspect IP" button on attack rows for quick access to IP insight.
  • Import all file types from the deception folder (not just .html, including dotfiles); preserve non-.html extensions in URL paths.
  • AI generation is now shielded from request cancellation, so slower self-hosted LLMs can complete.
  • Access History styling tweaks and a delete button on search results (authenticated only).
  • Reordered overview: Top User-Agents above Top IP Addresses.

Massive Code Linting & Security Vulnerability Management (#206)

by @Lore09

A large pass of code linting and dependency/security vulnerability cleanup across the codebase.

Suppress Server Header (#214)

by @BlessedRebuS

Added the --no-server-header flag to uvicorn to suppress the Server header, reducing fingerprinting (fixes #207).

Automated Banlist Updates (#210)

by @carnivuth

Added a GitHub Action that updates the banlist from the Krawl demo instance and opens a PR (closes #77).

Helm & Deployment Tuning (#215, #217)

by @BlessedRebuS

  • Clarified Traefik configuration documentation.
  • Added an init container for PostgreSQL in scalable deployments.

What's Changed

Full Changelog: v2.1.0...v2.2.0

Contributors

BlessedRebuS, carnivuth, and Lore09
Assets 2
Loading