ThreatCull 2.7.0

Backups that bring an install back with the same feed URLs, and an optional full database backup.
After upgrading, export your configuration so your backup has the new fields. If a cron job runs
threatcull run next to serve, remove it: serve runs its own schedule, and run now refuses to
start beside it.

Added

• 💾 Configuration and full backups. The configuration export now carries Feed Token and API
  token hashes (still no passwords or raw tokens). threatcull config import --restore-tokens
  brings them back, so old feed URLs and API tokens keep working on a fresh install. The
  Restore Feed Tokens from the file box on Settings restores Feed Tokens only; API tokens are
  restored from the command line. threatcull backup create copies the whole database while
  ThreatCull runs, and threatcull backup restore replaces it with a checked backup file while
  ThreatCull is stopped, keeping a safety copy. See the README's "Back up and restore". A
  configuration file exported by this version needs this version or newer to import, because older
  versions reject the new fields.

Changed

• 🔒 One ThreatCull per data directory. serve and run now hold a lock on the data directory.
  A second serve, or threatcull run while serve runs (for example from cron), stops with
  "ThreatCull is running on …: stop it first". Commands also refuse to open the database while a
  backup restore is running.

Documentation

• 📝 Safer full backups. The README told you to copy the threatcull-data volume for a full
  backup, which can give a corrupt database while ThreatCull runs. It now uses
  threatcull backup create and shows how to restore on a new host. The Quick start is numbered
  steps, the README says who ThreatCull is for, and an Air-gapped install section covers offline
  hosts.

ThreatCull 2.6.0

A weekly Catalog health check, a private /data for new volumes, Outputs and upgrades that survive a
power cut, and the abuse.ch Sources leave the Catalog. After upgrading, use Check for updates on
the Sources page to get Catalog revision 5.

Added

• 🩺 Catalog health check. A workflow that runs every Monday downloads every Catalog Source
  and keeps one issue listing Sources that stopped working or stopped changing, so a broken Source
  is caught before it ships. The Catalog request bot now also rejects suggested feeds that serve an
  HTML page.
  #87

Fixed

• 🔧 The image creates /data as 0700. Since 2.1.0 the image created /data as 0755, so every
  new Docker volume started readable by other users and ThreatCull logged a warning on each start.
  New volumes now start as 0700. A volume created before this release keeps 0755: run
  docker compose run --rm --entrypoint python threatcull -c "import os; os.chmod('/data', 0o700)"
  once to tighten it.
• 🔧 Outputs and upgrades survive a power cut and a second process. A Compile now writes each
  Output to disk before it replaces the published file, so a power cut leaves the previous Output or
  the new one, never an empty file. It records the new Output counts together in one transaction.
  Starting a CLI command while the server upgrades the database no longer runs a migration twice.

Removed

• 🗑️ abuse.ch Sources leave the Catalog (revision 5). The abuse.ch
  terms of use now give access to authenticated users only, for
  not-for-profit use within query limits, so anonymous downloads no longer fit the Catalog. The
  Catalog drops feodotracker-ipblocklist (enabled by default until now), urlhaus-hostfile and
  threatfox-hostfile; the last two already answer HTTP 410. Use Check for updates on
  the Sources page: ThreatCull disables all three on an existing install and marks them removed from
  the Catalog. Support for Sources that need an API key is tracked in
  #116.

Documentation

• 📝 Contribution policy. ThreatCull doesn't accept code pull requests from outside contributors
  for now. Bug reports, feature ideas and Catalog Source suggestions in issues are welcome; describe
  a fix in words and the maintainer implements it. The bug report and feature request forms have
  an optional "Suggested fix" field for that. See
  CONTRIBUTING.md.
• 📝 Supported platforms. The README lists where ThreatCull runs: Docker on Linux (amd64 or
  arm64) is supported and tested, and Docker Desktop on Windows or macOS runs the same image but
  isn't tested. 32-bit ARM and Windows without Docker aren't supported. Keep the named data volume
  instead of mounting a Windows folder as /data.

ThreatCull 2.5.2

Security fix for the CA bundle, and the image moves to Python 3.15. Nothing to do when upgrading.

Security

• 🔐 certifi 2026.10.23 drops the compromised emSign roots (GHSA-6q48-6c7w-9gw2).
  certifi 2026.7.22 trusted emSign root certificates whose intermediate keys leaked, and
  ThreatCull uses that bundle for every HTTPS feed fetch. Upgrade to stop trusting certificates
  signed with those keys.
• 🔐 libexpat1 2.9.0 in the base image. The Docker Hardened Images python:3.15 base ships
  libexpat1 2.9.0-1~deb13u1+dhi1, which fixes the six CVEs that 2.5.1 listed as not reachable, so
  the image scan no longer ignores them.

Changed

• 📦 Python 3.15 and dependency updates. The image runs on the Docker Hardened Images
  python:3.15 base (Python 3.15.2) and builds with uv 0.13.6. FastAPI goes to 0.162.0 and the web
  UI charts to Chart.js 4.6.1.

ThreatCull 2.5.1

Updated base images and dependencies. Nothing to do when upgrading.

Changed

• 📦 Dependency updates. The Docker Hardened Images python:3.14 and python:3.14-dev bases move
  to new digests, the image builds with uv 0.13.1, and FastAPI goes to 0.143.0 and pydantic to
  2.14.1.

Security

• 🔐 libexpat1 CVEs in the base image are not reachable. The python:3.14 base still ships
  libexpat1 2.8.3-1~deb13u1+dhi5, which has CVE-2026-66046, CVE-2026-77214, CVE-2026-93990,
  CVE-2026-76956, CVE-2026-76957 and CVE-2026-102633, fixed in 2.9.0-1~deb13u1. Expat parses XML
  and ThreatCull parses none, so the image scan ignores these CVEs until a Docker Hardened Images
  digest ships the fix.

ThreatCull 2.5.0

ThreatCull now warns about Sources whose list stopped changing, and the Catalog gains eight Sources.
Nothing to do when upgrading: database migration 11 runs on start and dates each Source's last change
from the newest entry it added. Use Check for updates on the Sources page to get Catalog revision 4
on an existing install.

Added

• 🆕 Eight new Catalog Sources, all disabled until you enable them. IP blocklists: Binary
  Defense Artillery ban list (binarydefense-banlist), ThreatCluster IPs (threatcluster-ips) and
  SiberKapan honeypot IPs (siberkapan-ips). Domain blocklists: abuse.ch ThreatFox
  (threatfox-hostfile), C2IntelFeeds C2 domains (c2intelfeeds-domain-30d), ThreatView
  high-confidence domains (threatview-domain-high), ThreatCluster domains
  (threatcluster-domains) and SiberKapan phishing domains (siberkapan-domains).
• 🧊 Frozen Sources. ThreatCull records when each Source's list last gained or lost an entry and
  marks an enabled blocklist Source frozen when that is 30 or more days ago
  (frozen_after_days). The Sources page shows "changed 4 months ago" for every Source and has a
  Frozen filter, the dashboard counts frozen Sources, /api/v1/sources returns last_changed_at and
  frozen, and threatcull sources list prints both. Frozen Sources still count towards scores.
  After the upgrade, the change date starts from the newest entry each Source added.
  #84

Changed

• 👪 Source Families for C2IntelFeeds and ThreatView. The IP and domain Sources from each
  publisher now share a Family (c2intelfeeds, threatview), so together they count once towards
  a Confidence Score.

ThreatCull 2.4.1

What's Changed

• feat: custom allowlists, busy-database fix, no CIDR ranges in Outputs by @spydisec in #4
• chore(release): v1.2.0 by @spydisec in #5
• chore: image scanning, container hardening and community files by @spydisec in #7
• feat: Catalog updates between releases, CHANGELOG.md and README badges by @spydisec in #9
• ci: configure CodeRabbit for security and correctness reviews by @spydisec in #10
• chore(release): v1.3.0 by @spydisec in #11
• feat(web): show times in the TZ timezone; README screenshots by @spydisec in #13
• chore(release): v1.4.0 by @spydisec in #14
• test: speed up the suite and fix a flaky token check by @spydisec in #17
• ci: skip tests on pull requests that change no code by @spydisec in #19
• chore: ignore local Claude Code settings by @spydisec in #16
• docs: add DeepWiki page plan and project notes by @spydisec in #18
• chore: keep the test suite local and drop the test workflow by @spydisec in #21
• docs(readme): replace the Tests badge with CodeRabbit reviews by @spydisec in #23
• ci: skip jobs a pull request cannot affect by @spydisec in #25
• docs: Catalog page, and checked Catalog suggestions from issues by @spydisec in #26
• feat!: remove licence data from Sources by @spydisec in #29
• chore(release): v2.0.0 by @spydisec in #30
• docs: 2.0 screenshots and the :2 image tag by @spydisec in #32
• feat(catalog): add blackhole-today-ips by @github-actions[bot] in #34
• ci(catalog): only the owner approves, and approval closes the issue by @spydisec in #35
• chore(release): v2.0.1 and releases from the Release UI by @spydisec in #37
• fix(indicators): reject values with line breaks or control characters by @spydisec in #41
• chore(release): v2.0.2 by @spydisec in #42
• chore(ci): drop Bandit and CodeRabbit's duplicate scanners by @spydisec in #44
• feat(docker): build on Docker Hardened Images by @spydisec in #40
• chore(release): v2.1.0 by @spydisec in #45
• ci(docker): let fork pull requests build without Docker Hub secrets by @spydisec in #47
• chore(deps): Bump python from 3.13 to 3.14 in the images group by @dependabot[bot] in #49
• chore(release): v2.2.0 by @spydisec in #50
• docs: add launch video to README by @spydisec in #52
• chore(deps): Bump the images group with 2 updates by @dependabot[bot] in #54
• chore(release): v2.2.1 by @spydisec in #55
• perf: skip parsing unchanged lists, wait for DNS, tune SQLite by @spydisec in #63
• fix: run durations, background Force Compile; data dir in image, feed_path, API docs by @spydisec in #66
• feat: enabling waits for the schedule, run status line, new Sources not Stale by @spydisec in #67
• chore(release): v2.3.0 by @spydisec in #68
• perf: stream large downloads to disk and parse them line by line by @spydisec in #70
• chore(release): v2.3.1 by @spydisec in #71
• perf: faster Fetch and Compile, stage timings, storage, benchmark by @spydisec in #73
• chore(release): v2.4.0 by @spydisec in #74
• chore(deps): Bump the minor-and-patch group with 2 updates by @dependabot[bot] in #78
• chore(deps): Bump the images group across 1 directory with 2 updates by @dependabot[bot] in #77
• ci(deps): Bump https://github.com/astral-sh/ruff-pre-commit from v0.16.8 to 0.16.9 in the hooks group by @dependabot[bot] in #76

New Contributors

• @github-actions[bot] made their first contribution in #34
• @dependabot[bot] made their first contribution in #49

Full Changelog: https://github.com/spydisec/threatcull/commits/v2.4.1

ThreatCull 2.4.0

Fetch and Compile are about twice as fast, runs show how long each stage took, and Settings shows
how much disk the database and Outputs use. Nothing to do when upgrading. After the upgrade, the
first Fetch of each Source parses its list in full once.

Changed

• 🚀 Faster Fetch and Compile. Domain validation skips steps that can't change its result, the
  database looks each new entry up once instead of three times, and Outputs read from one sorted
  index instead of sorting once each. Measured with scripts/bench.py on 1 million lines: the first
  Fetch went from 45 s to 19 s and a Compile from 29 s to 16 s. What counts as a valid entry is
  unchanged.
• ⏱️ Run timings per stage. The run history has a "Took" column with the time each stage took
  (download, parse, apply for a Fetch; prune, score, allowlist, stats, outputs, publish for a
  Compile), and /api/v1/runs returns duration_seconds and timings.
• 💽 Storage shown. Settings and the dashboard footer show the size of the database and of the
  Outputs; /api/v1/status returns database_bytes and outputs_bytes.
• 📏 Sizing guide for any host. The Sizing section of the README gives measured times, memory and
  database sizes for lists of 0.5 to 3 million lines and advice for small hosts.
  scripts/bench.py measures your own host.

ThreatCull 2.3.1

Fetches keep large lists on disk instead of in memory, so a big Source no longer needs hundreds of
MB of RAM. Nothing to do when upgrading. After the upgrade, the first Fetch of each Source parses
its list in full once.

Changed

• 💾 Large downloads go to disk. A Fetch writes the download to a temporary file on the data
  volume (anything over 8 MB) and parses it line by line, instead of holding the whole list in
  memory twice (as bytes and as text). On a 3 million-line, 49 MB list the peak memory of a Fetch
  fell from 165 MB to 76 MB, and it no longer grows with the list's size. The image sets
  TMPDIR=/data so the temporary file stays on the volume.

ThreatCull 2.3.0

The status line shows what a run is doing, unchanged lists refresh in seconds, and enabling Sources
no longer starts a burst of Fetches. ThreatCull upgrades its database on the first start (new
columns only). After the upgrade, the first Fetch of each Source parses its list in full once;
on a Raspberry Pi with multi-million-line lists, expect that first run to take as long as it did
on a new install.

Changed

• 📡 See what a run is doing. The status line on the dashboard and Runs page names the step
  ("Fetching Source (3 of 10)" or "Compiling the Outputs"), how long it has run, how long it
  took last time and what started it (Run now, Force Compile or the schedule). While idle it shows
  the next Fetch and Compile, and it now refreshes every 20 s, so a scheduled run shows up without
  a reload. Outputs that no Compile has written yet say "not published yet" on the Outputs page
  and in the status line, since their feed URLs answer 404 until then. GET /api/v1/status
  returns the same information. The elapsed time counts up every second between refreshes.
  #58

• ⚡ Unchanged lists skip parsing. A Fetch hashes each download and compares it with the list
  it last applied. When nothing changed, ThreatCull refreshes the Source's entries without parsing
  them again, even if the server sends no ETag header. A 3 million-line list re-fetched in 5 s
  instead of 4 min in testing. An upgrade, or a change to a Source's URL or format, parses every
  list in full once.

• 🌐 The first Fetch waits for DNS. After a restart, the first scheduled Fetch waits up to two
  minutes for the Source's host to resolve, so a server that boots before the router no longer
  records a row of failed Fetches.

• 🗄️ SQLite tuning. ThreatCull refreshes the query planner's statistics after each Compile
  (PRAGMA optimize) and keeps its temporary tables on disk, so large Compiles don't depend
  on how the Python image's SQLite was built.

• 🐳 docker exec needs no --data-dir. The image sets THREATCULL_DATA_DIR=/data, so
  docker exec threatcull threatcull api-token create <name> --user admin works as written. It
  also sets SQLITE_TMPDIR=/data, so SQLite's temporary files stay on the data volume even when
  /tmp is a tmpfs mount.

• 🔗 feed_path in /api/v1/outputs. Each Output lists its feed URL path (/o/<name>); add
  the Feed Token to get a working URL.

• 📖 API and sizing docs. docs/API.md covers every endpoint with curl examples,
  feed URLs and the difference between a feed URL and the Download button. The README has a
  Sizing section with measured numbers and advice for a Raspberry Pi, and docker-compose.yaml
  has a commented mem_limit example.

• 🗓️ Enabling a Source no longer starts a Fetch. A Source enabled in the web UI, the API, the
  CLI or by a Catalog update waits for the regular schedule (at most an hour for its first Fetch),
  so enabling several Sources in a row doesn't start a Fetch and a Compile for each one. Click
  Run now to fetch them at once. The Sources page shows when a new Source's first Fetch is
  due. After a restart, ThreatCull still catches up on overdue Sources within minutes.
  #60

Fixed

• 🕒 New Sources are not Stale. A Source that has never fetched counted as Stale, so enabling
  several new Sources could push the Stale share over max_stale_ratio and block the Compile until
  they had fetched. Only a Source whose last success is older than stale_after_hours counts now;
  the dashboard lists new ones as "waiting for first Fetch".
• 🔁 Re-enabled Sources fill up again. A Source disabled for longer than the retention period
  lost its entries to pruning, and when it was re-enabled the server's 304 Not Modified kept it
  empty until the upstream list changed. ThreatCull now notices the missing entries and downloads
  the full list.
• ⏱️ Runs show how long they took. Every Fetch and Compile stored its start time as its finish
  time, so the run history and /api/v1/runs showed 0 s for every run. ThreatCull now records the
  real finish time. #59
• 🧭 Force Compile no longer hangs the page. The page waited until the whole Compile finished,
  which takes minutes with large lists. Force Compile now starts in the background like Run now,
  and the run status shows when it ends. #57

ThreatCull 2.2.1

Updated base images with OpenSSL security fixes. Upgrading is recommended; nothing else to do.

Security

• 🔐 OpenSSL fixes in the base image. The Docker Hardened Images python:3.14 and
  python:3.14-dev bases move to new digests that ship OpenSSL 3.5.7-1~deb13u3, which fixes
  High-severity vulnerabilities in the 2.2.0 image (CVE-2026-54873, CVE-2026-84782, CVE-2026-84784,
  CVE-2026-72897). The image scan finds no High or Critical vulnerabilities again.

Changed

• 📦 uv 0.12.22 builds the image (was 0.12.19).