Repository navigation
ThreatCull 2.4.0
Fetch and Compile are about twice as fast, runs show how long each stage took, and Settings shows
how much disk the database and Outputs use. Nothing to do when upgrading. After the upgrade, the
first Fetch of each Source parses its list in full once.
Changed
- 🚀 Faster Fetch and Compile. Domain validation skips steps that can't change its result, the
database looks each new entry up once instead of three times, and Outputs read from one sorted
index instead of sorting once each. Measured withscripts/bench.pyon 1 million lines: the first
Fetch went from 45 s to 19 s and a Compile from 29 s to 16 s. What counts as a valid entry is
unchanged. - ⏱️ Run timings per stage. The run history has a "Took" column with the time each stage took
(download, parse, apply for a Fetch; prune, score, allowlist, stats, outputs, publish for a
Compile), and/api/v1/runsreturnsduration_secondsandtimings. - 💽 Storage shown. Settings and the dashboard footer show the size of the database and of the
Outputs;/api/v1/statusreturnsdatabase_bytesandoutputs_bytes. - 📏 Sizing guide for any host. The Sizing section of the README gives measured times, memory and
database sizes for lists of 0.5 to 3 million lines and advice for small hosts.
scripts/bench.pymeasures your own host.