Repository navigation
ThreatCull 2.3.1
Fetches keep large lists on disk instead of in memory, so a big Source no longer needs hundreds of
MB of RAM. Nothing to do when upgrading. After the upgrade, the first Fetch of each Source parses
its list in full once.
Changed
- 💾 Large downloads go to disk. A Fetch writes the download to a temporary file on the data
volume (anything over 8 MB) and parses it line by line, instead of holding the whole list in
memory twice (as bytes and as text). On a 3 million-line, 49 MB list the peak memory of a Fetch
fell from 165 MB to 76 MB, and it no longer grows with the list's size. The image sets
TMPDIR=/dataso the temporary file stays on the volume.