v2.4.0

HexDB 2.4.0 scans every uploaded file for viruses and malware before it's stored, and fixes the last causes of a rare full resync after an Overseer handover.

Added

• Malware scanning, on by default. Every upload is scanned after its content reaches disk and before its document commits, so a file that doesn't pass never becomes a document and never reaches another hex. Two engines, both free and open source, with no accounts:
  • ClamAV, through its daemon (clamd) over TCP or a Unix socket. HexDB never links ClamAV, so its GPL licence stays separate. files.scan.clamd = "auto" (the default) looks for a clamd on the same machine at startup; give an address (tcp://clamav:3310) to use another, or off.
  • YARA-X, built in, with HexDB's own rules (starting with the EICAR test file). files.scan.yara_rules adds a directory of your own rules or a community set such as YARA Forge (see the MANUAL for the download). A rule file that doesn't compile is left out and named in a warning; it doesn't stop HexDB.
• Quarantine. A file that matches is kept on the Overseer only, in the local-only _quarantine tessellation: it isn't in its tessellation, can't be downloaded, and never replicates. The upload gets 422 file_quarantined naming what matched. Administrators review it on the new Quarantine page (under Security), or with GET /quarantine, POST /quarantine/{id}/release (store a false positive as a normal file) and DELETE /quarantine/{id}. Every action is audited.
• Rescans. When ClamAV's signatures or the YARA rules change, the Overseer scans every stored file's current version again in the background, at most once every files.scan.rescan_hours (24), and quarantines what new signatures find. POST /quarantine/rescan (or "Rescan now") starts one at once. Progress is in /status (storage.scan.rescan).
• Fail closed. While a configured ClamAV can't answer, uploads are refused with 503 scanner_unavailable and Retry-After; nothing unscanned is stored.
• Turning it off and on. The lattice setting files.scanning (Settings > Files) turns scanning off or on for the whole lattice without a restart. files.scan.enabled = false turns it off on one hex. files.uploads = false refuses new uploads on a hex while stored files stay readable.
• Each file's verdict is in its document (scan: status, detections, engines, scanned_at), in the file details in the admin UI, and the dashboard has a Malware scanning card.
• ClamAV in Docker Compose and the Helm chart, connected to HexDB by default (clamav.enabled in the chart). The shipped deploy/clamav/clamd.conf raises ClamAV's limits to match HexDB's (2000 MB) and reports files it can't fully check instead of passing them.

Upgrading

• Scanning starts as soon as a hex runs 2.4.0. Without a ClamAV, uploads are scanned by the built-in YARA rules alone, and HexDB warns about it in its log, /status and the dashboard. See "Setting up ClamAV" in the MANUAL.
• ClamAV needs 3 to 4 GB of memory (about 1.2 GB for its signatures, twice that briefly while it loads new ones). ConcurrentDatabaseReload no keeps it near 1.3 GB, at the cost of a pause of about a minute while it reloads.
• The first rescan after upgrading scans every stored file once, one at a time in the background.
• YARA-X needs a 64-bit platform. Every release build is 64-bit; a 32-bit build from source turns it off with --no-default-features --features video-thumbnails and relies on ClamAV.

Fixed

• No full resync after a handover. Two rare races could leave a replica one change ahead of a new Overseer, which then made it copy everything again: a write that passed the writable check just before an Overseer began stepping down could still be published after the step-down compared replicas, and a long poll for changes accepted before a handover could answer with the old Overseer's first write as a replica. Neither happens now.

Dependencies

• Updated zip, tonic, lopdf and the admin UI's packages. Two advisories in YARA-X's dependencies are explained in .cargo/audit.toml: rsa (Marvin, RUSTSEC-2023-0071; YARA-X only verifies signatures, which isn't affected) and bincode (unmaintained).

What's Changed

• Bump @types/node from 26.6.5 to 26.6.6 in /drivers/node by @dependabot[bot] in #68
• Bump lopdf from 0.45.0 to 0.46.0 by @dependabot[bot] in #73
• Bump the admin-ui group across 1 directory with 5 updates by @dependabot[bot] in #69
• Bump the rust group across 1 directory with 4 updates by @dependabot[bot] in #72
• Bump cm6-graphql from 0.2.2 to 1.0.0 in /hexdb_admin by @dependabot[bot] in #70

Full Changelog: v2.3.3...v2.4.0