Skip digest of empty payload (or static whitelist)

[alexkiro]

Originally reported by gryphius.

We're seing a lot of pyzor "false positives" from messages with attachments but little or no body text. these messages are all different but generate the same digest da39a3ee5e6b4b0d3255bfef95601890afd80709, which is the sha1-sum of the empty string . It looks like this is is the digest produced if all content is stripped out by the pyzor normalizer.

current public.pyzor.org result for this hash:
public.pyzor.org:24441 (200, 'OK') 159015 5706
pyzord could maybe treat this special hash as statically whitelisted (whithout the need to have clients submit this hash into the whitelist first) and always return a zero hitcount.

This would be especially helpful in spamassassin setups, where only the hitcount is checked ( https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6108 )
if hardcoding this hash is not an option, you could maybe add a config option to read a static whitelist from a file.

Attached is a quick & dirty patch we're using to skip this hash.

[alexkiro]

I was under the impression that SpamAssassin checks the exit code of the pyzor check. Which is:

• 1 if the report count is 0 or the whitelist count is > 0
• 0 if the report count is > 0 and the whitelist count is 0

These thresholds are also now configurable as of 0.6 from the configuration file:

## These options specify the threshold for number of reports/whitelists. 
## According to these thresholds the pyzor client exit code will differ.
# ReportThreshold = 0
# WhitelistThreshold = 0

[alexkiro]

Comment by gryphius:

Even if SA checks the exit code, this doesn't solve the problem for this special digest. Hash collisions like this cause problems in every case: checking whitelist could cause false negatives, not checking whitelist causes false postivies - this is what I see in our setups.

Maybe the real fix would be changing the hashing algorithm so that empty strings (after stripping) are ignored and hash the full unstripped message or at least try to get some unique header value for building the hash?

[alexkiro]

Let's solve this with the local static whitelist in 1.0. We'll also need to add a command that adds remove digests from the list. It should accept messages in various formats (including digests) just like the normal whitelist/report commands.