UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499

[hildeb]

On the initial run of clamsbsync.py I'm getting:

UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499
UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499
UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499
UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499
UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499

[hildeb]

More context:

# clamsbsync.py -v build
root: INFO: Running update.py CLI
UpdateClient: INFO: processing changes for MALWARE.URL.ALL_PLATFORMS from NEW to Cg0IARAGGAEiAzAwMTABENTEBxoCGAnHSw42
UpdateClient: INFO: adding 41324 new entries to MALWARE.URL.ALL_PLATFORMS
UpdateClient: INFO: updating state for MALWARE.URL.ALL_PLATFORMS from NEW to Cg0IARAGGAEiAzAwMTABENTEBxoCGAnHSw42
UpdateClient: INFO: sync success for MALWARE.URL.ALL_PLATFORMS - db.update 0:2:26.142
UpdateClient: INFO: computed checksum of prefixes for MALWARE.URL.ALL_PLATFORMS = 0a95bdaf5c05d236e97dce9435450aba9ea99bb6b0c25854d167e616d8684220
UpdateClient: INFO: computed checksum of prefixes matches expected checksum
UpdateClient: INFO: processing changes for SOCIAL_ENGINEERING.URL.ALL_PLATFORMS from NEW to Cg0IAhAGGAEiAzAwMTABEOTxCBoCGAlmDTAL
UpdateClient: WARNING: prefix set does not fully match retrieved list: expected 500 =/= retrieved 499
UpdateClient: INFO: commiting additions segment for SOCIAL_ENGINEERING.URL.ALL_PLATFORMS

[kevlin2]

This shouldn't be too much of an issue, the safebrowsing database pulls from two parts of Google's API: list updates and full hash lookups.

It first retrieves the full list of changes since the last update with includes directions on changing an established prefix list. Then, since ClamAV requires the full hash, queries the full hash service with those prefixes.

Since there is some time in between the queries and Google is always constantly updating lists, there is a chance that the associated full hash to a prefix may go stale in between. Thus a lookup mismatch occurs.

The 500 =/= 499 means that of the 500 prefixes submitted for the given query only 499 full hashes were returned.

Ultimately, this is fine and will not affect future updates as the list updates and full hash lookups are independent.

[micahsnyder]

While the warnings described are not a problem, I think we should leave this issue open to make it easier for people who encounter the warning to find this explanation.

If anyone from the community is keen to submit a PR to reduce the severity of the message add additional explanation to the message, you're welcome to.