No block digest written for warcinfo records

[JustAnotherArchivist]

I noticed today that warcio doesn't generate a block digest for warcinfo records:

warcio/warcio/recordbuilder.py

Line 30 in 8e3ceb7

NO_BLOCK_DIGEST_TYPES = ('warcinfo')

This seems to have been introduced in a791617, but I was unable to figure out why. The spec permits block digests on any record (whereas a payload digest would make no sense on a warcinfo record due to the content type used normally), and it seems good practice to me to always store a digest to allow for integrity checks.

[JustAnotherArchivist]

Side note: NO_BLOCK_DIGEST_TYPES is actually a string, not a tuple as certainly intended. That won't have any negative effect though unless someone is writing custom records with a WARC-Type that is a substring of warcinfo.

[wumpus]

The ('warcinfo') aspect of the bug is old. It was (accidentally) introduced in

25194bc
warcwriter: if writing a record with no Content-Length, buffer and compute length, as well as digests, as needed

You're right that any recordtype that's a subset of the string 'warcinfo' will match, fortunately there is no such standard recordtype.

As for that explicit code that stops writing block digests for warcinfo records, that code was in the original checkin, when the code was split out of webrecorder. I'm kind of surprised that I didn't notice it while working on "warcio test".

[JustAnotherArchivist]

I dug around a bit more. a791617 is not the commit that introduced this behaviour. It appears that it was already like that back when warcinfo records were first added in webrecorder/pywb@d40edfc2; at the time, digests were only written for responses as far as I can tell.

I can't really think of any good reason why block digests shouldn't simply always be present for verification purposes. The only exception would be zero-length records, where the digest is useless, but those are so rare and odd that worrying about them isn't worth it. I'll create a PR shortly.