warcio doesn't verify digests on read

[wumpus]

I was experimenting with injecting digests from my crawler, so that digests aren't computed twice, and noticed that records with a bad WARC-Payload-Digest don't raise an exception on read. No code for it, so I suppose this is a feature request.

The check should be disable-able, and "warcio index" and recompress ought to have a command line flag to ignore digest errors.

Lacking this feature, I don't think that warcio currently has any test to ensure that it's correctly computing digests.

[wumpus]

I coded this up by hacking LimitReader to be configurable by recordloader.py to test digests. If I turn it on for all of the existing tests, the only ones that fail are the truncated warc (expected) and the block_digest revisit record in example.warc (which might be your bug, not mine... I'll figure it out)

Because of your enthusiasm for streaming, the digest check will only fire if the user reads to the end of the record. So be it.

I have a few style questions:

1. Should I raise ArchiveLoadFailed if the digest check fails? ArchiveDigestFailed? ValueError?

2. Should the digest check default off? I didn't check perf, but it ought to be a lot cheaper than decompressing.

[wumpus]

New try: #54

[wumpus]

Done by "warcio check"