Fix for --rolloverSize for individual WARCs in 1.x

[ikreymer]

Fixes #533

Fixes rollover in WARCWriter, separate from combined WARC rollover size:

• check rolloverSize and close previous WARCs when size exceeds
• add timestamp to resource WARC filenames to support rollover, eg. screenshots-{ts}.warc.gz
• use append mode for all write streams, just in case
• tests: add test for rollover of individual WARCs with 500K size limit
• tests: update screenshot tests to account for WARCs now being named screenshots-{ts}.warc.gz instead of just screenshots.warc.gz

[tw4l]

Looks good and testing well!

Only thought is that because we're only rolling over once the size has been exceeded, occasionally a file gets quite a bit over the rollover size before a new one is started. Maybe eventually we could look into optimizing that a little, but I think for now this is a good improvement.

[ikreymer]

Looks good and testing well!

Only thought is that because we're only rolling over once the size has been exceeded, occasionally a file gets quite a bit over the rollover size before a new one is started. Maybe eventually we could look into optimizing that a little, but I think for now this is a good improvement.

Yeah, we could potential adjust this to account for size after next records are written, but that's a bit more work, and generally this should be pretty good. For small rollover limits, there's always chance of a file exceeding it, eg. if a 100MB video is downloaded and limit is 10MB, there will still be a 100MB WARC