WARC splitting feature

[ivbeg]

Some cloud storages have limitations on file size. For example Rackspace Cloud Files do not allow files more than 3GB.
It would be great to have WARC splitting feature as part of website crawler.

[chfoo]

--warc-max-size NUM is implemented in v0.25 but the WARC files will be bigger than NUM bytes especially on large files. Using WARC record segmentation (described in section 7) would fix this.

[chfoo]

I think WARC record segmentation would not be helpful. Wpull requires using temporary files and if the temporary file is too large, then Wpull cannot construct the next WARC record.

--warc-max-size can still be used, but managing files on the filesystem is out of scope of Wpull. Feel free to reopen this issue if it is still a problem.