You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using a SequenceFileBolt won't allow us to generate a well formed WARC file. See [https://github.com/ept/warc-hadoop/blob/master/src/main/java/com/martinkl/warc/mapred/WARCOutputFormat.java] for an example of using a custom outputformat.
The WARC standard recommends to compress every record independently "record-at-time". This feature is required by wayback machines including http://index.commoncrawl.org/.
Copied from commoncrawl/news-crawl#4
The text was updated successfully, but these errors were encountered: