Megawarc

megawarc is useful if you have .tar full of .warc.gz files and you really want one big .warc.gz. With megawarc you get your .warc.gz, but you can still restore the original .tar.

The megawarc tool looks for .warc.gz in the .tar file and creates three files, the megawarc:

• FILE.warc.gz is the concatenated .warc.gz
• FILE.tar contains any non-warc files from the .tar
• FILE.json.gz contains metadata

You need the JSON file to reconstruct the original .tar from the .warc.gz and .tar files. The JSON file has the location of every file from the original .tar file.

Metadata format

One line with a JSON object per file in the .tar.

{
   "target": {
     "container": "warc" or "tar", // where is this file?
     "offset": number,             // where in the tar/warc does this file start?
                                   // for files in the tar this includes the tar header, which is
                                   // copied to the tar.
     "size": size                  // where does this file end?
                                   // for files in the tar, this includes the padding to 512 bytes
   },
   "src_offsets": {
     "entry": number,              // where is this file in the original tar?
     "data": number,               // where does the data start? entry+512
     "next_entry": number          // where does the next tar entry start
   },
   "header_fields": {
     ...                           // parsed fields from the tar header
   },
   "header_string": string         // the tar header for this entry
}

Usage

megawarc convert FILE

Converts the tar file (containing .warc.gz files) to a megawarc. It creates FILE.warc.gz, FILE.tar and FILE.json.gz from FILE.

megawarc pack FILE INFILE_1 [[INFILE_2] ...]

Creates a megawarc with basename FILE and recursively adds the given files and directories to it, as if they were in a tar file. It creates FILE.warc.gz, FILE.tar and FILE.json.gz.

megawarc restore FILE

Converts the megawarc back to the original tar. It reads FILE.warc.gz, FILE.tar and FILE.json.gz to make FILE.