Cannot read warc create by wget 1.19.4

[joshuaavalon]

Version 1.0.9 64 bit
OS: Window

warc.sh

USER_AGENT="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27"
SAVE_HOST="$1"
DATE=`date +%Y-%m-%d`
WARC_NAME="$SAVE_HOST-$DATE"
 
wget \
    -e robots=off --mirror --page-requisites \
    --waitretry 5 --timeout 60 --tries 5 --wait 1 \
    --warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
    -U "$USER_AGENT" "$SAVE_HOST"

sh warc.sh example.com

I have tested with wget 1.17.1 and wget 1.19.4. The program can only read created warc created by 1.17.1. It show blank page for warc created by 1.19.4.

[vitorio]

wget 1.19 writes WARC-Target-URI headers with brackets around the URL, which breaks some software. Maybe that's what happening here? If you're able, perhaps you could try rewriting the WARC without those brackets and see if that fixes it?

Here's some example Python code to do that using warcio:

>>> from warcio.archiveiterator import ArchiveIterator
>>> from warcio.warcwriter import WARCWriter
>>> output = open('lfes-not-in-ia-1.warc.gz', 'wb')
>>> writer = WARCWriter(output, gzip=True)
>>> with open('brackets.lfes-not-in-ia-1.warc.gz', 'rb') as stream:
...     for record in ArchiveIterator(stream):
...             if 'WARC-Target-URI' in record.rec_headers:                     
...                     record.rec_headers['WARC-Target-URI'] = record.rec_headers['WARC-Target-URI'].lstrip('<').rstrip('>')
...             writer.write_record(record)                                     
... 
>>> output.close()

[joshuaavalon]

@vitorio Yes it works.

[nvanderperren]

I have the same issue. I'm not that experienced that I can rewrite the WARC. Can I use the python code? (and how do I do this?)

[ikreymer]

Good news! We've recently added support for wget 1.19.4 WARCs in our WARC reader library (webrecorder/warcio#42) so these types of WARCs should "just work" without any changes.

The next update release of Webrecorder Player will include this fix.

[nvanderperren]

this is great news! thank you!

[ikreymer]

We've just released 1.6.0 (https://github.com/webrecorder/webrecorderplayer-electron/releases/tag/v1.6.0) and you should now be able to open wget 1.19.4+ WARCs