Pywb 2.0.0 doesn't work with WARCs generated by Wget 1.19

[ghost]

Commands to reproduce:

pip3 install -U pywb

wget --warc-file=example.com --warc-cdx "http://example.com/" -O /dev/null
# this creates example.com.warc.gz, example.com.cdx

wb-manager init example

cp example.com.warc.gz collections/example/archive/
cp example.com.cdx collections/example/indexes/

wb-manager cdx-convert collections/example/indexes/
# type 'y'

wayback

Then go to localhost:8080/example/, type in example.com, click the record, and you should get an error like the following (or at least I do):

Error Details:

{'args': {'coll': 'example', 'type': 'replay'}, 'error': '{"message": ": invalid literal for int() with base 10: \'example.com.warc.gz\'", "errors": {"WARCPathLoader": ": invalid literal for int() with base 10: \'example.com.warc.gz\'"}}'}

[ikreymer]

Looks like the CDX format used by wget is one that pywb doesn't currently support.

The recommended way is to let pywb reindex the WARC (and this is also less commands to type!):

wb-manager init example
wb-manager add example example.com.warc.gz

[ghost]

@ikreymer That was actually what I tried first: just do wb-manager add. However, if I do that, then when I search for example.com in the archive, it doesn't seem to find anything (0 captures of http://example.com), even though the index.cdxj file seems to be generated correctly. I'm not sure if this is a problem on my end. Can you reproduce it?

If it helps:
OS: macOS 10.13.2
wget: 1.19.4
python: 3.6.4
Edited to add details.

[ghost]

For what it's worth, I don't see anything in Webrecorder Player either. However, warcat is able to extract the content.

[ghost]

Also, I was able to reproduce the issue using wget 1.19.4 compiled from source on a Debian VM. However, there seems to be no issue when using wget 1.18.

Moreover, I believe I have determined the source of the problem:

Wget 1.18 gives WARC output like:

WARC-Target-URI: http://example.com/

However, 1.19.4 outputs the following:

WARC-Target-URI: <http://example.com/>

So you can actually access the page, but you have to type <http://example.com/> into the input field of pywb to get the page!

The wget maintainers seem to have noticed this:

http://lists.gnu.org/archive/html/bug-wget/2017-11/msg00050.html ("Validity of angle brackets around WARC-Target-URI value")

[ikreymer]

Nice find, thanks for looking into this!
This would indeed cause the problem.. most WARCs I've seen do not have the brackets, and most WARC tools do not produce brackets there. Seems like this is an unfortunate side-effect of ambiguity.. Looks like it'll be removed in a future version.

It might make sense to strip the brackets now since there are WARCs being created with them in place.

Would you mind attaching a sample WARC with brackets to this issue?

[ghost]

Here's an example WARC with brackets: example.com.warc.gz

[ikreymer]

warcio 1.6.x now supports reading the wget WARCs with WARC-Target-URI wrapped in angle brackets. (This has been in wget for a while and is quite widespread).

The change is in warcio, which can be upgraded independently of pywb with pip install -U warcio.

The above example WARC and other wget will then be read correctly by pywb.