Openwayback replays metadata record for the revisit records with duplicate hash instead of the original response

[aalsum]

I used H3.2.0 to crawl a set of seed URIs that are identical. The WARC file has been generated successfully with the recommend WARC-Refers-To-Target-URI and WARC-Refers-To-Date fields.

When I try to replay the WARC file in WM, it reads the metadata record instead of the revisit record.

I'm using WM built from the latest master branch on github.

WARC: http://stanford.edu/~aalsum/data/WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
CDX: http://stanford.edu/~aalsum/data/index.cdx

Original URI: http://www.cs.odu.edu/~aalsum/test1.txt (works fine)
Duplicate URI: http://www.cs.odu.edu/~aalsum/test3.txt

[PsypherPunk]

I've been looking at this but it occurs to me: isn't this expected behaviour? The metadata and request objects are in the CDX which means OpenWayback will try and return the first matching, indexed record for that URL (i.e. the metadata record which appears first in the sort-order; if they're removed from the CDX it tries to return the request with a Bad Content Exception and if they're also removed, it works).

Or should OpenWayback always skip these types of records? In which case, should they never appear in the CDX as they're just taking up space?

[aalsum]

I agree with you, it shouldn't appear in the CDX but I think this is another issue. The main issue right now is how to make the WM replies the records with warc/revisit because if I fixed the CDX manually by removing the request/metadata records, it still can't reply the warc/revisit records.

[PsypherPunk]

Okay, that's odd: I'm using this CDX with the current build of OpenWayback and visiting http://www.cs.odu.edu/~aalsum/test3.txt correctly returns the payload from test1.txt:

 CDX N b a m s k r M S V g
cs.odu.edu/favicon.ico 20140806165413 http://www.cs.odu.edu/favicon.ico warc/revisit - 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ - - 352 5637 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
cs.odu.edu/robots.txt 20140806165400 http://www.cs.odu.edu/robots.txt warc/revisit - PJ7OVFJC4MDZEY6JXVH4GAE7RRWNALQG - - 521 719 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
cs.odu.edu/~aalsum/test1.txt 20140806165403 http://www.cs.odu.edu/~aalsum/test1.txt text/plain 200 WAD5J5SCREFCU3W6B5APWJXKD7Q3KCRF - - 420 1992 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
cs.odu.edu/~aalsum/test2.txt 20140806165406 http://www.cs.odu.edu/~aalsum/test2.txt warc/revisit - WAD5J5SCREFCU3W6B5APWJXKD7Q3KCRF - - 519 3139 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
cs.odu.edu/~aalsum/test3.txt 20140806165409 http://www.cs.odu.edu/~aalsum/test3.txt warc/revisit - WAD5J5SCREFCU3W6B5APWJXKD7Q3KCRF - - 519 4385 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
warcinfo:/WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz/ia-web-commons.1.1.2-20140423093321 20140806165357 warcinfo:/WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz/ia-web-commons.1.1.2-20140423093321 warc-info - W4BPBMXLG2ROBPMZ7U5J3AMYDCOAAGVF - - 477 0 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz
www.cs.odu.edu 20140806165356 dns:www.cs.odu.edu text/dns - 4A45M67PVPEAAOO4QB2DYXVY6NPSWMV2 - - 242 477 WEB-20140806165357001-00000-2081~heritrix-dev.Stanford.EDU~8443.warc.gz

With the extra records stripped, is OpenWayback logging any errors?

[aalsum]

Thanks Roger to look to this. This is what I get when using your CDX file.

curl -L http://localhost:8080/wayback/20140806165409/http://www.cs.odu.edu/~aalsum/test1.txt
a
b
c
d

curl -I http://localhost:8080/wayback/20140806165409/http://www.cs.odu.edu/~aalsum/test3.txt
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Memento-Datetime: Wed, 06 Aug 2014 16:54:09 GMT
Link: http://www.cs.odu.edu/~aalsum/test3.txt; rel="original", http://localhost:8080/wayback/timemap/link/http://www.cs.odu.edu/~aalsum/test3.txt; rel="timemap"; type="application/link-format", http://localhost:8080/wayback/http://www.cs.odu.edu/~aalsum/test3.txt; rel="timegate", http://localhost:8080/wayback/20140806165409/http://www.cs.odu.edu/~aalsum/test3.txt; rel="first last memento"; datetime="Wed, 06 Aug 2014 16:54:09 GMT"
X-Archive-Orig-Content-Length: 9
X-Archive-Orig-Connection: close
X-Archive-Orig-Content-Type: text/plain
X-Archive-Orig-ETag: "9-4fff8b86d8487"
X-Archive-Orig-Accept-Ranges: bytes
X-Archive-Orig-Server: nginx
X-Archive-Orig-Last-Modified: Wed, 06 Aug 2014 16:46:58 GMT
X-Archive-Orig-Date: Wed, 06 Aug 2014 16:54:10 GMT
Content-Type: text/plain
Content-Length: 9
Date: Fri, 22 Aug 2014 14:00:22 GMT

curl http://localhost:8080/wayback/20140806165409/http://www.cs.odu.edu/~aalsum/test3.txt
curl: (18) transfer closed with 9 bytes remaining to read

From curl reference page, CURLE_PARTIAL_FILE (18): A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn't match the previously given size.

Wyabck machine doesn't log anything in this case.

[PsypherPunk]

@aalsum, are you still seeing this problem? Can't replicate this on a system here at all.

[aalsum]

Yes Roger, until now I couldn't avoid this behavior on our system.

[aalsum]

The problem was in the index file itself, I used an old indexer that indexed the metadata and request records. Now, it works fine with me.