Nondeterminism in documents indexed for Gov2?

[lintool]

Indexing Gov2 on streeling at UMD, I get 24899563 docs.
Luchen reports 24900602 docs indexing on hops.

Weird - some non-determinism the multi-threading?

Not that important if we can replicate effective results on standard test collections, but worth noting.

[LuchenTan]

I redo the Gov2 indexing on Hops at UWaterloo, and the same number of documents, 24900602 docs indexed.

[claclark]

1000 documents don't appear from nowhere

[lintool]

As Charlie suggested in email (and the sensical thing to do), we should dump out the docids and diff.

@LuchenTan so you can start learning the Lucene APIs, etc., can you write a simple program that does this?

Fork the repo, add in your code, and send a pull request. The ultimate product should be a command I can copy and paste to generate the output (documented in the README).

Thanks!

[LuchenTan]

OK, I'll do it.

[lintool]

Update on this: as is turned out, there was a minor corruption in the copy of Gov2 at UMD. After fixing this and rerunning the indexer, I get 24900602 docs, which machines @LuchenTan's number.

[LuchenTan]

Yes, exactly the same number of mine.

On Thu, Oct 15, 2015 at 2:02 PM, Jimmy Lin notifications@github.com wrote:

Update on this: as is turned out, there was a minor corruption in the copy
of Gov2 at UMD. After fixing this and rerunning the indexer, I get 24900602
docs, which machines @LuchenTan https://github.com/LuchenTan's number.

—
Reply to this email directly or view it on GitHub
#7 (comment).

Luchen Tan

David R. Cheriton School of Computer Science
University of Waterloo
Waterloo, Ontario, Canada N2L 3G1

[lintool]

Resolving issue... basically, no issue.

[lintool]

Merged in Luchen code to dump out docids in an index:
commit 491dd9e