Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial replication failure on ClueWeb12-B13 for ax models #592

Closed
lintool opened this issue Apr 15, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@lintool
Copy link
Member

commented Apr 15, 2019

According to #590 @matthew-z wasn't able to replicate results with ax models on ClueWeb12-B13. We should look into this...

@matthew-z

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

Somehow the AX results matched after indexing again with oracle JVM (1.8.0_201)....

@lintool

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

Interesting.

Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM. This would explain the differences in ax results, which rely on sampling docs.

@matthew-z Just to be clear, with Oracle JVM you are able to replicate our figures exactly, while with OpenJDK you come close? (If so, how close is "close" in the latter case?)

@matthew-z

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

Did you mean no-AX algorithms? They matched exactly.

E.g., 201-250, Bm25, NDCG@20, it got 0.12862 and expected 0.1286

201-250, QL, NDCG@20, it got 0.11067 and expected 0.1107

@lintool

This comment has been minimized.

Copy link
Member Author

commented Apr 18, 2019

Okay, thanks! I'll send a PR to capture this discussion and ask you to review.

@lintool

This comment has been minimized.

Copy link
Member Author

commented Apr 18, 2019

@matthew-z please look at #594

@lintool

This comment has been minimized.

Copy link
Member Author

commented Apr 18, 2019

Resolved with #594, closing.

@lintool lintool closed this Apr 18, 2019

@Peilin-Yang

This comment has been minimized.

Copy link
Collaborator

commented May 21, 2019

Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM.

I am not sure, but this is unlikely IMO.
I am going to test it again by myself

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.