Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CAR 2.0 regression #640

Merged
merged 2 commits into from May 11, 2019

Conversation

Projects
None yet
3 participants
@lintool
Copy link
Member

commented May 10, 2019

The regressions now replicate the BM25 baseline in https://github.com/nyu-dl/dl4marco-bert
(and provides additional models)

lintool added some commits May 10, 2019

@lintool lintool requested review from rodrigonogueira4 and Victor0118 May 10, 2019

@rodrigonogueira4
Copy link
Collaborator

left a comment

Thanks a lot for adding the 2.0 corpus.

@@ -11,7 +11,7 @@ lucene-index.car17v1.5.pos+docvectors+rawdocs -storePositions -storeDocvectors \
-storeRawDocs >& log.car17v1.5.pos+docvectors+rawdocs &
```

This comment has been minimized.

Copy link
@rodrigonogueira4

rodrigonogueira4 May 10, 2019

Collaborator

nit: CAR is all upper case.

This comment has been minimized.

Copy link
@lintool

lintool May 10, 2019

Author Member

I think for all the file names we downcase? E.g., robust04 not Robust04, nyt instead of NYT, etc.

@@ -20,7 +20,7 @@ For additional details, see explanation of [common indexing options](common-inde
Topics and qrels are stored in `src/main/resources/topics-and-qrels/`, downloaded from NIST:

+ `topics.car17v1.5.test200.txt`: [Topics for the test200 subset (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+ `qrel: qrels.car17v1.5.test200.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+ `qrels.car17v1.5.test200.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)

This comment has been minimized.

Copy link
@rodrigonogueira4

rodrigonogueira4 May 10, 2019

Collaborator

should we use benchmarckY1test as these are the official test queries of 2017?

This comment has been minimized.

Copy link
@lintool

lintool May 10, 2019

Author Member

yes, I'll circle around to do this, but there seems to be some problem with the 1.5 topics, which I gotta ask Laura about...

This comment has been minimized.

Copy link
@Victor0118

Victor0118 May 11, 2019

Member

By the time I build this doc, the qrel file for benchmarckY1test hadn't been out yet. So I use the test200 queries.

I think I noticed a little difference between queries between 2018 and 2017 before. We can handle them together in one topic reader if we have plan to add benchmarckY1test queries.

@Victor0118
Copy link
Member

left a comment

LGTM

@@ -20,7 +20,7 @@ For additional details, see explanation of [common indexing options](common-inde
Topics and qrels are stored in `src/main/resources/topics-and-qrels/`, downloaded from NIST:

+ `topics.car17v1.5.test200.txt`: [Topics for the test200 subset (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+ `qrel: qrels.car17v1.5.test200.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)
+ `qrels.car17v1.5.test200.txt`: [adhoc qrels (TREC 2017 Complex Answer Retrieval Track)](http://trec-car.cs.unh.edu/datareleases/v1.5/test200-v1.5.tar.xz)

This comment has been minimized.

Copy link
@Victor0118

Victor0118 May 11, 2019

Member

By the time I build this doc, the qrel file for benchmarckY1test hadn't been out yet. So I use the test200 queries.

I think I noticed a little difference between queries between 2018 and 2017 before. We can handle them together in one topic reader if we have plan to add benchmarckY1test queries.

@lintool lintool merged commit 2ba2b95 into master May 11, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@lintool lintool deleted the car2.0 branch May 11, 2019

@lintool lintool referenced this pull request May 11, 2019

Open

Unit tests for CAR #641

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.