Skip to content
Permalink
Browse files

Update README.md

Disabling leaderboard. Will be just a casual list of work that enforces PI (similarly to WikiSQL).
  • Loading branch information...
seominjoon committed Apr 27, 2019
1 parent 2e5dde5 commit 44697eb0fe1a51a87a0fc6536c69bf9bb26f422b
Showing with 2 additions and 3 deletions.
  1. +2 −3 README.md
@@ -1,8 +1,7 @@
# Phrase-Indexed Question Answering (PIQA)
- This is the official github repository for [Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension][paper] (EMNLP 2018).
- Much of the work and code is heavily influenced by our former [project][mipsqa] at Google AI.
- For paper-related inquiries, please contact [Minjoon Seo][minjoon] ([@seominjoon][minjoon-github]).
- For code-related inquiries, please create a new issue or contact the admins ([@seominjoon][minjoon-github], [@jhyuklee][jhyuklee-github]).
- For inquiries, please contact [Minjoon Seo][minjoon] ([@seominjoon][minjoon-github]).
- For citation, please use:
```
@inproceedings{seo2018phrase,
@@ -24,7 +23,7 @@ While the challenge shares some similarities with document retrieval, a classic

An important aspect of the challenge is the constraint of *independence* between the **document encoder** and the **question encoder**. As we have noted in our paper, most existing models heavily rely on question-dependent representations of the context document. Nevertheless, phrase representations in PIQA need to be completely *independent* of the input question. Not only this makes the challenge quite difficult, but also state-of-the-art models cannot be directly used for the task. Hence we have proposed a few reasonable baseline models as the starting point, which can be found in this repository.

Note that it is also not so straightforward to strictly enforce the constraint on an evaluation platform such as CodaLab. For instance, current SQuAD 1.1 evaluator simply provides the test dataset (both context and question) without answers, and ask the model to output predictions, which are then compared against the answers. This setup is not great for PIQA because we cannot know if the submitted model abides the independence constraint. To resolve this issue, a PIQA submission must consist of the two encoders with explicit independence, and the retrieval is performed on the evaluator side. While it is not as convenient as a vanilla SQuAD submission, we tried to make it as intuitive and easy as possible for the purpose :)
Note that it is also not so straightforward to strictly enforce the constraint on an evaluation platform such as CodaLab. For instance, current SQuAD 1.1 evaluator simply provides the test dataset (both context and question) without answers, and ask the model to output predictions, which are then compared against the answers. This setup is not great for PIQA because we cannot know if the submitted model abides the independence constraint. To resolve this issue, a submission should consist of the two encoders with explicit independence, and the retrieval is performed on the evaluator side. While it is not as convenient as a vanilla SQuAD submission, it strictly enforces independence constraint.

## Tasks

0 comments on commit 44697eb

Please sign in to comment.
You can’t perform that action at this time.