Skip to content

Latest commit

 

History

History
173 lines (120 loc) · 9.4 KB

odqa.rst

File metadata and controls

173 lines (120 loc) · 9.4 KB
Raw

Open Domain Question Answering Skill on Wikipedia

Task definition

Open Domain Question Answering (ODQA) is a task to find an exact answer to any question in Wikipedia articles. Thus, given only a question, the system outputs the best answer it can find. The default ODQA implementation takes a batch of queries as input and returns the best answer.

Quick Start

The example below is given for basic ODQA config en_odqa_infer_wiki <odqa/en_odqa_infer_wiki.json>. Check what other ODQA configs <odqa_configuration> are available and simply replace en_odqa_infer_wiki with the config name of your preference.

Before using the model make sure that all required packages are installed running the command:

python -m deeppavlov install en_odqa_infer_wiki

Training (if you have your own data)

from deeppavlov import configs, train_evaluate_model_from_config

train_evaluate_model_from_config(configs.doc_retrieval.en_ranker_tfidf_wiki, download=True)
train_evaluate_model_from_config(configs.squad.multi_squad_noans, download=True)

Building

from deeppavlov import build_model, configs

odqa = build_model(configs.odqa.en_odqa_infer_wiki, download=True)

Inference

result = odqa(['What is the name of Darth Vader\'s son?'])
print(result)

Output:

>> Luke Skywalker

Languages

There are pretrained ODQA models for English and Russian languages in DeepPavlov </index/>.

Models

The architecture of ODQA skill is modular and consists of two models, a ranker and a reader. The ranker is based on DrQA proposed by Facebook Research and the reader is based on R-NET proposed by Microsoft Research Asia and its implementation by Wenxuan Zhou.

Running ODQA

Note

About 24 GB of RAM required. It is possible to run on a 16 GB machine, but than swap size should be at least 8 GB.

Training

ODQA ranker and ODQA reader should be trained separately. Read about training the ranker here <ranker_training>. Read about training the reader in our separate reader tutorial <reader_training>.

Interacting

When interacting, the ODQA skill returns a plain answer to the user's question.

Run the following to interact with English ODQA:

python -m deeppavlov interact en_odqa_infer_wiki -d

Run the following to interact with Russian ODQA:

python -m deeppavlov interact ru_odqa_infer_wiki -d

Configuration

The ODQA configs suit only model inferring purposes. For training purposes use the ranker configs <ranker_training> and the reader configs <reader_training> accordingly.

There are several ODQA configs available:

Config Description
en_odqa_infer_wiki <odqa/en_odqa_infer_wiki.json> Basic config for English language. Consists of TF-IDF ranker and reader. Searches for an answer in enwiki20180211 Wikipedia dump.
en_odqa_infer_enwiki20161221 <odqa/en_odqa_infer_enwiki20161221.json> Basic config for English language. Consists of TF-IDF ranker and reader. Searches for an answer in enwiki20161221 Wikipedia dump.
ru_odqa_infer_wiki <odqa/ru_odqa_infer_wiki.json> Basic config for Russian language. Consists of TF-IDF ranker and reader. Searches for an answer in ruwiki20180401 Wikipedia dump.
en_odqa_pop_infer_enwiki20180211 <odqa/en_odqa_pop_infer_enwiki20180211.json> Extended config for English language. Consists of TF-IDF Ranker, Popularity Ranker and reader. Searches for an answer in enwiki20180211 Wikipedia dump.

Comparison

Scores for ODQA skill:

+-------------------------------------------------------------------------------------+------+----------------------+----------------+---------------------+---------------------+ | | | | | Ranker@5 | Ranker@25 | | | | | +----------+----------+-----------+---------+ | Model | Lang | Dataset | WikiDump | F1 | EM | F1 | EM | +-------------------------------------------------------------------------------------+------+----------------------+----------------+----------+----------+-----------+---------+ | | enwiki20180211 | 35.89 | 29.21 | 39.96 | 32.64 | +-------------------------------------------------------------------------------------+ + +----------------+----------+----------+-----------+---------+ En | SQuAD (dev) | | 37.83 41.86 | 34.73 | +-------------------------------------------------------------------------------------+ + + +----------+----------+-----------+---------+ | | | - | 27.1 | - | - | +-------------------------------------------------------------------------------------+ + + +----------+----------+-----------+---------+ | | enwiki20161221 | 37.5 | 29.1 | - | - | +-------------------------------------------------------------------------------------+------+----------------------+----------------+----------+----------+-----------+---------+ | | | 42.02 - | - | +-------------------------------------------------------------------------------------+ Ru + SDSJ Task B (dev) + ruwiki20180401 +----------+----------+-----------+---------+ | | | 28.56 | 18.17 | - | - | +-------------------------------------------------------------------------------------+------+----------------------+----------------+----------+----------+-----------+---------+

EM stands for "exact-match accuracy". Metrics are counted for top 5 and top 25 documents returned by retrieval module.

References