Skip to content

MRL-CQA for EMNLP 2020 submission. This work has been accepted by EMNLP 2020.

License

Notifications You must be signed in to change notification settings

DevinJake/MRL-CQA

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

33 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MRL-CQA: for EMNLP 2020 submission.

We aim to solve the CQA task [1], which is answering factual questions through complex inferring over a realistic-sized KG of millions of entities.
We could check the details of the CQA dataset here.
All the materials required for running the KG sever, training the model, and testing in this task could be downloaded from the data link.
Following this README, we will instruct how to use the relevant data from the data link.

The questions in the CQA could be categorized into seven groups.
The typical examples of these seven question types are displayed in the following table.

Question Type Question KB artifacts Action Sequence Answer
Simple Which administrative territory is Danilo Ribeiro an inhabitant of? E1: Danilo Ribeiro
R1: country of citizenship
T1: administrative territory
Select(E1, R1, T1) Brazil
Logical Which administrative territories are twin towns of London but not Bern? E1: London
E2: Bern
R1: twinned adm. body
T1: administrative territory
Select(E1, R1, T1)
Diff(E2, R1, T1)
Sylhet, Tokyo, Podgorica,
Phnom Penh, Delhi,
Los Angeles, Sofia, New Delhi, ...
Quantitative Which sports teams have min number of stadia or architectural structures as their home venue? R1: home venue
T1: sports team
T2: stadium
T3: architectural structure
SelectAll(T1, R1, T2)
SelectAll(T1, R1, T3)
ArgMin()
Detroit Tigers, Drbak-Frogn IL,
Club Sport Emelec, Chunichi Dragons, ...
Comparative Which buildings are a part of lesser number of architectural structures and universities than Midtown Tower? E1: Midtown Tower
R1: part of
T1: building
T2: architectural structure
T3: university
SelectAll(T1, R1, T2)
SelectAll(T1, R1, T3)
LessThan(E1)
Amsterdam Centraal, Hospital de Sant Pau,
Budapest Western Railway Terminal,
El Castillo, ...
Verification Is Alda Pereira-Lemaitre a citizen of France and Emmelsbull-Horsbull? E1: Alda Pereira-Lemaitre
E2: France
E3: Emmelsbull-Horsbull
R1: country of citizenship
T1: administrative territory
Select(E1, R1, T1)
Bool(E2)
Bool(E3)
YES and NO respectively
Quantitative Count How many assemblies or courts have control over the jurisdiction of Free Hanseatic City of Bremen? E1: Bremen
R1: applies to jurisdiction
T1: deliberative assembly
T2: court
Select(E1, R1, T1)
Union(E1, R1, T2)
Count()
2
Comparative Count How many art genres express more number of humen or concepts than floral painting? E1: floral painting
R1: depicts
T1: art genre
T2: human
t3: concept
SelectAll(T1, R1, T2)
SelectAll(T1, R1, T3)
GreaterThan(E1)
Count()
8

Now we will talk about how to training and testing our proposed model.

1. Experiment environment.

(1). Python = 3.6.4
(2). PyTorch = 1.1.0
(3). TensorFlow = 1.7.0
(4). tensorboardX = 2.0
(5). ptan = 0.4
(6). flask = 1.1.1
(7). requests = 2.22.0

All the materials required for running the KG sever, training the model, and testing could be found from the data link.

2. Accessing knowledge graph.

(1). Assign the IP address and the port number for the KG server.

Manually assign the IP address and the port number in the file of the project MRL-CQA/BFS/server.py.
Insert the host address and the post number for your server in the following line of the code:

app.run(host='**.***.**.**', port=####, use_debugger=True)

Manually assign the IP address and the port number in the file of the project MRL-CQA/S2SRL/SymbolicExecutor/symbolics.py.
Insert the host address and the post number for your server in the following three lines of the code in the symbolics.py:

content_json = requests.post("http://**.***.**.**:####/post", json=json_pack).json()

(2). Run the KG server.
Download the bfs data bfs_data.zip from the provided data link.
We need to uncompress the file bfs_data.zip and copy the three pkl files into the folder MRL-CQA/data/bfs_data.
Run the file MRL-CQA/BFS/server.py to activate the KG server for retrieval:

python server.py

3. MAML training.

(1). Load pre-trained model.
We pre-trained a model based on Reinforcement learning, and further trained our MAML model on the basis of the RL model.
We could download and uncompress the RL model truereward_0.739_29.zip in the folder MRL-CQA/data/saves/rl_even_TR_batch8_1% from the provided data link.

(2). Train the MAML model.
Furthermore, we need some extra files in folder MRL-CQA/data/auto_QA_data for training the model: share.question (vocabulary), CSQA_DENOTATIONS_full_944K.zip (the file that records the information relevant to all the training questions), CSQA_result_question_type_944K.json, CSQA_result_question_type_count944K.json, CSQA_result_question_type_count944k_orderlist.json, and 944k_rangeDict.json (the files that are used to retrieve the support sets).
Also, we have processed the training dataset and thus we need to download the file RL_train_TR_new_2k.question from the folder MRL-CQA/data/auto_QA_data/mask_even_1.0%.

All we need is to download the aforementioned files from the data link and further put them under the corresponding folders in our project.
Then in the folder MRL-CQA/S2SRL, we run the python file to train the MAML model:

python train_reptile_maml_true_reward.py

The trained models would be stored in the folder MRL-CQA/data/saves/maml_reptile.

4. MAML testing.

(1). Load trained model.
The trained models will be stored in the folder MRL-CQA/data/saves/maml_reptile.
We have saved a trained model epoch_020_0.784_0.741.zip in this folder, which could lead to the SOTA result.
We could download such model from the data link.
When testing the model, we could choose a best model from all the models that we trained, or simply use the saved model epoch_020_0.784_0.741.dat.

(2). Load the testing dataset.
We also have processed the testing dataset SAMPLE_FINAL_MAML_test.question (which is 1/20 of the full testing dataset) and FINAL_MAML_test.question (which is the full testing dataset), and saved them in the folder MRL-CQA/data/auto_QA_data/mask_test.
We could download the files from the data link and put them under the folder MRL-CQA/data/auto_QA_data/mask_test in the project.

(3). Testing.
In the file MRL-CQA/S2SRL/data_test_maml.py, we could change the parameters to meet our requirement.
In the command line:

sys.argv = ['data_test_maml.py', '-m=epoch_020_0.784_0.741.dat', '-p=sample_final_maml',
'--n=maml_reptile', '--cuda', '-s=5', '-a=0', '--att=0', '--lstm=1',
'--fast-lr=1e-4', '--meta-lr=1e-4', '--steps=5', '--batches=1', '--weak=1', '--embed-grad']

, we could change the following settings.
If we want to use the subset of the testing dataset to get an approximation testing result, we set -p=sample_final_maml, or we could set -p=final_maml to infer all the testing questions.
If we want to use the models stored in the named folder MRL-CQA/data/saves/maml_reptile, we set --n=maml_reptile.
If we want to use our saved model epoch_020_0.784_0.741 in the named folder to test the questions, we set -m=epoch_020_0.784_0.741.dat.

After setting, we run the file MRL-CQA/S2SRL/data_test_maml.py to generate the action sequence for each testing question:

python data_test_maml.py

We could find the generated action sequences in the folder where the model is in (for instance MRL-CQA/data/saves/maml_reptile), which is stored in the file sample_final_maml_predict.actions or final_maml_predict.actions.

(4). Calculating the result.
After generating the actions, we could use them to compute the QA result.
For example, we use the saved model MRL-CQA/data/saves/maml_reptile/epoch_020_0.784_0.741.dat to predict actions for the sample testing questions, and therefore generate a file MRL-CQA/data/saves/maml_reptile/sample_final_maml_predict.actions to record the generated actions for the testing questions.
Then in the file MRL-CQA/S2SRL/SymbolicExecutor/calculate_sample_test_dataset.py, we set the parameters as follows.
In the function transMask2ActionMAML(), we have a line of the code:

with open(path, 'r') as load_f, open("../../data/saves/maml_reptile/sample_final_maml_predict.actions", 'r') as predict_actions:

, which is used to compute the accuracy of the actions stored in the file MRL-CQA/data/saves/maml_reptile/sample_final_maml_predict.actions.
We could change the path of the generated file in the above line of the code.
Then we run the file MRL-CQA/S2SRL/SymbolicExecutor/calculate_sample_test_dataset.py to get the final result:

python calculate_sample_test_dataset.py

We download the file CSQA_ANNOTATIONS_test.json from the data link and put it into the folder MRL-CQA/data/auto_QA_data/ of the project, which is used to record the ground-truth answers of each question.
The result will be stored in the file MRL-CQA/data/auto_QA_data/test_result/maml_reptile.txt.

References:
[1]. Amrita Saha, Vardaan Pahuja, Mitesh M Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In ThirtySecond AAAI Conference on Artificial Intelligence.

About

MRL-CQA for EMNLP 2020 submission. This work has been accepted by EMNLP 2020.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%