This is the repository for the paper: Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering - EACL 2023 (Findings).

Dataset Information

We use two datasets in our experiments: 2WikiMultihopQA and HotpotQA-small

• Pre-process data (file .gz) for dev and train of 2Wiki (Please download raw data from the Github repository of the 2WikiMultihopQA dataset)
• Raw data and pre-process data for dev and train of HotpotQA-small
• Debiased data
• Adversarial data

We follow the steps in https://github.com/yuwfan/HGN to obtain file .gz data from raw data.

How to Run the Code

Set up environment

bash install_packages.sh

Prepare data for training

• Download bigbird-roberta-base model from this link: https://huggingface.co/google/bigbird-roberta-base
• Edit variables: data_dir, pretrained_model_dir, data_file
• Run: python3 preprocess.py

Training

python3 main.py

For evaluation on dev file

python3 predictor.py $checkpoint $data_file

python3 postprocess.py $prediction_file $processed_data_file $original_data_file

python3 official_evaluation.py path/to/prediction path/to/gold

Reproduce the results

• Download our checkpoints
• Run file predict_dev_all_settings.sh (Note: if you want to use this file for the test set in 2Wiki, comment line #25 about evaluation)

References

• We base on HGN for data preprocessing.
• We re-use the class Example from the HGN model and update it to work with our dataset.