Data and Code for paper Outlining and Filling: Hierarchical Query Graph Generation for Answering Complex Questions over Knowledge Graphs accepted by IEEE Transactions on Knowledge and Data Engineering, is available for research purposes.
We apply three KGQA benchmarks to evaluate our approach, ComplexWebQuestions (Talmor and Berant, 2018), LC-QuAD (Trivedi et al., 2017), and WebQSP (Yih et al., 2016).
| Dataset | Structure Acc. | Query Graph Acc. | Precision | Recall | F1-score | Hit@1 |
|---|---|---|---|---|---|---|
| ComplexWebQuestions | 66.96 | 51.68 | 65.27 | 68.44 | 64.95 | 65.25 |
| ComplexWebQuestions (Bert-base) | 72.88 | 57.80 | 68.89 | 73.30 | 68.88 | 68.80 |
| LC-QuAD | 78.00 | 60.90 | 75.82 | 75.22 | 75.10 | 76.00 |
| LC-QuAD (Bert-base) | 80.70 | 63.50 | 78.92 | 78.14 | 78.13 | 78.70 |
| WebQSP | 79.91 | 62.63 | 70.22 | 74.38 | 70.61 | 70.37 |
| WebQSP (Bert-base) | 85.03 | 70.74 | 76.66 | 79.28 | 76.62 | 76.92 |
-
Download and unzip our preprocessed data to
./, you can also running our scripts under./preprocessto obtain them again. -
Download GloVe Embedding glove.42B.300d.txt and put it to
your_glove_path. -
Download our vocabulary from here. Unzip and put it under
./. It contains our used SPARQL cache for Execution-Guided strategy.
The two KGs we use contain only triples in English by removing other languages. Please download the KGs virtuoso-opensource we use for the SPARQL query service for the downloaded Freebase and DBpedia. Here is a tutorial on how to install Virtuoso and import the knowledge graph into it.
- Get root access
- Edit
virtuoso-opensource/database/virtuoso.iniand set the property "DirsAllowed" to your path. - Execute the following commands to start the service.
cd virtuoso-opensource/database/
../bin/virtuoso-t -fdBefore training, first set the following hyperparameter in main_train_cwq.sh, main_train_lcq.sh, and main_train_wsp.sh.
--glove_path your_glove_pathExecute the following command for training model on ComplexWebQuestions.
sh train_cwq.shExecute the following command for training model on LC-QuAD.
sh train_lcq.shExecute the following command for training model on WebQSP.
sh train_wsp.shThe trained model file is saved under ./runs directory.
The path format of the trained model is ./runs/RUN_ID/checkpoints/best_snapshot_epoch_xx_best_val_acc_xx_model.pt.
Execute the following command for training model on ComplexWebQuestions.
sh train_plm_cwq.shExecute the following command for training model on LC-QuAD.
sh train_plm_lcq.shExecute the following command for training model on WebQSP.
sh train_plm_wsp.shThe trained model file is saved under ./runs directory.
The path format of the trained model is ./runs/RUN_ID/checkpoints/best_snapshot_epoch_xx_best_val_acc_xx_model.pt.
Before testing, need to train a model first and set the following hyperparameters in main_eval_cwq.sh, main_eval_lcq.sh, and main_eval_wsp.sh.
--cpt your_trained_model_path
--kb_endpoint your_sparql_service_ipYou can also directly download our trained models from here. Unzip and put it under ./.
Execute the following command for testing the model on ComplexWebQuestions.
sh eval_cwq.shExecute the following command for testing the model on LC-QuAD.
sh eval_lcq.shExecute the following command for testing the model on WebQSP.
sh eval_wsp.shBefore testing, need to train a model first and set the following hyperparameters in main_eval_plm_cwq.sh, main_eval_plm_lcq.sh, and main_eval_plm_wsp.sh.
--cpt your_trained_model_path
--kb_endpoint your_sparql_service_ipYou can also directly download our trained models from here. Unzip and put it under ./.
Execute the following command for testing the model on ComplexWebQuestions.
sh eval_plm_cwq.shExecute the following command for testing the model on LC-QuAD.
sh eval_plm_lcq.shExecute the following command for testing the model on WebQSP.
sh eval_plm_wsp.sh