This repository was created for our bachelor thesis where we tried to evaluate QA pipelines. The pipelines consist of Qanary components. We have updated the components with active APIs and fixed some bugs. There are two new components (EARL Relation Linker and SINA Query Builder [1][2]) that we were allowed to make publicly available.
Clone this repository. Make sure you have Stardog installed, we used version 6 during our work. Versions older than 5 will not work. We have used two environment variables to keep our scripts flexible.
%STARDOG_HOME% is used for Stardog as its working directory, we use that variable in our scripts aswell. %BA_HOME% is the path to where you cloned this repository.
Build all the needed components and the pipeline with mvn install -DskipDockerBuild. This will create a target folder in the component's folder or pipeline's folder. We have not used Docker in our work and cannot say if the components will work via Docker instances. This issue from the Qanary repository may help implementing the components with Docker.
To run Stardog and the pipeline (no components yet), use StardogPipeline.bat, if you want to use the web console in Stardog to query the triplesore, use StardogWithWebConsolePipeline.bat. This might take a short while until everything is ready to use.
Then you can either use any of the scripts to start a pipeline (e.g. 9_Tagme-NED_Earl_CLS-CLISNLIOD_SINA.bat ) or you can start the components from their target folder which was created during the build process. Make sure you start from the target folder, as there are relative paths in the components.
When the components have registered themselves to the pipeline, go to localhost:8080/#/overview to see a Spring overview of the pipeline. Here you can see logs and whether the component is online or offline. Go to localhost:8080/startquestionansweringwithtextquestion to test the pipeline with a question of your choice. Select and drag the components so you have a useful pipeline.
A big part of our work was evaluating the different pipelines with QALD-9 [3]. To evaluate and compare the pipelines, we have developed an evaluator.
-
[1] Kuldeep Singh, Arun Sethupat Radhakrishna, Andreas Both, Saeedeh Shekarpour, Ioanna Lytra, Ricardo Usbeck, Akhilesh Vyas, Akmal Khikmatullaev, Dharmen Punjani, Christoph Lange, Maria-Esther Vidal, Jens Lehmann, Sören Auer: Why Reinvent the Wheel: Let's Build Question Answering Systems Together. WWW 2018: 1247-1256
-
[2] Kuldeep Singh, Andreas Both, Arun Sethupat Radhakrishna, Saeedeh Shekarpour: Frankenstein: A Platform Enabling Reuse of Question Answering Components. ESWC 2018: 624-638
-
[3] R. Usbeck, R. H. Gusmita, M. Saleem, A.-C. Ngonga Ngomo: 9th Challenge on Question Answering over Linked Data (QALD-9), 2018.