This evaluator was created to evaluate Question Answering Systems with the JSON Qald datasets. Since it was part of a bachelorsthesis that included Qanary, the evaluator works best with those Pipelines.
First install all the Node_Modules.
npm install
Then run the application.
npm install
The app should now be available on: http://localhost:3000
If you want to evaluate your own system make sure that it returns the correct Format. The answers value follows the W3C defined standard.
{
"questions": [{
"id": "1",
"question": [{
"language": "en",
"string": ""
}],
"query": {
"sparql": ""
},
"answers": [{
"head": {
"vars": [
"uri"
]
},
"results": {
"bindings": [{
"uri": {
"type": "uri",
"value": ""
}
}]
}
}]
}]
}If you want to evaluate your own system make sure that it returns the correct Format. The answers value follows the W3C defined standard. If the answerset wasn't created with this evaluator, then you want have the "qanaryAnno" property.
[
{
"id": "99",
"data": {
"questions": [
{
"question": [{ "language": "en" }],
"query": "select * where { <http://dbpedia.org/resource/Time_zone> <http://dbpedia.org/ontology/timeZone> <http://dbpedia.org/resource/Salt_Lake_City>. }, ",
"answers": [
{
"head": { "vars": ["_star_fake"] },
"results": { "bindings": [] }
}
],
"qanaryAnno": {
"entities": "http://dbpedia.org/resource/Time_zone, http://dbpedia.org/resource/Salt_Lake_City, ",
"classes": "",
"properties": "http://dbpedia.org/ontology/timeZone, "
}
}
]
}
},...]To ensure the functionnality, 87 unit and integration tests have been developed and can be found in the test/ directory.
The interface was only tested with Google Chrome.
See also the list of contributors who participated in this project.