Skip to content

Home

Jump to bottom
Ria Hari Gusmita edited this page Oct 25, 2018 · 3 revisions

What is QUANT?

QUANT, Question Answering Benchmark Curator, a tool that supports developers and researchers in creating and curating QA benchmarks so that these benchmarks can be run against any given endpoint (version). QUANT generates smart suggestions for each question-SPARQL-answer tuple as well as QA metadata to increase curation productivity.

How to use QUANT?

  1. Access QUANT at this following address:
  2. Upload your dataset (that needs to be curated) through menu Curate my Dataset
  3. QUANT will display a list of questions fro dataset. Go to the details page of a question that will be curated. There will be three options available i.e. doing correction, give a confirmation that the question does not need any changes, and remove the question. The following are explanations for all options:
  • Doing correction Change value of any question’s attribute so that the value becomes correct regarding certain condition. This task can be done by clicking Start Correction button.
  • Give a confirmation that the question does not need any changes State that all attributes of the question already have correct values so it does not need to do correction or remove the question. This task can be done by clicking No Changes Needed button.
  • Remove the question: delete the question from the dataset as its SPARQL is not able to return results from current SPARQL endpoint and there is no way to modify the SPARQL to let it able to return results. This task can be done by clicking Remove Question button.
  1. In doing correction, check value of these following attributes: SPARQL, Answer Type, Out of Scope, Aggregation, Onlydbo, Hybrid, Multilingual Keyword, Multilingual Question and provide the correct value for them if their existing values are incorrect. QUANT will provide a suggestion for an attribute that either does not have value or the existing value is incorrect. SPARQL value is used to determine value of others attributes. Therefore, start to do checking on SPARQL value base on value of Answer from Current Endpoint. After finished the correction, click Done button so that the correction will be recorded. During correction process, annotator can decide whether the question should be removed (Remove Question button) or cancel the correction (Cancel button).
  2. If there is no value at Answer from Current Endpoint field, check whether QUANT provides View SPARQL suggestion button (below SPARQL text field). If it is the case, check the suggestion and take it if it is decided as correct. Otherwise, update the SPARQL value by yourself by doing correction directly in SPARQL text field. If there is a changing of SPARQL value, QUANT will update value at Answer from File and Answer from Current Endpoint text field and will adjust suggestion value for other attributes.
  3. Curation result and log of activities can be seen on User Activities menu. This page also completed with a menu to get curation results in JSON format.

How to Curate a Question

  1. These are rules to get correct value for each attribute if SPARQL is already correct in the sense it returns results from current endpoint.
  • Answer Type, is used to describe data type of Answer from Current Endpoint’s value. There are 5 possibilities data type of it i.e. Boolean, Date, Number, Resource, and String. Here are sample value of each data type:
  • Aggregation, describes whether SPARQL attribute’s value contains one or more aggregation function such as COUNT, SUM, AVG, MIN, MAX, SAMPLE, GROUP_CONCAT, GROUPCONCAT, VECTOR_AGG, and COUNT DISTINCT. If it is the case, aggregation value will be true. Otherwise, it will be false.
  • Hybrid, describes whether it is required to do searching both Linked Data and textual data for an answer. A hybrid question will have a SPARQL that contains phrase text:query in it. If it is the case, its value will be true. Otherwise, it will be false.
  • Onlydbo, describes whether SPARQL attribute’s value does not contain some URI outside of DBpedia, for instance http://yago-knowledge.org/resource/yagoLegalActor. If it is the case, Onlydbo’s value will be set as true. Otherwise, it will be false.
  • Out of Scope, describes whether SPARQL query is not able to retrieve results from current SPARQL endpoint or the returned results are semantically incorrect. If there is a result (some result) from current endpoint and the result is semantically correct, Out of Scope value must be false. It will be true if there is no result exists or the result is not correct semantically.
  • Multilingual Keywords, This attribute should show keywords of the question in 11 different languages. English is the main language.
  • Multilingual Question, Like the keywords, question also should have translations in 10 languages.

Clone this wiki locally