-
Notifications
You must be signed in to change notification settings - Fork 2
SemPub16_GeneralRules
This page describes the overall organization and evaluation process of the Semantic Publishing Challenge tasks. General information about the Challenge is available in SemPub2016.
Participants can participate in any number of tasks, as the tasks are independent.
The Challenge includes three tasks that share the same organization, though each of them uses its own dataset and information.
Details of each task and dataset are available at SemPub16_Task1, SemPub16_Task2 and SemPub16_Task3.
Each dataset is split in two parts:
- a public part, for the participants' training (TD),
- a private part, for the evaluation (ED), which is a superset of TD and will be disclosed a few days before the submission deadline.
The evaluation of the best-performing approach will consist of evaluating a set of queries against the produced dataset to assess its correctness and completeness.
We provide natural language queries and require participants to translate them to a SPARQL form that can be run against the produced LOD.
Participants can introduce their own vocabularies or reuse well-known ones but are required to provide SPARQL queries whose output is compliant with the rules defined for each task.
The queries' expected result format and the compliance rules are described in SemPub16_QueriesTask1, SemPub16_QueriesTask2 and SemPub16_QueriesTask3.
The produced LOD must be available under an open license (at least as permissive as that of the source data) so that publishers of source data and the whole research community can reuse it. We strongly encourage participants to release their extraction tool under an open license too, though this is not mandatory.
Participants will also be asked to submit an abstract, a paper and their tool (source and/or binaries, or a link these can be downloaded from, or a web service URL) for verification purposes.
Submission instructions will be available in this wiki.
More details will also be provided about the (in-progress and final) evaluation.
The Challenge will have six winners. For each task we will select:
- best performing tool, given to the paper which will get the highest score in the evaluation
- most original approach, selected by the Challenge Committee with the reviewing process
The selection of the best-performing approach will be done mostly automatically (with exceptions defined below) by measuring precision and recall.
Input queries have the following form:
- Provide all papers written by X, Y or Z
- Provide authors of the paper T
- …
We will run these queries on ED after substituting variables with instance values. We will compare the output against the expected results (that we have prepared in advance, in the expected format) and measure precision and recall of each query.
Each query will be given the same weight. Each query will be repeated with several different instance values.
We will also inspect the participants' queries and reserve the right to discard some of them if their structure is not correct and compliant to the Challenge rules.
We reserve the right to inspect the LOD provided by the authors and to devalue a submission if its LOD dataset helps to answer the given queries but contains serious mistakes, such as claiming that a paper had an author that it did not actually have, or omitting one of its authors.
Finally, we reserve the right to run the extraction tool ourselves, and to inspect its source code. The evaluation will be initially performed on the LOD provided by the authors.
After a first round of review, the Program Committee and the chairs will select a number of submissions that will be invited to demo their work at ESWC.
For each task, one winner for the most original approach will be selected.