Home
… co-located with: Extended Semantic Web Conference 2015
… sponsored by:
The Challenge is open! Please subscribe to the mailing list to be kept up to date.
The Concept-Level Sentiment Analysis Challenge is open to everyone from industry and academia working within the sentiment analysis area.
As the Web rapidly evolves, Web users are evolving with it. In an era of social connectedness, people are becoming increasingly enthusiastic about interacting, sharing, and collaborating through social networks, online communities, blogs, Wikis, and other online collaborative media. In recent years, this collective intelligence has spread to many different areas, with particular focus on fields related to everyday life such as commerce, tourism, education, and health, causing the size of the Social Web to expand exponentially.
The opportunity to automatically capture the opinions of the general public about social events, political movements, company strategies, marketing campaigns, and product preferences has raised growing interest within the scientific community, leading to many exciting open challenges, as well as in the business world, due to the remarkable benefits deriving from marketing prediction. The distillation of knowledge from such a large amount of unstructured information is an extremely difficult task, as the contents of today’s Web are perfectly suitable for human consumption, but remain hardly accessible to machines. Mining opinions and sentiments from natural language, involves a deep understanding of most of the explicit and implicit, regular and irregular, syntactical and semantic rules proper of a language. Existing approaches mainly rely on parts of text in which opinions and sentiments are explicitly expressed such as polarity terms, affect words and their co-occurrence frequencies. However, opinions and sentiments are often conveyed implicitly through latent semantics, which make purely syntactical approaches ineffective. This issue offers a research opportunity and an exciting challenge to the Semantic Web community. In fact, concept-level sentiment analysis aims to go beyond a mere word-level analysis of text and provide novel approaches to opinion mining and sentiment analysis that allow a more efficient passage from (unstructured) textual information to (structured) machine-processable data, in potentially any domain.
Concept-level sentiment analysis focuses on a semantic analysis of text through the use of web ontologies, semantic resources, or semantic networks, allowing the identification of opinion data which with only natural language techniques would be very difficult. By relying on large semantic knowledge bases, concept-level sentiment analysis steps away from blind use of keywords and word co-occurrence count, but rather relies on the implicit features associated with natural language concepts. Unlike purely syntactical techniques, concept-based approaches are able to detect also sentiments that are expressed in a subtle manner, e.g., through the analysis of concepts that do not explicitly convey any emotion, but which are implicitly linked to other concepts that do so.
A call for contributions will be prepared according to ESWC challenge track guidelines. It will require participant to submit their system as well as a paper describing their approach, which will undergo a peer-review process. Submissions must include:
-
Abstract: no more than 200 words.
-
Paper: containing the details of the system, including why the system is innovative, which features or functions the system provides, what design choices were made and what lessons were learned, and how the semantics has been employed.
-
Web Access: applications can either be accessible via the web or downloadable. If an application is not publicly accessible, password must be provided for reviewers. A short set of instructions on how to use the application should be provided as well.
Papers are submitted in PDF format via the challenge's EasyChair submission pages.
- May 7, 2015, 23:59 (Hawaii time): Challenge Paper Submission deadline
-
March 27, 2015, 23:59 (Hawaii time): Challenge Paper Submission deadline - April 9, 2015, 23:59 (Hawaii time): Notification
- April 24, 2015, 23:59 (Hawaii time): Camera-ready papers deadline
- May 25, 2015, 23:59 (Hawaii time): Test data set published
- May 27, 2015, 23:59 (Hawaii time): Communicate to the Challenge Track Chairs the final results of the evaluation
- May 31 - June 4, 2015: The Challenge takes place at ESWC-15
This challenge focuses on the introduction, presentation, and discussion of novel approaches to concept-level sentiment analysis. Participants will have to design a concept-level opinion-mining engine that exploits Semantic Web knowledge bases, e.g., ontologies, DBpedia, etc., to perform multi-domain sentiment analysis. The main motivation for this challenge is to go beyond a mere word-level analysis of natural language text and provide novel concept-level tools and techniques that allow a more efficient passage from (unstructured) natural language to (structured) machine-processable data in potentially any domain. The submitted systems must provide an output according to Semantic Web standards (RDF, OWL, etc.). Systems must have a semantic flavor (e.g., by making use of Linked Data or known semantic networks within their core functionalities) and authors need to show how the introduction of semantics improves the precision of their methods. Existing natural language processing methods or statistical approaches can be used too as long as the semantics plays a role within the core approach and improves the precision (engines based merely on syntax/word-count will be excluded from the competition). The target language is English and multi-language capabilities are a plus.
The Concept-Level Sentiment Analysis Challenge is defined in terms of different tasks. The first task is elementary whereas the others are more advanced. In order to be accepted for the challenge, each system has to deal with the first task.
The basic task of the challenge is binary polarity detection. The proposed semantic opinion-mining engines will be assessed according to precision, recall and F-measure of detected polarity values (positive OR negative) for each review of the evaluation dataset. The problem of subjectivity detection is not addressed within this Challenge, hence participants can assume that there will be no neutral reviews.
The output of this Task will be a set of aspects of the reviewed product and a binary polarity value associated to each of such aspects. So, for example, while for the Elementary Task an overall polarity (positive or negative) is expected for a review about a mobile phone, this Task requires a set of aspects (such as ‘speaker’, ‘touchscreen’, ‘camera’, etc.) and a polarity value (positive OR negative) associated with each of such aspects. Engines will be assessed according to both aspect extraction and aspect polarity detection using precision, recall and F-measure similarly as performed during the first Concept-Level Sentiment Analysis Challenge held during ESWC2014 and re-proposed at SemEval 2015 Task12.
The Challenge focuses on sentiment analysis at concept-level. This means that the proposed engines must work beyond word/syntax level, hence addressing a concepts/semantics perspective. This task will evaluate the capability of the proposed systems to identify the objects involved in a typical opinion frame according to their role: holders, topics, opinion concepts (i.e. terms referring to highly polarised concepts). For example, in a sentence such as The mayor is loved by the people in the city, but he has been criticized by the state government (taken from Sentiment Analysis and Opinion Mining, Bing Liu, 2012), an approach should be able to identify that the people and state government are the opinion holders, is loved and has been criticized represent the opinion concepts, The mayor identifies a topic of the opinion. The proposed engines will be evaluated according to precision, recall and F-measure.
A subjective evaluation will be performed by three members of the Advisory Board. For each system, reviewers will give a numerical score within the range [1-10] and details motivating their choice. The scores will be given to the following aspects:
-
Use of common-sense knowledge and semantics;
-
Computational time;
-
Graphical interface - including the number of features that is possible to query, usability of the system, appealing of the user interface;
-
Innovative nature of the approach including multi-language capabilities.
For systems that can be tuned with different parameters, please indicate a range of up to 5 sets of settings. Settings with the best precision, recall, F-measure will be considered for judgment. The objective evaluation will be performed according to precision, recall, and F-measure analysis.
Systems will be evaluated against a testing dataset that will be annotated using a crowd sourcing tool. Participants are recommended to train and/or test their own systems using the Blitzer Dataset. Precision, recall, F1-measure for all the tasks will be computed automatically by a tool that will be available for download so that each participant will be able assess their methods and make sure the output produced by their system is in compliance with the input required by the script.
###Input and output format Below, the input and output formats for the different tasks are presented. Both the input and the output syntaxes (presented in the next section) have been inspired to the ones used in the SemEval 2015 Workshop.
<sentence id='17'>
<text>
The long build-up given to their anticipated shoot out with Victor
Frankenstein is a dead end.
</text>
<polarity>0</polarity>
</sentence>
<sentence id='18'>
<text>
Koontz finds his footing in the final chapters, a Lovecraftian
showdown between Frankenstein and his artificial creations inside a
series of tunnels beneath a dump.
</text>
<polarity>1</polarity>
</sentence>In the training set every attributes will be filled, while in the test set, the “polarity” attribute will be set to “null” and it must be filled by the participant.
<Review rid="1199480">
<sentences>
<sentence id="1199480:0">
<text>Not sure where the previous reviewer, lonk, dined, but
Saul is in a great neighborhood and has great food!</text>
<Opinions>
<Opinion target="neighborhood" category="LOCATION#GENERAL"
polarity="positive" from="74" to="86"/>
<Opinion target="food" category="FOOD#QUALITY"
polarity="positive" from="101" to="105"/>
</Opinions>
</sentence>
<sentence id="1199480:1">
<text>I've been there three times and have always had
wonderful experiences.</text>
<Opinions>
<Opinion target="NULL" category="RESTAURANT#GENERAL"
polarity="positive" from="0" to="0"/>
</Opinions>
</sentence>
<sentence id="1199480:2">
<text>I'd highly recommend it for a special occasion -- it provides
and intimate setting and nice service.</text>
<Opinions>
<Opinion target="setting" category="AMBIENCE#GENERAL"
polarity="positive" from="75" to="82"/>
<Opinion target="service" category="SERVICE#GENERAL"
polarity="positive" from="92" to="99"/>
<Opinion target="NULL" category="RESTAURANT#MISCELLANEOUS"
polarity="positive" from="0" to="0"/>
</Opinions>
</sentence>
</sentences>
</Review>In the training set every attributes will be filled, while in the test set, the “Opinions” tag will be empty and it must be completed by the participant.
<sentence id='1'>
<text>
Robert thinks that Alex is a good and smart guy and Anna is a bad player.
</text>
<frame>
<holder value="Robert"/>
<topic value="Alex"/>
<opinion value="good">
<polarity>positive</polarity>
</opinion>
</frame>
<frame>
<holder value="Robert"/>
<topic value="Alex"/>
<opinion value="smart">
<polarity>positive</polarity>
</opinion>
</frame>
<frame>
<holder value="Robert"/>
<topic value="Anna"/>
<opinion value="bad">
<polarity>negative</polarity>
</opinion>
</frame>
</sentence>In the training set every attributes will be filled, while in the test set, the “Frame” tag will be empty and it must be completed by the participant. The output format is basically equivalent to the training set with the difference that the value for polarity, aspects and the associated polarity, holder, topics and oew will be automatically filled by the system.
###Judging and Prizes We propose four awards, one based on the subjective criteria and the others based on the objective criteria:
-
Subjective: the system with the highest average score in items 1-4 above;
-
Objective: the system with the highest score in precision, recall and F-measure analysis for task 1.
-
Objective: the system with the highest score in precision, recall and F-measure analysis for task 2.
-
Objective: the system with the highest score in precision, recall and F-measure analysis for task 3.
After the first round of evaluation a list of runners up will be defined. A certain number of systems with the highest subjective evaluation and objective evaluation will be the finalists and will have to present their work in a conference session. The exact number will depend on the scores they get and on the Conference policy. They will have a slot of approximately 15 minutes. The judges will be present and will evaluate again the systems in more detail. The judges will then meet in private to discuss the entries and to determine the final standings, one for subjective and the other for objective scores.
###How to Participate The following information has to be provided:
- Abstract: no more than 200 words.
- Description: It should contain the details of the system, including why the system is innovative, how it uses Semantic Web, which features or functions the system provides, what design choices were made and what lessons were learned. The description should also summarize how participants have addressed the evaluation tasks. Papers must be submitted in PDF format, following the style of the Springer's Lecture Notes in Computer Science (LNCS) series, and not exceeding 12 pages in length.
- Web Access: The application can either be accessible via the web or downloadable. If the application is not publicly accessible, password must be provided. A short set of instructions on how to use the application should be provided as well.
###Organizers
-
Mauro Dragoni, FBK, Italy (dragoni@fbk.eu) Mauro Dragoni is a post-doc researcher at the Fondazione Bruno Kessler in Trento since 2011. He received his Ph.D. degree in Computer Science from the Università degli Studi di Milano in 2010 and his major research interests concern the Computational Intelligence and Knowledge Management fields applied to the Information Retrieval, Ontology Matching, and Sentiment Analysis topics. In particular, he focuses on applying state of the art research paradigms to the implementation of real-world knowledge management systems. In the first edition of the Conceptual-Level Sentiment Analysis Challenge held at ESWC2014, his system won the Aspect-Based Sentiment Analysis Task and the Most Innovative System awards.
-
Valentina Presutti, CNR, Italy (valentina.presutti@cnr.it). Valentina Presutti is a researcher at ISTC-CNR. She is an expert in semantic sentiment analysis and more in general in knowledge extraction and engineering for the Semantic Web. She published more than 60 articles about this topic in international journals and peer-reviewed conferences. She has extensive experience in organising scientific events, for example she served as Program Chair for Semantics 2012, ESWC 2013 and as General Chair for ESWC 2014, and as organisers of several workshops.
-
Diego Reforgiato Recupero, CNR, Italy (diego.reforgiato@istc.cnr.it) Diego Reforgiato Recupero is a programmer, software developer, automation and ICT expert. In 2005 he was awarded a 3 year Post Doc fellowship with the University of Maryland where he won the Computer World Horizon Award 2006 in the USA for the best research project on OASYS, an opinion analysis system that was commercialized by SentiMetrix. In 2012 he got to the winning podium of the Startup Weekend event held in Catania with a pitch and demo related to Opinion, an opinion analysis platform. In 2014, at ESWC2014 he co-chaired the Semantic Sentiment Analysis Workshop and the Conceptual-Level Sentiment Analysis Challenge.
- Davide Buscaldi, Universit ́e Paris 13 (France)
- Aldo Gangemi, Universit ́e Paris 13 (France)
- Newton Howard, MIT Media Laboratory (USA)
- Cheng Xiang Zhai, University of Illinois at Urbana-Champaign (USA)
Challenge committee will be finalized after the submission of the systems to ensure that no conflict of interests between participants and committee will exist.
- Rada Mihalcea, University of North Texas (USA)
- Ping Chen, University of Houston-Downtown (USA)
- Yongzheng Zhang, LinkedIn Inc. (USA)
- Giuseppe Di Fabbrizio, Amazon Inc. (USA)
- Soujanya Poria, Nanyang Technological University (Singapore)
- Yunqing Xia, Tsinghua University (China)
- Rui Xia, Nanjing University of Science and Technology (China)
- Jane Hsu, National Taiwan University (Taiwan)
- Rafal Rzepka, Hokkaido University (Japan)
- Amir Hussain, University of Stirling (UK)
- Alexander Gelbukh, National Polytechnic Institute (Mexico)
- Bjoern Schuller, Technical University of Munich (Germany)
- Amitava Das, Samsung Research India (India)
- Dipankar Das, National Institute of Technology (India)
- Carlo Strapparava, Fondazione Bruno Kessler (Italy)
- Stefano Squartini, Marche Polytechnic University (Italy)
- Cristina Bosco, University of Torino (Italy)
- Paolo Rosso, Technical University of Valencia (Spain)
To ask questions and information please join our mailing list at http://groupspaces.com/ESWC2015-CLSA. After you join the group, you can post messages to the mailing list address ESWC2015-CLSA@groupspaces.com