Home
… co-located with: Extended Semantic Web Conference 2017
… sponsored by:
The Challenge is open! Please subscribe to the mailing list to be kept up to date.
The ESWC-17 Challenge on Semantic Sentiment Analysis is open to everyone from industry and academia working within the sentiment analysis area.
The development of Web 2.0 has given users important tools and opportunities to create, participate and populate blogs, review sites, web forums, social networks and online discussions. Tracking emotions and opinions on certain subjects allows identifying users' expectations, feelings, needs, reactions against particular events, political view towards certain ideas, etc. Therefore, mining, extracting and understanding opinion data from text that reside in online discussions is currently a hot topic for the research community and a key asset for industry.
The produced discussion spanned a wide range of domains and different areas such as commerce, tourism, education, health, etc. Moreover, this comes back and feeds the Web 2.0 itself thus bringing to an exponential expansion.
This explosion of activities and data brought to several opportunities that can be exploited in both research and industrial world. One of them concerns the mining and detection of users' opinions which started back in 2003 (with the classical problem of polarity detection) and several variations have been proposed. Therefore, today there are still open challenges that have raised interest within the scientific community where new hybrid approaches are being proposed that, making use of new lexical resources, natural language processing techniques and semantic web best practices, bring substantial benefits.
Computer World\footnote{Computer World, 25 October 2004, Vol. 38, NO 43.} estimates that 70%-80% of all digital data consists of unstructured content, much of which is locked away across a variety of different data stores, locations and formats. Besides, accurately analyzing the text in an understandable manner is still far from being solved as this is extremely difficult. In fact, mining, detecting and assessing opinions and sentiments from natural language involves a deep (lexical, syntactic, semantic) understanding of most of the explicit and implicit, regular and irregular rules proper of a language.
Existing approaches are mainly focused on the identification of parts of the text where opinions and sentiments can be explicitly expressed such as polarity terms, expressions, statements that express emotions. They usually adopt purely syntactical approaches and are heavily dependent on the source language and the domain of the input text. It follows that they miss many language patterns where opinions can be expressed because this would involve a deep analysis of the semantics of a sentence. Today, several tools exist that can help understanding the semantics of a sentence. This offers an exciting research opportunity and challenge to the Semantic Web community as well. For example, sentic computing is a multi-disciplinary approach to natural language processing and understanding at the crossroads between affective computing, information extraction, and common-sense reasoning, which exploits both computer and human sciences to better interpret and process social information on the Web.
Therefore, the Semantic Sentiment Analysis Challenge looks for systems that can transform unstructured textual information to structured machine processable data in any domain by using recent advances in natural language processing, sentiment analysis and semantic web.
By relying on large semantic knowledge bases, Semantic Web best practices and techniques, and new lexical resources, semantic sentiment analysis steps away from blind use of keywords, simple statistical analysis based on syntactical rules, but rather relies on the implicit, semantics features associated with natural language concepts. Unlike purely syntactical techniques, semantic sentiment analysis approaches are able to detect sentiments that are implicitly expressed within the text, topics referred by those sentiments and are able to obtain higher performances than pure statistical methods.
Two steps submission
First step:
- Abstract: no more than 200 words.
- Paper (max 4 pages): Paper: containing the details of the system, including why the system is innovative, which features or functions the system provides, what design choices were made and what lessons were learned, how the semantics has been employed and which tasks the system addresses. The paper should be up to 8 pages length. Industrial tools with non disclosure restrictions are also allowed to participate, and in this case they are asked to:
- explain even at a higher level their approach and engine macro-components, why it is innovative, and how the semantics is involved};
- provide free access (even limited) for research purposes to their engine, especially to make repeatable the challenge results or other experiments possibly included in their paper
Second step (for accepted systems only):
- Paper (max 15 pages): full description of the submitted system.
- Web Access: applications should be either accessible via web or downloadable or anyway a RESTful API must be provided to run the challenge testset. If an application is not publicly accessible, password must be provided for reviewers. A short set of instructions on how to use the application or the RESTFul API must be provided as well.
- If accepted, the authors will have the possibility to present a poster and a demo advertising their work or networking during a dedicated session.
Please note that:
- Papers must comply with the LNCS style
- Papers are submitted in PDF format via the EasyChair submission pages (remember to check the topic Challenge).
- Accepted papers will be published by Springer.
- Extended versions of best systems will be invited to journal special issues.
- All the participants are invited to submit a paper containing the research aspects of their systems to the ESWC 2017 Workshop on Emotions, Modality, Sentiment Analysis and the Semantic Web (http://www.maurodragoni.com/research/opinionmining/events/)
- Friday March 18th, 2016, 23:59 (CET): First step submission
- Friday March 25th, 2016, 23:59 (CET): Notification of acceptance
- Friday April 29th, 2016, 23:59 (CET): Second step submission
- Friday May 20th, 2016, 23:59 (CET): Test data published
- Sunday May 29 - June 2, 2016: The Challenge takes place at ESWC-16
This challenge focuses on the introduction, presentation, development and discussion of novel approaches to semantic sentiment analysis. Participants will have to design a semantic opinion-mining engine that exploits Semantic Web knowledge bases, e.g., ontologies, DBpedia, etc., to perform multi-domain sentiment analysis. The main motivation for this challenge is to go beyond a mere word-level analysis of natural language text and provide novel semantic tools and techniques that allow a more efficient passage from (unstructured) natural language to (structured) machine-processable data in potentially any domain.
The submitted systems must provide an output according to Semantic Web standards (RDF, OWL, etc.). Systems must have a semantic flavour (e.g., by making use of Linked Data or known semantic networks within their core functionalities) and authors need to show how the introduction of semantics improves the performance of their methods. Existing natural language processing methods or statistical approaches can be used too as long as the semantics plays a role within the core approach and improves the precision (engines based merely on syntax/word-count will be excluded from the competition). The target language is English and multi-language capability is a plus.
The Fine-Grained Sentiment Analysis Challenge is defined in terms of different tasks. The first task is elementary whereas the others are more advanced.
The basic task of the challenge is binary polarity detection. The proposed semantic opinion-mining engines will be assessed according to precision, recall and F-measure of detected polarity values (positive OR negative) for each review of the evaluation dataset. Participants can assume that there will be no neutral reviews. The output format for such a task is the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Sentences>
<sentence id="apparel_0">
<text>
GOOD LOOKING KICKS IF YOUR KICKIN IT OLD SCHOOL LIKE ME. AND COMFORTABLE.
AND RELATIVELY CHEAP. I’LL ALWAYS KEEP A PAIR OF STAN SMITH’S
AROUND FOR WEEKENDS
</text>
<polarity>
positive
</polarity>
</sentence>
<sentence id="apparel_1">
<text>
These sunglasses are all right. They were a little crooked, but still cool..
</text>
<polarity>
positive
</polarity>
</sentence>
Input is the same without the polarity tag. Dataset will be composed by one million OF reviews collected from the Amazon web site and split in 20 different categories: Amazon Instant Video, Automotive, Baby, Beauty, Books, Clothing Accessories, Electronics, Health, Home Kitchen, Movies TV, Music, Office Products, Patio, Pet Supplies, Shoes, Software, Sports Outdoors, Tools Home Improvement, Toys Games, and Video Games. The classification of each review (positive or negative) has been done according to the guidelines used for the construction of the Blitzer dataset [1]. Participants will evaluate their system by applying a cross-fold validation over the dataset where each fold is clearly delimited. The script to compute Precision, Recall, and F-Measure will be provided to participants through the website of the challenge.
[1] Blitzer J., Dredze M., Pereira F.. Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification. Association of Computational Linguistics (ACL), 2007.
The output of this Task will be a set of aspects of the reviewed product and a binary polarity value associated to each of such aspects. So, for example, while for the Task #1 an overall polarity (positive or negative) is expected for a review about a mobile phone, this Task requires a set of aspects (such as ‘speaker’, ‘touchscreen’, ‘camera’, etc.) and a polarity value (positive OR negative) associated with each of such aspects. Engines will be assessed according to both aspect extraction and aspect polarity detection using precision, recall and F-measure similarly as performed during the first Concept-Level Sentiment Analysis Challenge held during ESWC2014 and re-proposed at SemEval 2015 Task12. The output format for such a task is the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Review rid="1">
<sentences>
<sentence id="348:0">
<text>Most everything is fine with this machine: speed, capacity, build.</text>
<Opinions>
<Opinion aspect="MACHINE" polarity="positive"/>
</Opinions>
</sentence>
<sentence id="348:1">
<text>The only thing I don’t understand is that the resolution of the
screen isn’t high enough for some pages, such as Yahoo!Mail.
</text>
<Opinions>
<Opinion aspect="SCREEN" polarity="negative"/>
</Opinions>
</sentence>
<sentence id="277:2">
<text>The screen takes some getting use to, because it is smaller
than the laptop.</text>
<Opinions>
<Opinion aspect="SCREEN" polarity="negative"/>
</Opinions>
</sentence>
</sentences>
</Review>Input is the same without the Opinions tag and its descendants nodes. As training set, we will use the dataset provided by the last two editions of SemEval; as test set we will extract around 100 sentences from the web where we will annotate aspects and their related polarity. Two experts will annotate the sentences and disagreements will be analyzed. Precision, Recall and F-Measure will be computed with respect to the extraction of concepts and the computation of their polarity. The script to compute Precision, Recall, and F-Measure will be provided to participants through the website of the challenge.
The output of this Task will be a list of entities ordered by strength of positive judgements of any of their features. As an example, given an input list of reviews on smartphones, create a structured output of each review where smartphones are listed together with their features and opinions on each of them. This task includes Information Retrieval (detect features of given entities), Named Entity Recognition (detect smartphone models within the review possibly using some sort of knowledge base), Sentiment Analysis (aggregate features opinions for the entity sentiment for either overall or feature based retrieval). The input format for such a task is the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Sentences>
<sentence id="0">
<text>So far so good. My wife just loves the new Samsung S5: the display is awesome
and the colors are very brilliant. However, further memory is necessary for storing
everything.</text>
</sentence>
<sentence id="1">
<text>All the LG G3 have problems with videos: they often are not able to connect
with tv and when they can, the quality of the image is poor. The only strong point
is the amount of memory coming from the factory.</text>
</sentence>
</Sentences>The output format should look like the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Ranks>
<rank quality="display">
<position value="1" name="Samsung Galaxy S5"/>
<position value="2" name="LG G3"/>
</rank>
<rank quality="memory">
<position value="1" name="LG G3"/>
<position value="2" name="Samsung Galaxy S5"/>
</rank>
<rank quality="GENERAL">
<position value="1" name="Samsung Galaxy S5"/>
<position value="2" name="LG G3"/>
</rank>
</Ranks>The entire dataset (train + test) will be built from scratch. 2-4 experts will validate the annotations for this task, which consist in the computation of the relevance of documents according to the Normalized Discounted Cumulated Gain measure The script to compute Normalized Discounted Cumulated Gain measure will be provided to participants through the website of the challenge.
The Challenge focuses on semantic fine-grained sentiment analysis. This means that the proposed engines must work beyond word/syntax level, hence addressing a concepts/semantics perspective. This task will evaluate the capabilities of the proposed systems to identify the objects involved in a typical opinion frame according to their role: holders, topics, opinion concepts (i.e. terms referring to highly polarised concepts). For example, in a sentence such as "The mayor is loved by the people in the city, but he has been criticized by the state government", an approach should be able to identify that the people and state government are the opinion holders, is loved and has been criticized represent the opinion concepts, mayor identifies a topic of the opinion and that there are two different opinion polarities mentioned in the sentence. The proposed engines will be evaluated according to precision, recall and F-measure. The output format for such a task is the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Sentences>
<sentence id="348:0">
<text>The mayor is loved by the people in the city,
but he has been criticized by the state government.
</text>
<Frames>
<Frame>
<holder start="22" end="32" value="the people"/>
<topic start="0" end="9" value="The mayor"/>
<opinion start="10" end="18" value="is loved"/>
<polarity>positive</polarity>
</Frame>
<Frame>
<holder start="76" end="96" value="the state government"/>
<topic start="0" end="9" value="The mayor"/>
<opinion start="53" end="72" value="has been criticized"/>
<polarity>negative</polarity>
</Frame>
</Frames>
</sentence>
</Sentences>Input is the same without the Frames tag and its descendants nodes. As training set, we will use the dataset adopted for the last edition of the challenge; a new set of 100 annotated sentences will constitute the test set and it will be built from scratch. 2-4 experts will validate the annotations. Precision, Recall, and F-Measure will be computed against the number of recognized entities and the script to compute it will be provided to participants through the website of the challenge.
A human would easily understand that the people referred to by the sentence People hope that the President will resign have a rather negative opinion on the President because they envision his/her resignation. This simple sentence however lacks of terms explicitly indicating a positive or negative opinion, e.g. about the President, making it hard for a NLP-based tool to catch it. However, the term hope evokes a positive attitude towards what is referred to by the subordinate proposition the President will resign. This means that people refers to the holder of a positive opinion about a possible resign event (i.e., main topic) whose agent is the President (i.e. a subtopic). Intuitively, a subtopic is an entity that is indirectly targeted by an opinion sentence. In this case the opinion holder indirectly expresses an opinion on the President, while it directly expresses an opinion on a resign event. Being a resignation a generally negative event for its agent, a positive judgement of it implies a negative one on its agent. In this task a list of VerbNet verbs roles (around 40 verbs will be indicated by the challenge chairs, around 20 when the training set is released and around 20 when the test set is released) should be annotated and the developed algorithm should take into account the annotated resource in order to answer to sentences as the one mentioned at the beginning of this section. Basically each verb's role should have an annotation (positive, negative or neutral) indicating whether that role can be affected by an opinion on that verb. Challengers might have a look at a similar research paper [2] where a resources called Sentilonet and a tool Sentilo have been developed for the same purpose. The expected output should be a polarity value (positive, negative) on detected VerbNet roles of identified verbs in the sentences included in the list of selected verbs. The output format for such a task is the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Sentences>
<sentence id="348:0">
<text>Tom is happy that the President and the VicePresident were condemned.</text>
<Sentilonet>
<opinion targetverb="were condemned" verbrole="Theme"
roleobject="President" polarity="negative"/>
<opinion targetverb="were condemned" verbrole="Theme"
roleobject="VicePresident" polarity="negative"/>
</Sentilonet>
</sentence>
</Sentences>Input is the same without the Sentilonet tag and its descendants nodes. The entire dataset (train + test) will be built from scratch. We are currently annotating 100 sentences where each contains an opinion (direct or indirect) toward a verb having a positive or negative connotation with respect to some of its Verbnet roles. The list of such a verb list together with the connotation of its roles will be provided and the participants will have to identify entities (verbal form, verb role, role object, polarity of the opinion). 2-4 experts will validate the annotations. Precision, Recall, and F-Measure will be computed giving higher priority to the polarity and the role object and lower priority to the target verb and verb role tags. The script to compute it and the list of the verbs and their annotated roles will be provided to participants through the website of the challenge.
[2] Sentilo: Frame-Based Sentiment Analysis, Diego Reforgiato Recupero, Valentina Presutti, Sergio Consoli, Aldo Gangemi, Andrea Giovanni Nuzzolese, Cognitive Computation, 2014, Springer Science+Business Media New York 2014.
One award will be given for each task (the winner of each task will be the one with the highest score in precision-recall analysis) and one more award will be given for the most innovative approach (the system with the best use of common-sense knowledge and semantics and innovative nature of approach). The awards will consist in Springer vouchers and cash prize (depending on sponsors availability). Besides, each challenge paper will be included in a Springer book as already done in challenge editions of 2014 and 2015.
###Organizers
-
Mauro Dragoni, FBK, Italy (dragoni@fbk.eu) Mauro Dragoni is a Post Doctoral Researcher at Fondazione Bruno Kessler in Trento since 2011. He received his Ph.D. degree in Computer Science from the Universita` degli Studi di Milano in 2010 and his major research interests concern the Computational Intelligence and Knowledge Management fields applied to the Information Retrieval, Ontology Matching, and Sentiment Analysis topics. In particular, he focuses on applying state of the art research paradigms to the implementation of real-world knowledge management systems. He co-organized the ESWC 2015 edition of Challenge on Concept-Based Sentiment Analysis and participated at the 2014 edition by winning the awards of Most Innovative System and Best Performer on the Aspect-Based Sentiment Analysis task.
-
Diego Reforgiato Recupero, University of Cagliari, Italy (diego.reforgiato@unica.it) Diego Reforgiato Recupero is an Associate Professor at the Department of Mathematics and Computer Science of the University of Cagliari. He is also associate to the ISTC institute of the CNR working within the STLAB laboratory on Semantic Web and Natural Language Processing. In 2005 he was awarded a 3 year Post Doc fellowship with the University of Maryland where he won the Computer World Horizon Award in the USA for the best research project on OASYS, an opinion analysis system commercialized by SentiMetrix, which he co-founded. He is a patent co-owner in the field of data mining and sentiment analysis (20100023311). Dr. Reforgiato is also a co-founder of R2M Solution, where he currently serves on the board of directors. He co-organised the ESWC 2014 and ESWC 2015 Challenges on Concept-Based Sentiment Analysis and the first edition of the Workshop on Semantic Sentiment Analysis held at ESWC 2014. He has research experience across a wide array of industrial and FP7 research projects.
- Aldo Gangemi, University of Paris13 and CNR (France and Italy)
- Valentina Presutti, CNR (Italy)
- Malvina Nissim, University of Bologna (Italy)
- Erik Cambria, Nanyang Technological University (SG)
- Giuseppe Di Fabbrizio, Amazon Inc. (USA)
- Hassan Saif, Open University (UK)
- Rada Mihalcea, University of North Texas (USA)
- Ping Chen, University of Houston-Downtown (USA)
- Yongzheng Zhang, LinkedIn Inc. (USA)
- Giuseppe Di Fabbrizio, Amazon Inc. (USA)
- Soujanya Poria, Nanyang Technological University (Singapore)
- Yunqing Xia, Tsinghua University (China)
- Rui Xia, Nanjing University of Science and Technology (China)
- Jane Hsu, National Taiwan University (Taiwan)
- Rafal Rzepka, Hokkaido University (Japan)
- Amir Hussain, University of Stirling (UK)
- Alexander Gelbukh, National Polytechnic Institute (Mexico)
- Bjoern Schuller, Technical University of Munich (Germany)
- Amitava Das, Samsung Research India (India)
- Dipankar Das, National Institute of Technology (India)
- Stefano Squartini, Marche Polytechnic University (Italy)
- Cristina Bosco, University of Torino (Italy)
- Paolo Rosso, Technical University of Valencia (Spain)
To ask questions and information please join our Google Group (https://groups.google.com/forum/#!forum/semantic-sentiment-analysis). After you join the group, you can post messages to the topic "ESWC2016 Fine-Grained Sentiment Analysis Challenge"