Skip to content

Latest commit

 

History

History
245 lines (152 loc) · 7.58 KB

sentiment_analysis.md

File metadata and controls

245 lines (152 loc) · 7.58 KB
Raw

Chinese Sentiment Analysis

Background

Sentiment Analysis detects identifies and extracts subjective information from text.

Example

Input:

总的感觉这台机器还不错，实用的有：阴阳历显示，时间与日期快速转换, 记事本等。

Output:

Positive

Standard Metrics

Accuracy

  • The percentage of correctly classified samples on test set.

F1-score

  • Combination of precision and recall.
  • Wiki Page

SemEval-2016 Task 5.

SemEval-2016 Task 5 contains 2 test sets with over 5000 reviews in total from digital camera and mobile phone area.

Source Genre # Classes Size(sentences) Size(words)
SemEval 2016 Task 5 – CAM Test Digital Camera reviews (Chinese) 3 2256 ~25k
SemEval 2016 Task 5 – PHNS Test Mobile Phone reviews (Chinese) 3 3191 ~34k

Metrics

  • Accuracy

Results

Accuracy(PHNS Test) Accuracy(CAM Test)
SenHint 0.7958 0.8711

Resources

Source Genre # Classes Size(sentences) Size(words)
SemEval 2016 Task 5 – CAM Train Digital Camera reviews (Chinese) 3 5784 ~61k
SemEval 2016 Task 5 – PHNS Train Mobile Phone reviews (Chinese) 3 6330 ~62k

NLP&CC 2012.

NLP&CC 2012 Test: Chinese Weibo sentiment analysis evaluation data.

Source Genre # Classes Size(sentences) Topics
NLP&CC 2012 Test Weibo reviews 2 1908 10

Metrics

  • F1-score
  • Accuracy

Results

F1 Accuracy
Chen, et al 2018 -- 88.35
Wang, et al 2013 63.60 74.00

Resources

Source Genre # Classes Size(sentences) Size(words)
NLP&CC 2012 Train Weibo reviews(Chinese) 2 1765 ~116k

ChnSentiCorp.

ChnSentiCorp: It contains 1021 documents in three domains: education, movie and house.

Source Genre # Classes Size(sentences) Size(words)
ChnSentiCorp Test Hotel reviews(Chinese) 2 1999 ~725k

Metrics

  • Accuracy
  • F1-score

Results

F1 Accuracy
Chen et al., 2020: 3SiBert 0.967
ERNIE 2.0 0.958
ERNIE 0.954
BERT * 0.943
fastText ** 0.9218 0.9218
MCCNN 0.9208 0.9208

*Bert accuracy result is cited from ERNIE paper.

**fastText accuracy result is cited from MCCNN paper.

Resources

Source Genre # Classes Size(sentences) Size(words)
ChnSentiCorp Train Hotel reviews(Chinese) 2 8000 ~2.9M

IT168TEST.

IT168TEST: A product review dataset presented by Zagibalov and Carroll. This dataset contains over 20000 reviews, in which 78% were manually labeled as positive and 22% labeled as negative.

Source Genre # Classes Size(sentences)
IT168Test Product review 2 29531

Metrics

  • Accuracy
  • F1-score

Results

F1 Accuracy
fastText* 0.9261 0.9261
MCCNN 0.9302 0.9304
Zhang, P., & He, Z. (2013) 0.9402 0.9500

*Accuracy result is cited from MCCNN paper.

Dianping.

Dianping: Chinese restaurant reviews were evenly split as follows: 4 and 5 star reviews were assigned to the positive class while 1-3 star reviews were in the negative class.

Source Genre # Classes Size(sentences)
Dianping restaurant reviews 2 500,000

Metrics

  • Accuracy

Results

Accuracy
Sun, Baohua, et al 77.8
Zhang and Lecun 2017 77.7

Resources

Source Genre # Classes Size(sentences)
Dianping restaurant reviews 2 2,000,000

JD Full.

JD Full: Chinese shopping reviews were evenly split for predicting full five stars.

Source Genre # Classes Size(sentences)
JD Full shopping reviews 5 250,000

Metrics

  • Accuracy

Results

Accuracy
Sun, Baohua, et al 54.1
Zhang and Lecun 2017 52.0

Resources

Source Genre # Classes Size(sentences)
JD Full shopping reviews 5 3,000,000

JD Binary.

  • JD Binary: Chinese shopping reviews are evenly split into positive (4-and-5 star reviews)and negative (1-and-2 star reviews) sentiments, ignoring 3-star reviews.
Source Genre # Classes Size(sentences)
JD Binary shopping reviews 2 360,000

Metrics

  • Accuracy

Results

Accuracy
Sun, Baohua, et al 92.2
Zhang and Lecun 2017 91.3

Resources

Source Genre # Classes Size(sentences)
JD Binary shopping reviews 2 4,000,000

Other Resources

Name Description Domain/ Source Size (positive/ negative where applicable) Accuracy F1 Link
Chinese Sarcasm Dataset Text manually labelled as sarcastic or not news 2500 / 90 000 0.7611 0.7368 Gong et al., 2020
CH-SIMS Individually labelled multi-modal (text, video, audio) movies, TV shows 2281 video segments - 0.827 Yu et al., 2020
FiTSA Aspect-based sentiment analysis for financial news news 8314 sentences, 647 000 characters - 0.798 Yuan et al., 2020
MPDD Emotion in multi-party dialogs TV shows 25 500 utterances 0.595 - Cheng et al., 2020
MIMN Multimodal (text, image) and aspect-based analysis zol.com (shopping site) 5200 reviews 0.616 0.605 github

Suggestions? Changes? Please send email to chinesenlp.xyz@gmail.com