word_segmentation.md

Chinese Word Segmentation

Overview

Chinese is written using characters (hanzi), where each character represents a syllable. A word is usually taken to consist of one or more character tokens. There are no spaces between words. Less than 3500 distinct characters are normally encountered. Word segmentation (or tokenization) is the process of dividing up a sequence of characters into a sequence of words.

Example

Input:

亲 请问有什么可以帮您的吗？

Output:

亲 请问 有 什么 可以 帮 您 的 吗 ？

Metrics

Word F1 score:

Gold: 共同 创造 美好 的 新 世纪 —— 二○○一年 新年 贺词

Hypothesis: 共同 创造 美 好 的 新 世纪 —— 二○○一年 新年 贺词

Precision = 9 / 11 = 0.818

Recall = 9 / 10 = 0.9

F1 = 0.857

The Second International Chinese Word Segmentation Bakeoff in SIGHAN 2005 Workshop (Emerson, 2005).

• Website, Detailed Instruction, Overview Paper
• Includes 4 datasets: AS, CityU in traditional Chinese, PK, MSR in simplified Chinese.

Corpus	Abbrev.	Encoding	Test Size (Tokens/Types)
Traditional Chinese
Academia Sinica(Taipei)	AS	Unicode/Big Five Plus	122K / 19K
City University of Hong Kong	CityU	HKSCS Unicode/Big Five	104K / 13K
Simplified Chinese
Peking University	PK	CP936/Unicode	41K / 9K
Microsoft Research	MSR	CP936/Unicode	107K / 13K

Results

Model	AS	CITYU	MSR	PKU
Tian, Song, Xia, Zhang, Wang (2020)	96.6	97.9	98.4	96.5
Meng et al. (2019)	96.7*	97.9*	98.3	96.7
Huang et al. (2019)	96.6	97.6	97.9	96.6
Ma et al. (2018)	96.2	97.2	97.4	96.1
Yang et al. (2017)	95.7	96.9	97.5	96.3
Zhou et al. (2017)			97.8	96.0

^* Unlike others, Meng et al. (2019) do not report converting traditional Chinese to simplified Chinese.

Resources

Train set	Training Size(Words)
AS	5.45M
CityU	1.46M
MSR	2.37M
PKU	1.1M

Chinese Penn Treebank.

• Website
• Includes 3 datasets:
  • CTB6: consisting of 780,000 words (over 1.28 million Chinese characters)
  • CTB7: consists of 2,448 text files, 51,447 sentences, 1,196,329 words and 1,931,381 hanzi (Chinese characters)
  • CTB9: consists of 3,726 text files, 132,076 sentences, 2,084,387 words, 3,247,331 characters (hanzi or foreign)

Data set	Test set (Tokens)
CTB6	82K
CTB7	245K
CTB9	242K

Results

Model	CTB6	CTB7	CTB9
Tian, Song, Ao, Xia, Quan, Zhang, Wang (2020)	97.5	97.3	97.8
Tian, Song, Xia, Zhang, Wang (2020)	97.3
Yan et al. (2020)		97.1	97.6
Huang et al. (2019)	97.6
Ma et al. (2018)	96.7	96.6**
Yang et al. (2017)	96.2
Zhou et al. (2017)	96.2

^** Ma et al. (2018) report different statistics for their CTB7 split (950K/60K/82K), so the results might not be comparable.

Resources

Train set	Training Size (Words)
CTB6	641K
CTB7	718K
CTB9	1,696K

Chinese Universal Treebank (UD).

• Website, Github

Data set	Test set(Tokens)
UD	12,012

Results

Model	UD
Tian, Song, Ao, Xia, Quan, Zhang, Wang (2020)	98.3
Huang et al. (2019)	97.3
Ma et al. (2018)	96.9

Resources

Train set	Training Size(Words)
UD	98,608

NLPCC2016 WordSeg Weibo.

• Github
• Paper describe the dataset

	# Sentences	# Words	# Characters
Weibo	8,592	-	315,857

Results

Model	Weibo
Yang et al. (2017)	95.5

Resources

	# Sentences	# Words	# Characters
Train	20,135	421,166	688,734
Dev	2,052	43,697	73,244

Other Resources

• Chinese Word Segmentation: Another Decade Review (2007-2017)

---

Suggestions? Changes? Please send email to chinesenlp.xyz@gmail.com