Skip to content
Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model 搜喵拼音输入法
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
.gitignore Initial commit Aug 29, 2017
README.md typo Sep 13, 2017
build_corpus.py update encoding Sep 9, 2017
data_load.py success Sep 10, 2017
eval.py success Sep 10, 2017
hyperparams.py update encoding Sep 9, 2017
modules.py first comit Sep 9, 2017
prepro.py update encoding Sep 9, 2017
train.py update encoding Sep 9, 2017

README.md

Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model

中文Blog

Personalized Chinese Pinyin Input Method with Seq2seq model

Original code in https://github.com/Kyubyong/neural_chinese_transliterator for research purpose.

This repository intends to experiment with different training data and interactive user inputs, and possibly develop towards a real data-personalized and model-localized Pinyin Input product.

Requrements

  • Python (>=3.5)

  • TensorFlow (>=r1.2)

  • xpinyin (for Chinese pinyin annotation)

  • distance (for calculating the similarity score between two strings)

  • tqdm

Usage

Training:

  • STEP 1. Download Leipzig Chinese Corpus

    Extract it and copy zho_news_2007-2009_1M-sentences.txt to data/ folder.

    Or use your own Chinese Corpus with the same format.

  • STEP 2. Build a Pinyin-Chinese parallel corpus.

#python3 build_corpus.py
  • STEP 3. Run prepro.py to make vocabulary and training data.
#python3 prepro.py
  • STEP 4. Adjust hyperparameters in hyperparams.py if necessary.

  • STEP 5. Train the model

#python3 train.py

Inference with command line input:

For command line input testing, run:

python3 eval.py

You may change the main function name to use the original testing data evaluation.

Testing with pre-trained models:

Download the pre-trained model from blog, unzip it to generate /log and /data.

Remember to overwrite the pickle files in /data with the pre-trained model data.

Then run for command line input testing:

python3 eval.py

Sample Results

Model is trained from Chinese News in 2007-2009. So many now common Chinese sayings are not learned.

请输入测试拼音:nihao
你好

请输入测试拼音:chenggongle
成功了

请输入测试拼音:wolegequ
我了个曲

请输入测试拼音:taibangla
太棒啦

请输入测试拼音:dacolehuizenmeyang
打破了会怎么样

请输入测试拼音:pujinghehujintaotongdianhua
普京和胡锦涛通电话

请输入测试拼音:xiangbuqilaishinianqianfashengleshenme
想不起来十年前发生了什么

请输入测试拼音:meiguohongzhawomenzainansilafudedashiguan
美国轰炸我们在南斯拉夫的大事馆

请输入测试拼音:liudehuanageshihouhaonianqing
刘德华那个时候好年轻

请输入测试拼音:shishihouxunlianyixiabilibilideyuliaole
是时候训练一下比例比例的预料了

TODOLIST

  • Pretrained models on different contexts

  • Model selection for using different models while input different things (chatting? writing scientific papers? etc...)

  • Function to record LOCALLY what user has input as personalized corpus

  • User Interface

  • ...

You can’t perform that action at this time.