Skip to content
master
Go to file
Code

Files

Permalink
Failed to load latest commit information.

Readme.md

A Fast and Accurate Vietnamese Word Segmenter

The implementation of RDRsegmenter, as described in our paper:

@InProceedings{NguyenNVDJ2018,
author={Dat Quoc Nguyen and Dai Quoc Nguyen and Thanh Vu and Mark Dras and Mark Johnson},
title={{A Fast and Accurate Vietnamese Word Segmenter}},
booktitle={Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)},
pages={2582--2587},
year={2018}
}

Please CITE our paper whenever RDRsegmenter is used to produce published results or incorporated into other software.

RDRsegmenter has also been incorporated into our Java NLP annotation pipeline VnCoreNLP for Vietnamese. VnCoreNLP provides rich linguistic annotations through key NLP components of word segmentation, POS tagging, named entity recognition and dependency parsing.

Usage

Supposed that Java 1.8+ is already set to run in command line or terminal (for example: adding Java to the path environment variable in Windows OS). To compile RDRsegmenter, users simply run the following command:

$ javac -encoding UTF-8 RDRsegmenter.java

Then users can run RDRsegmenter to segment a raw text corpus (e.g. a collection of news content):

$ java RDRsegmenter PATH-TO-RAW-TEXT-CORPUS

An output .WS file will be generated in the same directory containing the raw text corpus.

See Readme.md in train folder for details of retraining RDRsegmenter.

API

//Change path to "VnVocab" in line 14 in "Vocabulary.java" if necessary
//Change path to "Model.RDR" in line 24 in "RDRsegmenter.java" if necessary
RDRsegmenter segmenter = new RDRsegmenter(); //Model loading

//Perform word segmentation on text string
String outStr1 = segmenter.segmentRawString("Tính thuế thu nhập cá nhân (TNCN) từ tiền lương.");
String outStr2 = segmenter.segmentTokenizedString("Tính thuế thu nhập cá nhân ( TNCN ) từ tiền lương .");

//Perform word segmentation on text corpora
segmenter.segmentRawCorpus("PATH-TO-RAW-TEXT-CORPUS");
segmenter.segmentTokenizedCorpus("PATH-TO-TOKENIZED-TEXT-CORPUS");

rdrsegmenterexample

About

A Fast and Accurate Vietnamese Word Segmenter (LREC 2018)

Topics

Resources

License

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.