Vitk -- A Vietnamese Text Processing Toolkit
NOTE: This repos is now obsolete. Interested programmers should consider to use the new repo vlp (github.com/phuonglh/vlp) We have preferred using Scala instead of Java since 2016.
This is the third release of a Vietnamese text processing toolkit, which is called "Vitk", developed by Phuong LE-HONG at College of Science, Vietnam National University in Hanoi.
There are some toolkits for Vietnamese text processing which are already published. However, most of them are not readily scalable for large data processing. This toolkit aims at the ability of processing big text data. For this reason, it uses Apache Spark as its core platform. Apache Spark is a fast and general engine for large scale data processing. Therefore, Vitk is a fast cluster computing toolkit.
If you don't want to use Apache Spark, you should download and use a standalone Vietnamese tokenizer or tagger from their website, only one JAR file is needed to run the program. vnTokenizer 5.1 and vnTagger
Despite of its name, this toolkit supports processing in various natural languages providing that suitable underlying models or linguistic resources are available for the different languages. The toolkit is packaged with models and resources for processing Vietnamese. The users can build models for other languages using the underlying tools.
- The word segmentation tool of Vitk can tokenize a text of two million Vietnamese syllables in 20 seconds on a cluster of three computers (24 cores, 24 GB RAM), giving an accuracy of about 97%.
- The part-of-speech tagger of Vitk can tag about 1,105,000 tokens per second, on a single machine, giving an accuracy of about 95% on the Vietnamese treebank.
- The dependency parser of Vitk parses 12,543 sentences (204,586 tokens) of the English universal dependency treebank (English UDT) in less than 20 seconds, giving an accuracy of 68.28% (UAS) or 66.30% (LAS).
Currently, Vitk consists of three fundamental tools for text processing:
- Word segmentation
- Part-of-speech tagging
- Dependency parsing
The word segmentation tool is specific to the Vietnamese language. The other tools are general and can be trained to parse any language. We are working to develop and integrate more fundamental tools to Vitk such as named entity recognition, constituency parsing, opinion mining, etc.
Setup and Compilation
Download a prebuilt version of Apache Spark. Vitk uses Spark version 1.6.x. Unpack the compressed file to a directory, for example
~is your home directory.
Download Vitk, either a binary archive or its source code. The repository URL of the project is Vitk. The source code version is preferable. It is easy to compile and package Vitk: go to the top-level directory of Vitk and invoke the following command at a shell window:
mvn compile package
Apache Maven will automatically resolve and download dependency libraries required by Vitk. Once the process finishes, you should have a binary jar file
vn.vitk-3.0.jarin the sub-directory
Data files used by Vitk are specified in sub-directories of the directory
corresponding to its integrated tools.
- The data used by word segmentation are in the sub-directory
- The data used by part-of-speech tagging are in the sub-directory
- The data used by dependency parsing are in the sub-directory
These folders can contain data specific to a natural language in
use. Each language is specified further by a sub-directory whose name
is an abbreviation of the language name, for example
en for English,
fr for French, etc.
Vitk can run as an application on a stand-alone cluster mode or on a real cluster. If it is run on a cluster, it is required that all machines in the cluster are able to access the same data files, which are normally located in a shared directory readable by all the machines.
If you use a Unix-like operating system (Unix/Linux/MacOS), it is easy to share or
"export" a directory via a network file system
searches for data files in the directory
you need to copy the sub-directories
dat/* into that directory, so
that you have some folders as follows:
If you run Vitk on a stand-alone cluster mode, it is sufficient to create the data folders specified above on your single machine. The NFS stuffs can be ignored.
Vitk is an Apache Spark application, you run it by submitting the
main JAR file
vn.vitk-3.0.jar to Apache Spark. The main class of the
vn.vitk.Vitk which selects the desired tool by following
arguments provided by the user.
The general arguments of Vitk are as follows:
-m <master-url>: the master URL for the cluster, for example
spark://192.168.1.1:7077. If you do not have a cluster, you can ignore this parameter. In this case, Vitk uses the stand-alone cluster mode, which is defined by
local[*], that is, it uses all the CPU cores of your single machine for parallel processing.
-t <tool>: the tool to run, where
toolis an abbreviation of the tool:
tokfor word segmentation (or tokenization);
tagfor part-of-speech tagging,
depfor dependency parsing. If this argument is not specified, the default
toktool is used.
-l <language>: the natural language to process, where
languageis an abbreviation of language name which is either
en(English). If this argument is not specified, the default language is Vietnamese.
-v: this parameter does not require argument. If it is used, Vitk runs in verbose mode, in which some intermediate information will be printed out during the processing. This is useful for debugging.
Note that if you are processing very large texts, for a better performance,
you should consider to set appropriate options of the
command, in particular,
more on submitting Apache Spark
In addition to the general arguments above, a specific tool of Vitk requires its own arguments. The usage of each tool is described in their corresponding page as follows:
You can also import the source code of Vitk to your favorite IDE
(Eclipse, Netbeans, etc), compile and run from source, for example,
launch the class
vn.vitk.tok.Tokenizer for word segmentation,
providing appropriate arguments as described above.
The algorithms used by the tools of Vitk can be found in some related scientific publications. However, some of the main methods implemented in Vitk have been, and will be described in a more accessible way by blog posts. For example, the word segmentation method is described in:
- Writing tests
- Code review
Any bug reports, suggestions and collaborations are welcome. I am reachable at:
- LE-HONG Phuong, http://mim.hus.edu.vn/lhp/
- College of Science, Vietnam National University in Hanoi