Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
An Efficient Language Model Using Double-Array Structures
C++ C Ruby Python Shell Makefile
Branch: master

README.md

DALM

http://www.jnory.com/research/dalm

This is an implementation of the double-array language model (DALM). This method is described on:

Makoto Yasuhara, Toru Tanaka, Jun-ya Norimatsu, Mikio Yamamoto. 2013. An Efficient Language Model Using Double-Array Structures. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.

Build Instruction

To build DALM, use waf build system. Run the command:

./waf configure build install --prefix=[install dir]

(Experimental) CMake build

To build DALM by CMake: mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=[install dir] make make install

Usage

Building a DALM model

To build a DALM model, run:

[install dir]/scripts/build_dalm.sh [ARPA File] [Division Number] [Output Directory]

If you need to build a large language model (for example, over 30GB in ARPA Format), we recommend to set the division number "8" or "16" (or larger!).

You may need Ruby 2.0 to run the script. This script generates following files into [Output Directory]:

  • dalm.ini : information of the model.
  • dalm.bin : binary dumps of a double-array trie.
  • words.bin : binary dumps of vocabulary data.
  • words.txt : a set of words of the model.

Building a DALM model in parallel

By default, build_dalm.sh uses all cores in your CPU. If you want to reserve some computing resources, you may create /tmp/cpu_reserved file and write the number of cores you want to reserve. If you have 4 cores and you want to use only 3, you may write only '1' in /tmp/cpu_resered file.

Using DALM on your system

We include a sample program to use DALM on your system. Please see sample/query_sample.cpp. On the build time, you may link the libraries which are stored in the [install dir]/lib directory.

External Libraries

We use following libraries:

To build the source code, we use waf build system version 1.7.13. We include them into our code. Here is an original download site of waf.

License

The source code is provided under LGPL v3.

Something went wrong with that request. Please try again.