An Efficient Language Model Using Double-Array Structures
Clone or download
Latest commit 7c8fb6f Oct 31, 2015
Type Name Latest commit message Commit time
Failed to load latest commit information.
darts-clone First commit. Nov 4, 2013
include Bugfix. Oct 31, 2015
sample fix #11. Feb 16, 2015
scripts Bugfix. Oct 31, 2015
src Bugfix. Oct 31, 2015
AUTHORS contributors. Feb 13, 2015
CMakeLists.txt mkworddict in C++. #10 Feb 18, 2015
CONTRIBUTORS contributors. Feb 13, 2015
LGPL First commit. Nov 4, 2013
LICENSE add support site information and modify licensing information. Feb 17, 2014
Makefile fix #11. Feb 16, 2015 add CMake features (experimental). Feb 13, 2015
waf change shebang to python2. fix #9 Feb 13, 2015
wscript Bugfix. Oct 31, 2015


This is an implementation of the double-array language model (DALM). This method is described on:

Makoto Yasuhara, Toru Tanaka, Jun-ya Norimatsu, Mikio Yamamoto. 2013. An Efficient Language Model Using Double-Array Structures. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.

Build Instruction

To build DALM, use waf build system. Run the command:

./waf configure build install --prefix=[install dir]

(Experimental) CMake build

To build DALM by CMake: mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=[install dir] make make install


Building a DALM model

To build a DALM model, run:

[install dir]/scripts/ [ARPA File] [Division Number] [Output Directory]

If you need to build a large language model (for example, over 30GB in ARPA Format), we recommend to set the division number "8" or "16" (or larger!).

You may need Ruby 2.0 to run the script. This script generates following files into [Output Directory]:

  • dalm.ini : information of the model.
  • dalm.bin : binary dumps of a double-array trie.
  • words.bin : binary dumps of vocabulary data.
  • words.txt : a set of words of the model.

Building a DALM model in parallel

By default, uses all cores in your CPU. If you want to reserve some computing resources, you may create /tmp/cpu_reserved file and write the number of cores you want to reserve. If you have 4 cores and you want to use only 3, you may write only '1' in /tmp/cpu_resered file.

Using DALM on your system

We include a sample program to use DALM on your system. Please see sample/query_sample.cpp. On the build time, you may link the libraries which are stored in the [install dir]/lib directory.

External Libraries

We use following libraries:

To build the source code, we use waf build system version 1.7.13. We include them into our code. Here is an original download site of waf.


The source code is provided under LGPL v3.