Skip to content
A python implementation of the neural network joint language model and an extension of it using global source context.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Neural Network Joint Language Model (NNJM) using Python/Theano

Update: You can find the technical report corresponding to this code base here: Neural Network Joint Language Model: An Investigation and An Extension With Global Source Context, by Yuhao Zhang and Charles R. Qi.

This is an implementation of a neural network joint language model (NNJM) using Python. NNJM is proposed in the context of machine translation and jointly model target language and its aligned source language to improve machine translation performance. The implementation presented here is based on a paper authored by BBN: Fast and Robust Neural Network Joint Models for Statistical Machine Translation. Besides, this implementation also allows for an extension of the NNJM model, which utilizes the entire source sentence (global source context) to improve the model peroformance measured by perplexity. We called this NNJM-Global for simplicity. Thus, this code could run in three modes:

  • NNLM: model only the target language with target N-gram.
  • NNJM: jointly model the target language and source language with target N-gram and source window.
  • NNJM-Global: jointly model the target language and source language with target N-gram, source window, and entire source sentence.

Built on top of Theano, our code could also be run on GPU supported by CUDA with no additional effort. GPU computation can speed up the neural network training for ~x10 in many cases. Please see below for detail.

Note: The code base and documentation is still in preparation. You can check later for a cleaner version.


  • README - this file
  • code/ - directory contains all the python code files.
  • data/: each subdirectory contains 3 files -.(en|zh|align) where -.align contains alignments for a pair of sentences per line. Each line is a series of pairs Chinese positions - English positions.
    • bitex/ - toy training data
    • tune/ - tuning data
    • p1r6_dev/ - test data


This code is implemented in Python. For numeric computation, we use Numpy. For symbolic computation and the construction of the neural network, we use Theano. Note that Theano is still in development, so the current version might be unstable sometimes.


Data Preprocessing

You have to preprocess the original text data in order to run the following code. Please run under the root directory.

python code/ --joint --src_lang en --tgt_lang zh data/bitex/tiny_training 1000 toy_joint
python code/ --joint --src_lang en --tgt_lang zh data/p1r6_dev/p1r6_dev 1000 toy_joint
python code/ --joint --src_lang en --tgt_lang zh data/tune/tiny_tune 1000 toy_joint

Train basic NNJM

This is an example on the toy dataset. Note that if you want to use GPU, you have to add the prefix THEANO_FLAGS setting. If you run it with CPU, you do not need to use the prefix.

THEANO_FLAGS='device=gpu,floatX=float32' python ./code/ --act_func tanh --learning_rate 0.5 --emb_dim 16 --hidden_layers 64 --src_lang zh --tgt_lang en ./data/bitex/tiny_training ./data/tune/tiny_tune ./data/p1r6_dev/p1r6_dev 5 100 1000 2 ./toy_joint

Train NNJM-Global

THEANO_FLAGS='device=gpu,floatX=float32' python ./code/ --act_func tanh --learning_rate 0.5 --emb_dim 16 --hidden_layers 64 --src_lang zh --tgt_lang en ./data/bitex/tiny_training ./data/tune/tiny_tune ./data/p1r6_dev/p1r6_dev 5 100 1000 ./toy_joint


This code is based on an original implementation provided by Thang Luong in the Stanford Natural Language Processing Group.


Thang Luong, Yuhao Zhang, Charles Ruizhongtai Qi @ Stanford University

You can’t perform that action at this time.