Skip to content

aonotas/deep-crf

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

DeepCRF: Neural Networks and CRFs for Sequence Labeling

A implementation of Conditional Random Fields (CRFs) with Deep Learning Method.

DeepCRF is a sequence labeling library that uses neural networks and CRFs in Python using Chainer, a flexible deep learning framework.

Which version of Python is supported?

  • Python 2.7
  • Python 3.4

Which version of Chainer is supported?

  • Chainer v1.24.0
  • Chainer v2.1.0

How to install?

# if you use Ubuntu
sudo apt install libhdf5-dev

git clone https://github.com/aonotas/deep-crf.git
cd deep-crf
python setup.py install

# if you want to use Chainer v1.24.0
pip install 'chainer==1.24.0'

# if you want to use Chainer v2.1.0
pip install 'chainer==2.1.0'
pip install cupy # if you want to use CUDA

How to train?

train Ma and Hovy (2016) model

$ deep-crf train input_file.txt --delimiter=' ' --dev_file input_file_dev.txt --save_dir save_model_dir --save_name bilstm-cnn-crf_adam --optimizer adam

Note that --dev_file means path of development file to use early stopping.

$ cat input_file.txt
Barack  B−PERSON 
Hussein I−PERSON 
Obama   E−PERSON
is      O 
a       O 
man     O 
.       O

Yuji   B−PERSON 
Matsumoto E−PERSON 
is     O 
a      O 
man    O 
.      O

Each line is word and gold tag. One line is represented by word [ ](space) gold tag. Note that you should put empty line (\n) between sentences. This format is called CoNLL format.

Deep BiLSTM-CNN-CRF model (three layers)

$ deep-crf train input_file.txt --delimiter=' ' --n_layer 3  --dev_file input_file_dev.txt --save_dir save_model_dir --save_name bilstm-cnn-crf_adam --optimizer adam

Deep BiLSTM-CNN-CRF model (three layers) with Multiple Input files

If input file is multiple due to large input files or many lines, please following commands. Please add this arg : --use_list_files 1

$ deep-crf train input_file_list.txt --delimiter=' ' --n_layer 3  --dev_file input_file_dev.txt --save_dir save_model_dir --save_name bilstm-cnn-crf_adam --optimizer adam --use_list_files 1
$ cat input_file_list.txt
./path_to_file/input_file_1.txt
./path_to_file/input_file_2.txt
./path_to_file/input_file_3.txt

set Pretrained Word Embeddings

$ deep-crf train input_file.txt --delimiter=' ' --n_layer 3 --word_emb_file ./glove.6B.100d.txt --word_emb_vocab_type replace_all --dev_file input_file_dev.txt

We prepare some vocab mode.

  • --word_emb_vocab_type: select from [replace_all, replace_only, additional]
  • replace_all : Replace training vocab by Glove embeddings's vocab.
  • replace_only : Replace word embedding exists in training vocab.
  • additional : Concatenate training vocab and Glove embeddings's vocab.

If you want to use word2vec embeddings, please convert Glove format.

$ head glove.6B.100d.txt
the -0.038194 -0.24487 0.72812 -0.39961 0.083172
dog -0.10767 0.11053 0.59812 -0.54361 0.67396
cat -0.33979 0.20941 0.46348 -0.64792 -0.38377
of -0.1529 -0.24279 0.89837 0.16996 0.53516
to -0.1897 0.050024 0.19084 -0.049184 -0.089737
and -0.071953 0.23127 0.023731 -0.50638 0.33923
in 0.085703 -0.22201 0.16569 0.13373 0.38239

Additional Feature Support

$ deep-crf train input_file_multi.txt --delimiter=' ' --input_idx 0,1 --output_idx 2 --dev_file input_file_dev.txt --save_dir save_model_dir --save_name bilstm-cnn-crf_adam_additional --optimizer adam
$ cat input_file_multi.txt
Barack  NN B−PERSON 
Hussein NN I−PERSON 
Obama   NN E−PERSON
is      VBZ O 
a       DT  O 
man     NN  O 
.       .   O

Yuji  NN B−PERSON 
Matsumoto NN E−PERSON 
is      VBZ O 
a       DT  O 
man     NN  O 
.       .   O

Note that --input_idx means that input features (but word feature must be 0-index) like this example.

Multi-Task Learning Support

(Now developing this multi-task learning mode...)

$ deep-crf train input_file_multi.txt --delimiter ' ' --model_name bilstm-cnn-crf --input idx 0 --output idx 1,2 

How to predict?

$ deep-crf predict input_raw_file.txt --delimiter=' ' --model_filename ./save_model_dir/bilstm-cnn-crf_adam_epoch3.model --save_dir save_model_dir --save_name bilstm-cnn-crf_adam  --predicted_output predicted.txt

Please use following format when predict.

$ cat input_raw_file.txt
Barack Hussein Obama is a man .
Yuji Matsumoto is a man .

Note that --model_filename means saved model file path. Please set same --save_name in training step.

How to predict? (Additional Feature)

$ deep-crf predict input_file_multi.txt --delimiter=' ' --input_idx 0,1 --output_idx 2 --model_filename ./save_model_dir/bilstm-cnn-crf_multi_epoch3.model --save_dir save_model_dir --save_name bilstm-cnn-crf_multi  --predicted_output predicted.txt

Note that you must prepare CoNLL format input file when you use additional feature mode in training step.

$ cat input_file_multi.txt
Barack  NN B−PERSON 
Hussein NN I−PERSON 
Obama   NN E−PERSON
is      VBZ O 
a       DT  O 
man     NN  O 
.       .   O

Yuji  NN B−PERSON 
Matsumoto NN E−PERSON 
is      VBZ O 
a       DT  O 
man     NN  O 
.       .   O

How to evaluate?

$ deep-crf eval gold.txt predicted.txt
$ head gold.txt
O
O
B-LOC
O
O

B-PERSON

How to update?

cd deep-crf
git pull
python setup.py install

Help (how to use)

deep-crf train --help

If CUDNN ERROR

if you got CUDNN ERROR, please let me know in issues.

You can cudnn-off mode with --use_cudnn=0

Features

DeepCRF provides following features.

  • Bi-LSTM / Bi-GRU / Bi-RNN
  • CNN for character-level representation
  • Pre-trained word embedding
  • Pre-trained character embedding
  • CRFs at output layer
  • CoNLL format input/output
  • Raw text data input/output
  • Training : Your variable files
  • Test : Raw text file at command-line
  • Evaluation : F-measure, Accuracy

Experiment

POS Tagging

Model Accuracy
CRFsuite 96.39
deep-crf 97.45
dos Santos and Zadrozny (2014) 97.32
Ma and Hovy (2016) 97.55

Named Entity Recognition (NER)

Model Prec. Recall F1
CRFsuite 84.43 83.60 84.01
deep-crf 90.82 91.11 90.96
Ma and Hovy (2016) 91.35 91.06 91.21

Chunking

Model Prec. Recall F1
CRFsuite 93.77 93.45 93.61
deep-crf 94.67 94.43 94.55
Huang et al. (2015) - - 94.46

About

An implementation of Conditional Random Fields (CRFs) with Deep Learning Method

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages