About Xuezhe's result

[ZhixiuYe]

hello,
I replicated the LSTM-CNN-CRF model, and my best result is 91.23, which is close to Xuezhe's reported result.
I wonder why in your paper, the mean result is better than Xuezhe's reported result in LSTM-CNN-CRF model.
It is because you modified Xuezhe's code or anything else?

Thank you vary much if you can tell me about this.

[LiyuanLucasLiu]

Hi Zhixiuye,

Thanks for reaching out :-)

I'm not sure whether you re-implemented LSTM-CNN-CRF or just re-run the experiments with Xuezhe's code.

As for our experiments, we tried to fine-tune hyper parameters for all baselines (for fair comparison), and I think that's why we achieve better performance. Also we re-implemented LSTM-CNN-CRF and LSTM-CRF (would release these code later), which, however, fails to have the same performance.

[frankxu2004]

Hi Zhixiu Ye,

The experiments in our paper are compared across different hyper-parameter sets, and most importantly we used GPU to run the experiments. The GPU and CPU have some non-negligible differences in results though.

Note that with --output_prediction flag, you can output the testing result to ./tmp/ folder (remember to create it first) and then use corresponding eval script to calculate the score.

THEANO_FLAGS='floatX=float32,device=gpu' python2 bi_lstm_cnn_crf.py --fine_tune --embedding glove --oov embedding --update momentum --batch_size 10 --num_units 200 --num_filters 30 --learning_rate 0.015 --decay_rate 0.05 --grad_clipping 5 --regular none --dropout --train "train.tsv" --dev "devel.tsv" --test "test.tsv" --embedding_dict "../glove/glove.6B.100d.txt.gz" --patience 5 --output_prediction