Confuse problem

[chzeze]

Hi Max
Is NERCRF.py the same to the bi_lstm_cnn_crf.py in the LasagneNLP?

[XuezheMax]

Yes, it is almost the same.

…

On Mon, Nov 20, 2017 at 8:40 PM, chzeze ***@***.***> wrote: Hi Max Is NERCRF.py the same to the bi_lstm_cnn_crf.py in the LasagneNLP? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADUtlg_xoWW7FuPacOa_01C5Kl9oCMaOks5s4in2gaJpZM4QlKHA> .

-- ------------------ Best regards, Ma，Xuezhe Language Technologies Institute, School of Computer Science, Carnegie Mellon University Tel: +1 206-512-5977

[chzeze]

I use the database UKPLabhttps://github.com/UKPLab/acl2017-neural_end2end_am/tree/master/data/conll/Paragraph_Level run NERCRF.py and bi_lstm_cnn_crf.py they display diferent result.
NERCRF.py run result precision recall F1 almost zero:
Epoch 1 (LSTM(std), learning rate=0.0100, decay rate=0.0500 (schedule=1)):
train: 96 loss: 1050766784.1735, time: 351.21s
dev acc: 11.67%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
Epoch 2 (LSTM(std), learning rate=0.0095, decay rate=0.0500 (schedule=1)):
train: 96 loss: 102558966.3255, time: 258.39s
dev acc: 11.69%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
Epoch 3 (LSTM(std), learning rate=0.0091, decay rate=0.0500 (schedule=1)):
train: 96 loss: 47132442.5896, time: 257.53s
dev acc: 11.67%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)

......

[chzeze]

I use the eval to count f1 ：
@ManniSingh @XuezheMax
processed 12227 tokens with 501 phrases; found: 6261 phrases; correct: 0.
accuracy: 16.33%; precision: 0.00%; recall: 0.00%; FB1: 0.00
Claim: precision: 0.00%; recall: 0.00%; FB1: 0.00 2
MajorClaim: precision: 0.00%; recall: 0.00%; FB1: 0.00 0
Premise: precision: 0.00%; recall: 0.00%; FB1: 0.00 6259
why they are zero?

one of tmp/942fb2_dev11
942fb2_dev11.txt

[ManniSingh]

It seems PyTorch problem, you should clean restart.

[XuezheMax]

make sure that you have the tmp dir so that the program can dump prediction files for evaluation.

…

On Tue, Nov 21, 2017 at 6:50 AM, chzeze ***@***.***> wrote: I use the eval to count f1 ： processed 12227 tokens with 501 phrases; found: 6261 phrases; correct: 0. accuracy: 16.33%; precision: 0.00%; recall: 0.00%; FB1: 0.00 Claim: precision: 0.00%; recall: 0.00%; FB1: 0.00 2 MajorClaim: precision: 0.00%; recall: 0.00%; FB1: 0.00 0 Premise: precision: 0.00%; recall: 0.00%; FB1: 0.00 6259 why they are zero? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADUtlr6g6v1he7R49ZZGuShIjNRsjljiks5s4rkVgaJpZM4QlKHA> .

-- ------------------ Best regards, Ma，Xuezhe Language Technologies Institute, School of Computer Science, Carnegie Mellon University Tel: +1 206-512-5977

[chzeze]

Yes ,I have the tmp dir ,the dir contain dev predicton file and score file.
I just confuse why run NERCRF.py and bi_lstm_cnn_crf.py they display diferent result.
use database https://github.com/UKPLab/acl2017-neural_end2end_am/tree/master/data/conll/Paragraph_Level

[bbruceyuan]

hello, Max, thank you for your great contribution on the awesome work.

I met the exactly same problem because I use the same dataset as described by @chzeze .

Does anyone solved this problem?

If there exist any solution, please let me know.

**thank you all of you **

[XuezheMax]

Hi @hey-bruce and @chzeze ,
I guess I have found the reason. Each line in the data you provided are separated by '\t', where the format in the NERCRF.py is whitespace ' '.

[bbruceyuan]

thank you for your reply.

I reformated the data. and now each line in the date are separated by ' '(whitespace).

the problem is also there.

I can offer you the data through my github repo, can you test it?

thank u

[XuezheMax]

Yes, please share me the data.
Thanks.

[bbruceyuan]

You can get the data from here

thank you.

the motivation that I used your method is I am trying my best to reproduce the artical. And the author used your repo "LasagneNLP"。thank you again

[XuezheMax]

Hi @hey-bruce ,
My previous reader cannot handle multiple continuous blank lines. I have revised my code to handle it.
Now the stats info can match the one reported in the paper.
But the performance is still zero. I guess it is not an issue for the model. Please first check if you use the model the right way. Second, please make sure that the evaluation script for NER is suitable for the new task. The evaluation script used in my code is from CoNLL 2003 shared task, which is designed for NER.

Before you run your code locally, please make sure that you do the following two things:

1. git pull to get the latest version.
2. remove the data/alphabets/ folder to create a new one. If the code detect the folder, it will assumes that the alphabets have already been created and will try to load them from disk.

[bbruceyuan]

I really appreciate your help.

I tried it just now. And yes, it's not your model's issue and the evaluation script is not suitable for my task. maybe I should find a new strategy to evaluate it.

And I think you can close this issue now