About 'X' label

[kugwzk]

In my opinion, you should remove the 'X' label's signal in evaluation, because you add more label than standard dataset, so I can't know very well the F1-score increase because the more label of 'X'. I think the 'X' label is not equal the 'O' label in standard dataset and the BERT paper, but in your code it may be same.

[kamalkraj]

Label 'X' is not considered for f1 metrics

BERT-NER/run_ner.py

Line 515 in dba5e7d

if m and label_map[label_ids[i][j]] != "X":

Label 'X' is not equal to Label 'O'

[kugwzk]

But did you use in train?

[kamalkraj]

For training I used "X".
While Inference and f1 metrics only the first output label of token is considered . same as in BERT paper

[kugwzk]

I think add more label in the conll2003 NER standard dataset make it not very comparable for previous works. Could you remove 'X' label during training and get a similar result?

[kamalkraj]

If you remove "X" while training or replace "X" with "O" , model performance will drop to ~89 f1 score

[kugwzk]

So this is my opinion, use ‘X’ label make it high F1-score, it's not fair. I get the similar result about 91.3 F1 score. And I think BERT origin paper is also remove 'X' label because they use document information to get high F1-score. In short, 'X' label don't have any signal.

[kamalkraj]

91.3 without using "X" ??

[kugwzk]

Yes. I get the word piece output from BERT model, and then map the first token's vector so I get the same numbers vectors as the standard dataset. And then use a softmax matrix to get the final result. But I only use the BERTModel in pytorch_pretrained_bert.

[kamalkraj]

For example
After extracting features for the sentence below
Jim Hen ##son was a puppet ##eer
You're giving only [Jim , Hen , was , a, puppet] hidden states to linear classification layer ?

[kugwzk]

Yes. Because I think the fine tune bert could learn this pattern.

[kamalkraj]

I will try this way and let you know

[kamalkraj]

@kugwzk
Can you share your code ?
How are you handling padding after extracting first sub-token hidden states from BERT ?

[kugwzk]

I just record the origin word position use a dict in python. For example: [Jim Hen ##son was a puppet ##eer] for [0,1,3,4,5], so I padding the origin word sequence again in the classifier layer. It may be slowly :).

[alphanlp]

i think we can add mask to X label when training

[ereday]

I agree with @kugwzk on "misusage of X label". @tkukurin In the latest version, are you still using X during training (or evaluation) or have you already removed it as suggested ?

[kamalkraj]

@ereday
Latest version still use X , I have code without usingX label , I need to clean the code a bit. I will try to push code by monday

[Nic-Ma]

Hi @kamalkraj ,

I am Nic from NVIDIA, thanks for your contribution on this project!
I tried to replace [CLS], X with O directly, and removed [SEP].
I think you supposed to release new code today, have you finished?
Thanks.

[kamalkraj]

@toxic2m
Check out branch experiment

[Nic-Ma]

Hi @kamalkraj ,

Actually, I already done this part locally, and I suggest you to map [CLS] and [SEP] to O directly.
Then your FC layer only output real number of classifications, can get better performance.
Thanks.

[sbmaruf]

Is there anyone able to reproduce BERT_NER paper's results (92.4F1 for BERT Base) ?
The experiment on the master branch supports the result given in the BERT paper. But as @kugwzk mentioned the problem. Is there anyone able to reproduce the results without inferred X tags?

[kugwzk]

@sbmaruf The result of Conll03 NER reported in the BERT origin paper used document context, which is different from the standard sentence-based evaluation. You can see something about that in here: allenai/allennlp#2067 (comment)

[sbmaruf]

@kugwzk thanks for your reply. but how to add document level context with NER?
any idea or code repo?