I wonder whether this softLabelCRF can be used on NLP task? NER for example?

[lkqnaruto]

Hi

Thank you for the great work, and it is very attractive. I wonder whether this softLabelCRF can be used on NLP task? NER for example? If so, how should I modify the code, any suggestions?

Thank you in advance!

[liujch1998]

Hello! The SoftLabelCRF algorithm is useful when your sequence labeling task may require multiple ground truth labels on the same token/span.

If this is the case for your NER task, you can try using a learnable matrix as your transition matrix. (We use a model to predict each entry of this matrix, but for NER since the labels are pre-defined you can use something simpler.) You may also enforce the value of some entries by rules (e.g. O->I should have zero transition probability, or equivalently, negative infinity transition potential).

You may refer to our paper for a more intuitive understanding of the algorithm. The code is a vectorized implementation so it may be hard to read.

Hope that helps!