[Question] How to compute a marginal probability over a (contiguous) set of nodes?

[kimdev95]

Hi.

Thank you for the great library. I have one question that I hope you could help with.

How can I compute a marginal probability over a (contiguous) set of nodes? Right now, I am using your LinearChain-CRF to do NER. In addition to the best sequence itself, I also need to compute the model’s confidence in its predicted labeling over a segment of input. For example, what is the probability that a span of tokens constitute a person name?

I read your example and see how you get the marginal prob for each individual node. But I was not quite sure how to compute the marginal prob over a subset of nodes. If you could give any hint, it would be great.

Thank you.

[srush]

Hi great question.

So there are two ways to do this:

1. Use the SemiMarkov CRF. It is a model designed to given probabilities over spans. It's marginals are of the form of spans, which should give you exactly what you want.

2. Make some rough independence assumptions and use for instance the B-ROLE marginal at the beginning of the span and E-ROLE marginal at the end position.

[kimdev95]

Thank you for the suggestions. That makes a lot of sense. I will try out the SemiMarkov CRF. Thanks.

[srush]

Yeah, semimarkov is generally a really nice distribution for this sort of task. Let me know if you have any issues. You may need to change the underlying inputs somewhat.