Add Polyglot-NER Dataset

[joeddav]

Adds the Polyglot-NER dataset with named entity tags for 40 languages. I include separate configs for each language as well as a combined config which lumps them all together.

[stefan-it]

Hi @joeddav thanks for adding this! (I did a long webarchive.org session to actually find that dataset a while ago).

One question: should we manually correct the labeling scheme to (at least) IOB1?

That means "LOC" will be converted to "I-LOC". IOB1 is not explict. mentioned in the paper, but it is used in the documentation:

https://polyglot.readthedocs.io/en/latest/NamedEntityRecognition.html

[joeddav]

@stefan-it I went back and forth on this. My biggest problem with it is that once you are in IOB, there is the expectation that the beginning of new entities are marked with a B- (at least in the case of two back-to-back entities):

Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O 
ate      O
lasagna  O

If we just prepend I- to everything, Bob would be incorrectly tagged I-PER, meaning Bob Alice is a single entity. The current format is bad but is at least clear that it does not contain that information.

But I could go either way if someone has a strong opinion.

[lhoestq]

Indeed I'm not sure we can convert them to IOB because of this issue. I'm fine with keeping it like that

[lhoestq]

I'll do a release later today, hopefully we can include this dataset in the release :)

Let me know if you need help with the dummy data

[joeddav]

@lhoestq cool thanks, I think I've got it right now – just zipped them wrong. I'm running tests locally now and then will push.

[joeddav]

@lhoestq set to merge?

[stefan-it]

@joeddav I'm fine with keeping the original labeling scheme :)