Treat entire document as sequence

[alanakbik]

In sequence labeling, we currently split documents at the sentence level and shuffle sentences at each epoch during training. However, some information may carry over between sentences in the same document, for instance an entity might be mentioned several times in different sentences in the same document.

Similarly, some datasets like CoNLL-03 explicitly mark up document boundaries with the -DOCSTART- token.

Task: Create a modified dataset reader for sequence labeling datasets that lets us provide a document separator token so that each full document is read into a single sequence object.

[mauryaland]

It is definitely a nice feature to develop! You could be interested by the paper released by Valentin Barrière and I about named entity recognition on long documents, specifically french court decisions : May I Check Again? -- A simple but efficient way to generate and use contextual dictionaries for Named Entity Recognition. Application to French Legal Texts

[alanakbik]

@mauryaland interesting paper! Did you try this method for publicly available datasets like CoNLL-03?

[mauryaland]

My colleague is currently working on it, I let you know for the results!

[alanakbik]

Thanks!

[pommedeterresautee]

@mauryaland can you publish your results here? (if it s ok for the Court)
I will do the same on my dataset and update the article Flair / Spacy accordingly?
Spacy has included some data augmentation on their side. Should improve their results out of the box (closer to what I get with pre train on synthetic data)
@alanakbik have you noticed better scores on CONLL 2003?

[alanakbik]

There was a slight improvement on CoNLL-03, but I only did one experiment run so far so I am not sure if it is significant. I'll do a 5x run to get better numbers soon and report them here!

However, I think with sequence of this length it might make sense to add attention to the sequence tagger so that it can more explicitly learn document-level entity tags.

[mauryaland]

@pommedeterresautee No problem, I will post results here, probably next week!

@alanakbik Bringing attention is a good idea indeed!