adding option for spacy large tokenizer

[yenniejun]

I found that spacy large model performed better in retrieving annotated entities than scispacy ... would be good to have this additional tokenizing option!

[smitkiri]

Thank you so much for pulling up this PR!! I've added some comments which might need some changes, but feel free to ignore the "Nit" ones.

Also, can you share the results that you got with this model? If they are significantly better and reproducible, we can also change the defaults in the code!

[yenniejun]

In terms of spacy_lg, I found that it does better in tokenizing the clinical note as compared to scispacy (both models) into BIO format. Using spacy_lg I could match a greater proportion of the BIO annotations as compared to the annotations using scispacy tokenization. (I did not do the e2e task yet, just doing the BIO formatting so far)

[smitkiri]

Thanks for making the changes!! But I noticed that some tokens that spacy_lg generates contain just \n which throws off the format of train.txt / test.txt used by the models. Every new line in train.txt represents a new token and it should have a [token]<space>[label], but because of the \n characters, there are some line which are just blank (\n) followed by a line with just the label.

This was the reason why I use the function scispacy_plus_tokenizer for training the BiLSTM-CRF model, using just scispacy raised errors. (Now that I'm writing this, I feel I should remove the scispacy option altogether).

From a quick look, I agree with you that the spacy large tokens look better than scispacy ones. Did you face any issues when trying to train a model with the generated files?

[smitkiri]

It is much much faster to run generate_data.py if you change this line to tokenizer = spacy.load("en_core_web_lg").tokenizer