Do we expect BertSum to work on text tokenized using a tokenizer other than Core-NLP?

[enzoampil]

Is it safe to assume that BERTSUM will perform without issues on input text tokenized using a tokenizer other than Core-NLP as long as we implement sentence splitting?

[nlpyang]

Hi, if the sentence splitting is correct, it should work fine. And the tokenizer is not that important, since we always use BERT's subtokenizer after the tokenization.

[enzoampil]

Thank you for confirming!

[Santosh-Gupta]

Hello,

I was wondering if you could provide a sample of what the tokenized data should look like. I am not able to install the Stanford Tokenizer from Java in Google Colab, so I am not not able to check the output files , and format the output using an alternative tokenizer.