How the TokenizeProcessor batching work on unpunctuted input?

[liad-inon]

I get out of memory errors on unpunctuated text input. And I believe the reason might be the batch dividing method on the TokenizeProcessor. The docs claim that the batches are divided based on "paragraphs", but I couldn't find how Stanza defines a paragraph. Is it by punctuation?

I need to confirm this before starting to write a batching method of my own.

My text input comes from OpenAI Whisper model, so punctuation is sometimes missing in my input for less supported languages.

[AngledLuffa]

It looks for multiple line breaks for paragraphs. However, if the text is more than 1000 characters long, it chunks it into batches of 1000 for going through the tokenizer model anyway. Are you seeing out of memory or out of gpu memory errors? How much text are you putting in at once?

[liad-inon]

@AngledLuffa After your explanation and more testing, I believe that the problem is further down the pipeline, with the DepparseProcessor. The missing punctuation is causing the sentence segmentation to fail horribly, parsing the all text as one sentence, and inputting a huge amount of text into the DepparseProcessor. Causing the unproportional memory usage.

I will check further and update.

[liad-inon]

Yes, the problem is indeed the DepparseProcessor getting a sentence hundreds of characters long. And decreasing the depparse_batch_size doesn't seem to help. I speculate that the batching isn't splitting sentences, which will make sense if true.

I will probably need to use a more resilient and robust sentence segmentation tool.

[AngledLuffa]

It is true that individual sentences are not split.

There is a flag to separate individual sentences into their own batch:

depparse_min_length_to_batch_separately

This might help, but it depends on just how long the non-tokenized sentences are. We could look into adding a flag that doesn't parse super long sentences no matter what. An upgraded tokenizer is a long term goal, but one we haven't worked on much recently.

[liad-inon]

Thank you @AngledLuffa.
for my use case I decided to out source the sentence segmention to the wtpsplit lib. But I will use depparse_min_length_to_batch_separately
If it fails.

I also found out that the memory consumption is dependent on the language. I couldn’t reproduce the out of memory error on English for example, while on Traditional Chinese any 5,000 characters long none punctuated text produces on my machine.

If you think creating a detailed issue with a sample to replicate will benefit the project, I will do it gladly.

This is a problem a lot of people can get unexpectedly on production, if they're input is coming from ASR models.

[liad-inon]

@AngledLuffa I will also add that more than improving the tokenizer, the docs also must get an upgrade. As a "user" of the project I had no way of knowing that the depparse_min_length_to_batch_separately flag existed, without someone telling me, or finding it in the code.

[AngledLuffa]

Good point. I added a line for that to the depparse page.