Using wordninja on ocr text

[Dnkhatri]

A sometimes after ocr the text come out as long string of combined words. Using wordninja specially with subtitle edit dictionary where we have added the names etc should make it a lot easier to clean up the text. It could be part of the spell check or just another option to fix text.
https://github.com/jiawenhao2015/wordninja

[niksedk]

Looks nice... is a c# version available?

[Dnkhatri]

Sadly no just Python, go and rust, no c# that I could find

[niksedk]

OK, it's fairly easy to do this... it guess it comes down to two issues:

1. Word list for all languages
2. How often will this give bad results

[Dnkhatri]

Well after looking around some more I think I have found something better it even has c# versions
https://github.com/wolfgarbe/WordSegmentationTM
https://github.com/wolfgarbe/WordSegmentationDP
https://github.com/wolfgarbe/LinSpell

[niksedk]

OK, an early test version is up now: https://github.com/SubtitleEdit/subtitleedit/releases/download/3.6.4/SubtitleEditBeta.zip
This test version includes a English split word file called Dictionaries/eng_WordSplitList.txt with 2.784 words extracted from 70 .srt files via this tool:

To activate this tool - press Ctrl+shift+alt+F12 in the main SE windows.

Works in OCR or in Fix common errors:

[Dnkhatri]

So we can use the tool to generate word lists to improve the correction when we run ocr or fix common errors?

[niksedk]

Yes - but this feature will require some testing + tuning... also, it's not using the names list atm.
Some words can be split even when it's wrong, like "meat" or "me at" ?

[Dnkhatri]

It works quite well already though its not perfect. To make it really accurate you would need to generate word lists for each language as well as word pair frequency etc by ingesting large amount of text for each language which is not really feasible. People being able to generate word list for languages they use will be good enough for most people.
I would suggest adding a short cut key as well similar to the spell check we get a line by line check so we can accept or reject the change.

[niksedk]

The Word split dictionary generator already removes non frequent words.
ATM, long words are taken first, if words are of equal length, then most frequent words are used first.
Also, if a split fails because a "wrong" word is used (e.g. "needspecial" is changed to "needs pecial"), the split will be retried without using the wrong word (e.g. "needs").

Seems to work okay, but I'll test more - and try adding "names"

[niksedk]

Name list is now used in latest beta: https://github.com/SubtitleEdit/subtitleedit/releases/download/3.6.4/SubtitleEditBeta.zip

Please test :)

[Dnkhatri]

The split dictionary now is showing the actual dictionaries. I tried combining a few names does not seem to work

[niksedk]

I need more details and concrete examples to check anything.

The "Word split dictionary generator" is only for generating a new word-split-list.

[Dnkhatri]

ThisiscityofNanxing is the subtitle. Nanxing is the name included in the name list. It stays the same when run thorough the fix common OCR errors. When I change it to Thisiscityof Nanxing then the words gets separated out properly to This is city of Nanxing . After some testing the names are no longer being split up once added to the name list. But they are not separated out either so if a name is in a string of text contains a name it is not separated out.

[Dnkhatri]

After running the generator on 1000 srt files that I had the resulting wordsplit dictionary seems to work very well. A better curated word list I expect would give even better results.

I have question as I no longer have access to the files that were giving me long string of text when using subtitle edit ocr. I am wondering is the wordsplit working before we get the ocr results as I am not getting similar long strings like previously

[niksedk]

OK, names list is now sorted (by word length descending) into the word-split-list + spell check after split is improved to use name list, so ThisiscityofNanxing should now work: https://github.com/SubtitleEdit/subtitleedit/releases/download/3.6.4/SubtitleEditBeta.zip

Yes, the word-split-list is working in the OCR (fixes should be visible in logs/guesses).

[niksedk]

I've improved the split word a bit: ignore workin' (and other *in' words) + include more words in the word split list + added some words to the English user dic.

Also, not having words like "stepmom" or "badass" in the normal dictionaries gives some annoying results - so this works best if the spell checker is not missing too many informal words often used in dialog.

https://github.com/SubtitleEdit/subtitleedit/releases/download/3.6.4/SubtitleEditBeta.zip

[Dnkhatri]

I think the last version is almost perfect with rare manual intervention required and that mostly for names only. There is one error I keep getting though I think it has to do with tesseract rather than the split word but just in case I am attaching the files. Sometime when the sentence starts with "I'm" the sentence seems to break down to single letters or double letters. I am attaching 3 files 2 where it happens 1 where it does not. Using tesseract 5 with original tesseract setting other settings disabled. For image preprocessing invert colors and crop transparent colors are used.

[Boswell-Scrubbs]

A huge thank you for this! Just tried it out and it worked beautifully. Saved hours of time.

Just BTW, here's how your fix solved my problem:

I had a set of subtitles for a TV series (a couple dozen .srt files). In every file there were many instances where two lines had been combined into one long line with no space between the two (I'm not sure if these were originally created with OCR, but somehow this error had been introduced). Example:

I'm driving into town today and willattend a meeting at the new library.

Should be:

I'm driving into town today and will
attend a meeting at the new library.

I was working my way (tediously) through each file, manually breaking the lines. I had the thought, "I wish there was some solution that could detect where the lines needed to be broken and do this for me. But, no, that's pie in the sky." However, I decided to do some googling and lo-and-behold, your amazing fix was just included in the latest release!

I used these three steps:

1. Fix common errors... Fix common OCR errors (using OCR replace list)
   I assume this is where you created your fix.
   Spaces were automagically inserted in the jammed together words.
2. _Fix common errors... Fix missing spaces
   Took care of the instances where the error included a punctuation mark, as in
   "Hey.You."
3. _Fix common errors... Break long lines

And voilà! Worked like a charm.

Thank you again for saving hours of dreary editing for me, but more importantly, for creating a tool that improves subtitle files and enhances the viewing experience for many, many people. :-)

P.S. New member -- prompted to sign up so I could leave this "thank you."

[niksedk]

Nice to hear your experience :)

[niksedk]

Yes, it seems Tesseract has issues with this font (not related to word splitting)

[niksedk]

And this feature can be turned on/off in Settings.xml via the tag "OcrUseWordSplitList"

[niksedk]

Or in UI - Options - Settings - Tools: