Recognition of russian abbr names

[SergeyKishenin]

It will be cool if tokenizer will recognize russian abbr names like "В.Ф. Иванов" and make it a single token.

I've checked in irb a regexp that should do that but when i put it to word_tokenizer.rb nothing happens.
I put the following there:

[/([\p{Word}]{1}\.[\p{Word}]{1}\.)\s([\p{Word}]{1,}[^\.\s])/u, '\1\2']

Any ideas?

[zencephalon]

@SergeyKishenin is this still a problem? I want to go through the regexes to fix this issue maybe I can fix the Russian abbreviations at the same time.

[SergeyKishenin]

Seems like it's still an issue. I'll create a branch and attach a pull request here with test cases.

Definition like В.Ф.Иванов is tokenized correctly. В.Ф. Иванов and В. Ф. Иванов (with spaces) are not.

[zencephalon]

Thanks for the test cases!