Detection error when encounter full-width characters

[joewong826]

The langid mistakens full-width English texts like 'ｈｅｌｌｏ　ｗｏｒｌｄ' as CJK language texts.
>>> import langid
>>> langid.classify('ｈｅｌｌｏ　ｗｏｒｌｄ')
('zh', 0.9339664571825803)

[saffsd]

Thanks for reporting this! Unfortunately there is no easy fix for this - langid.py training data didn't contain any "full-width" English text. If this is an issue for you in a real use case, here are possible options:

1. detect and pre-process "full-width" text into normal text
2. re-train langid.py with "full-width" text