You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jun 15, 2024. It is now read-only.
Most language detectors don't work well on very short texts (in this case a single word).
You could use the model's output scores to define a threshold under which no language is detected. Otherwise the language labels on short texts will probably be noisy.
Why are language detectors so bad on short text? I get that the sample size is small but one would think they would switch approaches to a basic sanity check. e.g., the characters "age" have absolutely no correlation with the characters found in Korean. This seems to be an issue with every language detection library we've used -- pure randomness!
Please check the below sheet. For most of the simple English words it detects as different language
The text was updated successfully, but these errors were encountered: