Add language detection

[tfmorris]

A function to detect the language of a piece of text would be useful. There's a high performance (99% accuracy), Apache licensed implementation in Java here: https://github.com/shuyo/language-detection (originally http://code.google.com/p/language-detection/)

[kiminoa]

One interesting use case to note for fine tuning ... I use OpenRefine for work on alphasyllabaries. (The clustering algorithms are fantastic for a lot of language / inflection-identifying / etc. work.) At least for Mycenaean Greek (Linear B) and Crypiot Greek alphasyllabaries, these are represented by delimiting between morphemes with white space and internally with dashes, i.e. ti-ri-to represents a 3-symbol word. Would be slick if any language detection in OpenRefine was also wise enough to say "oh, hey, this is isn't an alphabetic transliteration, is it? huh."

[tfmorris]

@kiminoa I'm not sure whether you are requesting detection of transliterations or just making sure it doesn't inappropriately try to detect a transliteration as a language. We can do the latter with a confidence threshold, but the language models are all pre-built for a fixed set of languages.

[kiminoa]

The latter, for starters. Chrome's language detection (Compact Language Detection (CLD)) usually matches Linear B transliterations to Croatian, amusingly. Are the CLD and Apache's implementation using the same underlying code?

[magdmartin]

The Google Language API can also do that job see old wiki for details: http://code.google.com/p/google-refine/wiki/FetchingURLsFromWebServices

[tfmorris]

@kiminoa I'm pretty sure Apache's implementation is different from the Chrome implementation. We'll try to make sure that the confidence thresholds get set appropriately, but I think Linear B transliterations might be considered an edge case.

@magdmartin That page is available in the current wiki as well

[symac]

I found this issue while trying to find a solution to this problem, seeing this issue was still opened, I have tried to find a solution using jython, it proved to work quite well: http://www.geobib.fr/blog/2020-04-29-openrefine-detect-lang

[elroykanye]

Hello @tfmorris @wetneb , I wish to ask if it is okay that I work on this

[wetneb]

Sure! It seems a bit difficult for a first issue actually, I am not sure I agree with my previous assessment (it might involve adding a new dependency, and that is not so easy since we need to watch out for licensing issues and size considerations)

[elroykanye]

Ah I see 🤔
Worth the try anyway. I'll let you know of every step I take 😄🙏🏾

[elroykanye]

@wetneb
Just to be clear, I'd like to ask. This feature will help detect language from which piece of text? The datasource? Or is it to detect from the browser settings?
I am a bit confused. Please can I get some pointers with, perhaps, an example showing the use case?

[wetneb]

I imagine this could be implemented as a GREL function, for instance called detectLanguage, which would work like this:

• detectLanguage("hello world, what a beautiful day") evaluates to "en"
• detectLanguage("bonjour le monde, quel jour magnifique") evaluates to "fr"
• and so on

[elroykanye]

Okay. From what I understand, it should be something that implements this interface:

https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/grel/Function.java

Am I in the right track?