Sorting using ICU

[kneth]

Sorting is currently done in "English locale". Using Android's ICU library could make it possible to specify the locale to sort in.

[cmelchior]

This would also fix #581

[saket]

Any updates on this? This drawback makes search kinda difficult for non-English users. I'm forced to manually iterate through RealmResult and compare text.

[kneth]

Sorry, we don't have updates on it. We fully understand searching and sorting for non-English character sets are painful. ICU is a very large dependency to add (our jar file could easily double in size), and we would like to find a best solution.

[saket]

Android already has ICU. Why add it again?

[kneth]

Different Android versions are using different ICU versions. And using non-NDK libraries in C++ code is often a bit cumbersome.

[saket]

Agreed with the NDK part, but choosing the version of ICU is your job. Android will do it automatically, right? Simply use the Locale APIs? Please correct if I'm wrong.

[kneth]

Using the Locale API implies that you will have to copy data (strings) between the JVM and our storage engine. It is definitely not acceptable from a performance perspective. Supporting ICU is important and we will get around the feature at some point.

[saket]

Ah, understood. Would you suggest any workaround in the meanwhile, other than iterating a table content manually?

[kneth]

If you are outside the range 0-591 (UTF-8), there is no easy workaround.

[cmelchior]

For reference: realm/realm-core#1946

If we go the route where the bindings are responsible for sorting, we should look into building ICU specifically for a given app. Given the size of the default ICU (16 MB - http://userguide.icu-project.org/packaging) it will be unfeasible for most apps to package that.

If we automatically detected which languages are supported by an app, I think we could make a smart build of ICU with only the required languages. This is most likely not trivial though.

Right now ICU is only official supported in Android N through https://developer.android.com/preview/features/icu4j-framework.html, however there is no native support yet. The earliest that would happen is in O, and even then it will take years for it to become widely available.

Alternatively we should look into providing these tables in an alternative manner, e.g. we could create a special dependency with only the lookup table and distribute that as a normal dependency. Realm could then dynamically load these if found.

[cmelchior]

Maybe there will soon be light at the end of the tunnel: android/ndk#548