Locale/language-sensitive case mapping ("Whats wrong with Turkey?")

[ozgur-yalcin]

Whats wrong with turkey ?
Please test the code below after reading the above blog.

assert_eq!("istanbul".to_uppercase(), "ISTANBUL");

Is it OK ? It will work fine, right ?
Don't be so sure!
If you care a whit about localization or internationalization, force your code to run under the Turkish locale as soon as reasonably possible.

Please add support localized case folding to Rust like other programming languages below:

JS case folding
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleLowerCase
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleUpperCase

GO case folding
https://golang.org/pkg/strings/#ToLowerSpecial
https://golang.org/pkg/strings/#ToUpperSpecial

JAVA localized case folding
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html#toLowerCase(java.util.Locale)
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html#toUpperCase(java.util.Locale)

.NET localized case folding
https://docs.microsoft.com/en-US/dotnet/api/system.string.tolower
https://docs.microsoft.com/en-US/dotnet/api/system.string.toupper

When it is supported, the code below should works like a charm !

assert_eq!("istanbul".to_uppercase("tr-TR"), "İSTANBUL");

or with a new function

assert_eq!("istanbul".to_localeuppercase("tr-TR"), "İSTANBUL");

[hbina]

Would this test be sufficient to indicate that this conversion is impelemented correctly? https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=63caf15374a3c580bbe9c5260522d102

Edit: I assume what needs to be changed is just the tables here?

rust/src/libcore/unicode/unicode_data.rs

Line 609 in f3fadf6

('\u{130}', ['i', '\u{307}', '\u{0}']), ('\u{132}', ['\u{133}', '\u{0}', '\u{0}']),

Edit 2: Hmmm according to the unicode FAQ here, it seems like they suggest against adding this support claiming that it will confuse the end user.
https://unicode.org/faq/casemap_charprop.html

[iago-lito]

Would a new method .to_locale_uppercase(locale) be both compliant with Unicode recommendation (because it leaves .to_uppercase() unchanged) and useful to explicitely support these specific capitalization cases?

Under this view, .to_uppercase() might need to explicitely become .to_ascii_uppercase().. which ends up not much different from .to_locale_uppercase("ascii") or something in the end :\

[ozgur-yalcin]

I don't know which function would be more appropriate but I can suggest a test like this when it is supported.

#[test]
fn test_turkish_case() {
    let lowercase = "abcçdefgğhıijklmnoöprsştuüvyz".chars();
    let mut uppercase = "ABCÇDEFGĞHIİJKLMNOÖPRSŞTUÜVYZ".chars();
    for lower in lowercase {
        let upper = uppercase.next().unwrap();
        assert_eq!(lower.to_string().to_locale_uppercase("tr-TR"), upper.to_string());
    }
}

[bluss]

Does this feature already exist in a crate in the ecosystem, and what shape of the API have they settled on? I can't see it in unic, but it might exist.

[canbakiskan]

@hbina It wouldn't suffice to just change those tables (In fact they are implemented correctly for non-conficting cases ı->I and İ->i). There needs to be specific lowercase/uppercase functions for TR locale (as @iago-lito suggested) since in Turkish "i" maps to uppercase "İ" and "I" maps to lowercase "ı" which conflicts with Latin versions (I->i and i->I). In my humble opinion, it would be better if it was part of the language rather than some crate; after all even emojis seem to get first-class support.

[workingjubilee]

There are many elements of full internationalization support, even just for handling "simple" cases, some of them quite important, that are not part of the Rust language. For instance, there is zero in-std support for many features that are widely considered part of "emojis", like emoji composition. These are instead part of external crates like unicode-segmentation, despite them being deeply important to how emoji support works and is understood by humans, instead of machines. Indeed, I would say we do not have truly "first-class" emoji support as long as that crate is external to std, but there are very good reasons why it is not going to be added.

You must remember, the str and char type of Rust function even in embedded contexts, and all of the named programming languages you are asking for full, package-free parity with require full operating system support, that I am aware of (except perhaps Go?). These interfaces are subjected to an extremely heavyweight stability promise that has a tendency to freeze their API in undesirable forms, when language is mutable: it would be pointless to add this only for it to be wrong 20 years later, in 2043. And yes, it is possible for a language's localization characteristics to change in so short a time. For this and more reasons, adding "locale-based" support to the language would be actually fairly unprecedented: we support some actually-quite-narrow bands of "UTF-8" and "Unicode Scalar Values", currently.

An issue request with an ecosystem package dedicated to handling this, like icu4x, is more appropriate. Granted, I believe it only barely existed at the time this issue was opened, but I believe it does handle locale-specific uppercasing for at least some languages. The API they have is what I will simply refer to as "extremely nontrivial", because this is an extremely nontrivial problem, though I believe for casemapping in particular it is actually a semi-separate crate?

[Dylan-DPC]

Closing this as it is better suited for a crate and there's nothing much we can do here to get this implemented in a clean way