String.downcase doesn't convert Greek letter 'Σ' correctly when this is at the end of a word

[vjorjo]

Environment

• Elixir & Erlang versions (elixir --version):
  Erlang/OTP 20 [erts-9.0] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10]
  Elixir 1.5.1
• Operating system:
  Windows 7

Current behavior

String.downcase("ΣΣ") returns "σσ"

Expected behavior

String.downcase("ΣΣ") should return "σς"
This is because "Σ"'s lower case is "σ" unless it is at the end of a word when then it becomes "ς".

[nox]

http://unicode.org/faq/casemap_charprop.html#12

[asonge]

I wanted to add some notes from some quick research for reference:

• This is the only language-neutral "conditional" special casing in Unicode, and is named Final_Sigma
• Conditional special casing is currently removed from SpecialCasing unicode.ex
• All of the other Unicode conditional special casing are locale-specific, so they can continue to be ignored, and because this is the only general conditional special casing to exist, maybe it can just be a special case in the code
• An example implementation with snippet of a reference text on Final_Sigma:

  These two entries are for the Greek capital letter sigma (Σ, U+03A3). This letter has two lowercase forms: one (ς, U+03C3) that appears only at the ends of words and another (σ, U+03C3) that appears at the beginning or in the middle of a word. The FINAL_SIGMA at the end of the first entry indicates that this mapping applies only at the end of a word (and when the letter isn’t the only letter in the word). The second entry, which maps to the initial/medial form of the lowercase sigma, is commented out (begins with a #) because this mapping is in the UnicodeData.txt file.

• Finally, from the Unicode page on Greek:
  http://unicode.org/faq/greek.html#5

  Finally, a detailed analysis of Greek corpora and the usage of final sigma and non-final sigma makes it clear that no simple positional context rule would cover all the cases. The rule is actually rather complex and has lots of exceptions, for abbreviations and other special cases.

Implementing this may be rather simple, but it might be useful to get feedback from a few Greek speakers or language users to really know if this should be implemented, and if it may be expected that the cited abbreviations are an actual concern or if it's more important to be right with the general final word case. [Edit/Update:] Java and Javascript do seem to implement the rule.

[voger]

From the top of my head:

• Without the general rule String.downcase/1 produces wrong results for Greek as many words end with ς.
• I tried in Microsoft Word (very popular business application) how does it handle two cases of abbreviation that I could think. It failed. Here are the two examples with the result from ms word:
  • ΑΘΑΝΑΣ. becomes αθανας. instead of αθανασ. which is understandable since it is impossible to differentiate if the dot ends the sentence or ends the abbreviation.
  • Σ' ΑΓΑΠΩ which means "I love you". This becomes ς' αγαπω instead of σ' αγαπω.

[vjorjo]

I tried python and rust. They both give the same results:

"ΑΘΑΝΑΣ." => "αθανας."  // wrong, it's an abbreviation, but no way to figure it out.
"ΑΘΑΝΑΣ. ΔΗΜΗΤΡΗΣ" => "αθανας. δημητρης" //wrong because "ΑΘΑΝΑΣ." is an abbr. here but there is no way to know it, unless you know Greek
"Σ' ΑΓΑΠΩ" => "σ' αγαπω"  // correct. "Σ'" could be nothing else but an abbreviation. 
"ΑΣ' ΤΟ" => "ας' το" // wrong because "ΑΣ'" is an abbr. here. However, no way to know it, unless you know Greek

but when you leave out any spaces between the abbreviated word and the next one they yield correct results. They probably interpret it as a way to denote an abbreviation:

"ΑΘΑΝΑΣ.ΔΗΜΗΤΡΗΣ" => "αθανασ.δημητρης"
"ΑΣ'ΤΟ" => "ασ'το"

[voger]

@vjorjo

"ΑΣ' ΤΟ" => "ας' το" // wrong because "ΑΣ'" is an abbr. here. However, no way to know it, unless you know Greek

Still it is behind an apostrophe and the apostrophe defines an abbreviation. I believe we may have a straightforward rule with this.

[vjorjo]

@voger

AFAIK you are correct. An apostrophe at the end of a word defines an abbreviation.
One exception I can think of is in geometry where e.g. one could say the line ΑΓ' or ΚΣ' but even then the correct lowercase is κσ'.

[filipemonteiroth]

@vjorjo

I just tested that piece of code on erlang (erl)

string:lowercase("ΣΣ"). and it returns σσ.

However that is weird, since on Erlang docs, it says it should convert correctly.

Check that out: http://erlang.org/doc/apps/stdlib/unicode_usage.html

[filipemonteiroth]

It looks we have some issue around that piece of code:

UnicodeData.txt

03A3;GREEK CAPITAL LETTER SIGMA;Lu;0;L;;;;;N;;;;03C3; (Line 923)

It will always convert 03A3 Σ to 03C3 σ

There's a SpecialCasing.txt file, however there's nothing implemented to treat that special case.

[josevalim]

Those cases need to be implemented manually as it cannot be done by programmatically parsing those files.

[filipemonteiroth]

@josevalim that make sense!