upper/lower

[Peter-Jacob]

how should we treat upper/lower in the context of Unicode? ASCII translates just the standard letters of the alphabet, not the special ones.
In Germany, we have ä/Ä, ö/Ö, ü/Ü, and ß (without an upper case version). In the Netherlands are different ones, also Scandinavia owes several ones. How do we treat them without knowing of other special letters in other areas of the world?

[rvjansen]

I see that IBM has a convenient table. https://www.ibm.com/docs/en/i/7.2?topic=tables-unicode-lowercase-uppercase-conversion-mapping-table But I would expect something in the C runtime to help us. Or maybe the iconv library. Best regards, René.

…

On 22 Oct 2021, at 17:31, Peter-Jacob ***@***.***> wrote:  how should we treat upper/lower in the context of Unicode? ASCII translates just the standard letters of the alphabet, not the special ones. In Germany, we have ä/Ä, ö/Ö, ü/Ü, and ß (without an upper case version). In the Netherlands are different ones, also Scandinavia owes several ones. How do we treat them without knowing of other special letters in other areas of the world? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[rvjansen]

http://www.unicode.org/Public/UNIDATA/CaseFolding.txt René.

…

On 22 Oct 2021, at 17:31, Peter-Jacob ***@***.***> wrote:  how should we treat upper/lower in the context of Unicode? ASCII translates just the standard letters of the alphabet, not the special ones. In Germany, we have ä/Ä, ö/Ö, ü/Ü, and ß (without an upper case version). In the Netherlands are different ones, also Scandinavia owes several ones. How do we treat them without knowing of other special letters in other areas of the world? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[adesutherland]

The c library I included - https://github.com/adesutherland/CREXX/blob/develop/utf8/utf.h - from https://github.com/sheredom/utf8.h has

// Make a codepoint lower case if possible.
utf8_weak utf8_int32_t utf8lwrcodepoint(utf8_int32_t cp);

// Make a codepoint upper case if possible.
utf8_weak utf8_int32_t utf8uprcodepoint(utf8_int32_t cp);

However, it is not a complete implementation, see its readme here https://github.com/sheredom/utf8.h#codepoint-case

But good enough for the moment?

[adesutherland]

So maybe assembler instruction like upper reg and lower reg?

[Peter-Jacob]

great! An assembler instruction should be easy enough, thanks!

[Peter-Jacob]

LOWCHAR/UPCHAR instructions run fine.
I wonder now, do we really need to translate character by character, or wouldn't it be better to translate the entire string? The REXX lower/upper functions treat the entire string.
In UTF8.h is a lower/upper translation of a string that does it in one go.

Any suggestion?

If we go for the string version, the instructions would be STRLOW/STRUP

[Peter-Jacob]

I discussed it with my alter ego we came to the conclusion we implement strupper and strlower as instructions. The modules are committed. I'll place the test-rexxes later in the day/tomorrow

[rvjansen]

I fully agree with your alter ego.

…

On 23 Oct 2021, at 12:39, Peter-Jacob ***@***.***> wrote: I discussed it with my alter ego we came to the conclusion we implement strupper and strlower as instructions. The modules are committed. I'll place the test-rexxes later in the day/tomorrow — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#162 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3WJQJVYLZ2BV6KZJ6SV7LUIKGEJANCNFSM5GQ2JDQA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[adesutherland]

Peter - there is one hazard you may not have considered

https://stackoverflow.com/questions/14792841/are-uppercase-utf8-characters-always-the-same-number-of-bytes-as-their-lowercase

So you just need to make sure value.string_length is right after conversion.

Sorry if you have already done this - I just thought!

[Peter-Jacob]

Thank you, I read the discussion and got the point. I am not sure what to do:
1/ if the length is not equal return the untranslated string
2/ character by character translation and leave a character if it changes length
But even when the length is equal, it isn't a 100% guarantee that something unwanted happens. You could have a one to two translation, and vice versa a 2 to 1 translation - length remains equal.
For the moment I will introduce a length check and return 1/. I'll also add a todo in the code to reinspect the code in a later stage.

[adesutherland]

Peter, I hope you don't mind, I took out the length check as it seemed pointless.

The todo remains.

I am thinking we will need to write our own version of the utf.h string upper functions you use to make sure the destination buffer is big enough and sets the destination length attributes. When we get to it ...

Meantime - I suspect it will work for all our tests (famous last words!)

[Peter-Jacob]

this is perfectly fine, I found the length check weird too.