Add tests to reverse-string to teach correct string handling

[tobega]

Currently the tests for reverse string only contain strings with 7-bit ASCII. This is teaching people the wrong way to handle strings as you can just treat it as an array of bytes. Incorrect handling of non-ASCII characters (or surrogate pairs if you use something like Java) is one of the biggest sources of bugs in the industry.

I propose adding at least a test with a multibyte UTF-8 character as is common in most European languages, e.g. "skåp" which becomes "påks" (this would make a difference in the Julia solutions, for example)

For even better benefit (for UTF-16 languages), it could be good to have a string containing a surrogate pair, e.g. "\uD834\uDD1E", a G-clef, which should come out as it was.

That is probably enough, and it might be too difficult to also handle combining marks, e.g. "as⃝df̅" should become "f̅ds⃝a", not "̅fd⃝sa" (from rosettacode.org)

[SaschaMann]

Could you post a screenshot of the last example, please? I'm not sure if it's rendered correctly for me:

[siebenschlaefer]

I agree that students should be aware of encodings.

But as you wrote reversing unicode strings is hard because you have to be aware of surrogate pairs, composite characters, combining marks, and things like the BOM or direction marks.
Also, not all programming languages have builtin facilities to handle Unicode, e.g. the standard strings in C and C++ are just sequences of bytes. They would require external libraries.

Currently the exercise is pretty easy, and AFAIK comes early in the core track of many languages. Changing it would make it much much harder, and IMHO make it a different exercise.

[iHiD]

@tobega I would suggest creating a new exercise that centres around working with unicode characters. This would make a great "Concept Exercise" for Exercism version 3. If you let me know which language(s) you are proficient in I can point you in the right place to create that :)

[glennj]

The micro-blog exercise deals with handling unicode characters, so there's a bit of overlap with this suggestion.

[tobega]

Maybe "reverse string" should then be remade into "reverse array"?

…

On Tue, 26 May 2020 at 17:25, Glenn Jackman ***@***.***> wrote: The micro-blog <https://github.com/exercism/problem-specifications/blob/master/exercises/micro-blog/description.md> exercise deals with handling unicode characters, so there's a bit of overlap with this suggestion. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1661 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGACJ52FYRRYXKVX77YE2KTRTPNNFANCNFSM4NKJGBUA> .

[tobega]

The rendering is correct, the ring is attached to the 's' and covers the letter after the 's'

…

On Tue, 26 May 2020 at 16:17, Sascha Mann ***@***.***> wrote: Could you post a screenshot of the last example, please? I'm not sure if it's rendered correctly for me: [image: grafik] <https://user-images.githubusercontent.com/20866761/82911543-4d46ee80-9f6c-11ea-9b85-f68020576a47.png> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1661 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGACJ5ZA6H2ICIQSSAJOJV3RTPFOXANCNFSM4NKJGBUA> .

[tobega]

After considering the objections here, a good resolution to this would be to specify that the strings are 7-bit ASCII in the problem. It makes quite a difference in Julia, for example.

[iHiD]

This feels like it will be very different? In Ruby, for example, Strings are arrays of chars, not bytes, so the test would be a noop (so fine adding it, but won't do a huge amount either way):

"skåp".reverse
 => "påks" 

"skåp".chars 
 => ["s", "k", "å", "p"] 

For some languages, it will be make the exercise much harder, which would then (as @siebenschlaefer says) mean those tracks needed to move it around. This is the problem at the heart of #1560.

My suggestion would be to open this PR at exercism/julia and make the change locally in that repo for now, and we can readdress this in this repo once we've understood how we're going to handle situations like this.

[ErikSchierboom]

We've recently discussed the fact that we have fairly little unicode exercises. The reverse-string exercise could be a good candidate for that. It would increase the difficulty for tracks that don't have nice unicode support. My suggestion would therefore to add a new test case with a unicode input string, but add the unicode scenario to it to allow tracks to exclude this test case. Thoughts?

[siebenschlaefer]

I'd rather not do that. I'd prefer doing that in a different exercise.

Currently it's a pretty simple exercise, well suited for beginners.
Handling unicode properly would make it much more complicated (depending on language of course), would shift the focus from the reversal to unicode, and make it essentially a different exercise.

BTW: It's surprisingly hard to reverse a unicode string and cover all the cases, e.g. combining characters, ligatures, directions markers, BOM. (I forgot I already mentioned that.)

[kotp]

That said, we use this as the basis for the same exercise with the additional requirements. It means knowing how to reverse a string (ASCII) and applying the concerns that UTF-8 bring.

So I would vote for not changing this exercise, but using it as the base for the more complex exercise. "Reverse String Revisited".

It would mean not really losing out on the existing exercise, while being opt-in for tracks that have this as a concern.

[iHiD]

@siebenschlaefer Could you expand a bit more on not adding the test/scenario pair? Doing so would mean that you could opt out for your tracks if you wanted to so as not to make the exercise harder, but for other tracks who want to ensure that UTF8 is handled properly, they could add it relatively trivially. As this is sort of the point of the "scenarios", I want to understand more about why you don't feel this would work well.

[siebenschlaefer]

@iHiD: tobega is of course correct, strings that contain human language are more complicated than a sequence of bytes, I get the objection that we should not teach that simple view, and I agree that handling Unicode/UTF-8/UTF-16 strings incorrectly is far too common.

But "just" treating a string as a sequence of independent multi-byte UTF-8/UTF-16 code points is also not correct, and I think we should not teach that either. That would treat one "issue" and ignore too many others.

E.g. the "å" in "skåp" can be encoded as "'\u00e5" (LATIN SMALL LETTER A WITH RING ABOVE) or decomposed as "\u0061\u030a" (LATIN SMALL LETTER A followed by COMBINING RING ABOVE). Only handling the former correctly is IMHO similar to the current situation where we only handle single bytes correctly.

Yes, we could support some subset of western languages where characters are encoded as single code points, e.g. by specifying that the input will be "canonically combined". But do we really want to follow the western tradition of ignoring the rest of the world? Many of our students come from cultures with more complicated scripts. And AFAIK there are quite a few graphemes without a composed form.

Or what about bi-directional text? (AFAIK Hebrew is written from right to left but any decimal numbers are written from left to right.) Sometimes the direction is implicit, sometimes there are explicit directional embeddings.

Writing a full-fledged Unicode-aware reverse_string() function would IMHO be far too complex (although I would be thrilled to be proven wrong.) And anything less than that will have restrictions, e.g. no bidirectional text, no combining characters, or only ASCII-characters (which is basically what we have right now).

To be clear: I'm by no means an expert on Unicode. I find the complexity fascinating and each time I try to understand something I learn more about what I don't know. (E.g. country flags are encoded with two letters from the range U+1F1E6 ... U+1F1FF, reversing them can turn Spain into Sweden :-)

So IMHO these are the options:

1. Leave this exercise like it is and add a big disclaimer that it will only handle ASCII or 1-byte characters.
2. Expand this exercise so that it handles some additional characters, e.g. single code points possible encoded with multiply bytes. And add a big disclaimer about the limitations.
3. Change this exercise to reverse-array as suggested by @tobega

Personally I favor (1). This exercise is very easy to understand, even for novice programmers, everybody is familar with "text" and I guess almost all of us did try to write out names in reverse, right? It is pretty easy to solve but still offers the opportunity to practice writing idiomatic and efficient code. (And I've seen quite a few students who still struggle with it, e.g. trying to include string[string_length] in the result.
My second favorite would be (3). That avoids all the complexities of natural language and encodings, but IMHO it's more abstract than text and therefore a little bit less approachable.

@ErikSchierboom I like the idea of exercises targeting Unicode. But (1) I'm not sure whether reversing a string would be a good fit, and (2) I'd rather have a separate exercise than making this exercise much more complicated (as @kotp wrote).