Accents in open/save dialogs are displayed wrongly

[rvgr]

OS: MacOS 10.11.4
Version: 0.0.4

When saving or loading a game, if a file has accents in its name, it will be shown as the unaccentuated letter followed with a ?. However on the disk, accents are fine.

For example, "é" becomes "e?", "à" becomes "a?".

• Reproducable in RCT2 (vanilla)?
• Multiplayer?

Screenshots / Video:

« Frontie?res Forestie?res » should read as « Frontières Forestières ».

It's fine in the Finder.

[IntelOrca]

I can't reproduce this on Windows, so possible OSX / Posix only?

[HaasJona]

AFAIK osx saves the accents in the file system as Unicode combining characters. So it's first an e character and then a combining ` character for example.

We probably need to canonicalize the filenames first before displaying them. Or alternatively support Unicode properly, probably not possible with the bitmap fonts.

[IntelOrca]

@HaasJona no, it works fine in Windows with accents - the problem is the conversion from the OS encoding to UTF-8, in platform_enumerate_files_begin. I don't know whether this differs or not between Linux and OSX.

[HaasJona]

I know it works fine in Windows with accents. But that is because windows doesn't use combining characters. In Windows the é is a single character, on osx, it is 2 characters.

[IntelOrca]

@HaasJona The problem lies in posix.c:platform_enumerate_files_begin... the code is not correctly converting the string from whatever the OS stores it as to UTF-8. Windows is fine because I correctly convert it from widechar to UTF-8.

[HaasJona]

Well, it works fine in Linux, and the fact that the e is displayed correctly makes me 90% sure that I'm correct. You could test it by switching to a locale with ttf fonts, then the ` character should be there if I'm right.

[IntelOrca]

If it works in Linux, then OSX needs some extra code to convert to UTF-8 just like Windows has. This has to happen in the platform_enumerate_files_begin function.

[HaasJona]

You are certainly correct, but what I'm saying is that I believe that OSX already returns the file name in UTF-8, just that its "weird" UTF-8, because for example ç isn't encoded as 0xC48D (which would be the usual UTF-8 encoding as expected by the game), but as 0x63CC8C which would be a normal c (0x63) and a combining caron (0xCC8C). But both are technically correct UTF-8 representation, It's probably just an issue with the game that there is no sprite for the combining character (and a generally missing functionality of overlaying multiple sprites for a single character), so we either need to handle all UTF-8 correctly or we need to "recombine" (normalize) the file names the OS give us, by "translating" 0x63CC8C back to 0xC48D. There are probably library functions for that, but I only know the API for Java and not C/C++.

[IntelOrca]

The only function that should be changed is platform_enumerate_files_begin so that the filename is correctly converted into the encoding the game expects. The bitmap sprite will then be drawn correctly, as for TTF - that will depend on how SDL_TTF works on OSX.

[HaasJona]

The bitmap sprite will then be drawn correctly

I really doubt that the current code bitmap is able to draw combining characters correctly.

[IntelOrca]

@HaasJona It will draw correctly because the filename will have been converted to the correct UTF-8 encoding the game understands where an é will be one single codepoint that maps to a sprite.

[HaasJona]

@IntelOrca OK. What I mean is that the representation as 2 codepoints is technically just as correct as the normal 1 codepoint representation. You may have seen text like ̷ͮt߳̀̽͘ͅh̸͓͓͉̦́͌̑᷄҆߱í̱̝̻̉͒͡s͚̓̀ which is just normal UTF-8 with random combining characters, too. So there is nothing wrong or incorrect about that.

Normalizing the file names OSX gives us in platform_enumerate_files_begin will solve the problem and is probably an okay solution. Strings like ̷ͮt߳̀̽͘ͅh̸͓͓͉̦́͌̑᷄҆߱í̱̝̻̉͒͡s͚̓̀ will obviously still render incorrectly in game because they aren't normalizable, but that's probably fine for now.

[Gymnasiast]

This needs the same treatment as loading the RCT2 files for a folder with accents, whcih @rwjuk fixed.

[Gymnasiast]

@rwjuk I haven't yet been able to check whether this bug has been solved by your earlier work or not. If it's already fixed then this obviously can be closed.
When this is closed, don't forget the changelog entry.

[rwjuk]

I will have a look - if not, it'll almost certainly just be adding in appropriate calls to the precomposition function for macOS.