Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEVTOOLS: allow correcting filenames when punycoding #3521



Copy link

@mistydemeo mistydemeo commented Nov 11, 2021

macOS has trouble with encodings for non-MacRoman HFS filesystems, which means the mac command's punycode option generates gibberish when given languages like Japanese, etc. I wanted a good solution and ended up landing on this. It requires the user to specify the source encoding on the disc so that it can determine how to reinterpret the text. The reason for that is a little annoying...

When not able to figure out the correct encoding for legacy non-Unicode HFS filesystems, which is most of the time, macOS interprets filenames as though they're MacRoman. Once mounted, the files are presented via all of the macOS filesystem APIs as though they're UTF-8. This is great for Western European languages, but falls over for other languages. For example, Japanese filenames will be rendered as gibberish (mojibake). This can be fixed by normalizing the filenames' UTF-8 encoding, transforming it back to "MacRoman", then correctly reinterpreting via the correct encoding.

This needs some testing on non-Mac computers probably to make sure I didn't break the existing punycoding workflow, though it should just do the same thing as it always did if the source encoding isn't specified - and it can only get set in the mac command.

@aquadran aquadran merged commit 1e82f07 into scummvm:master Nov 12, 2021
8 checks passed
@mistydemeo mistydemeo deleted the companion_mac_support_punycoding branch Nov 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
2 participants