TokenTool won't run if install path has accented characters

[Phergus]

Describe the bug
TokenTool won't run after installing to a directory with accented characters in the name.

Known to be an issue are these from the Polish language: ą ę ś ż ź ó ł ń ć

To Reproduce
Steps to reproduce the behavior:

1. Install TokenTool to a path with one or more of the characters above in the path.
2. TokenTool won't launch.

Expected behavior
Expect it to handle any valid Windows path.

TokenTool Info

• Version: 2.1
• Install: New

Desktop (please complete the following information):

• OS: Windows
• Version: 10

Additional context
Reported by wyelky-jaahr on Discord.

[Azhrei]

Tested the latest TT on MacOS Mojave (10.14.6) with multiple different accented characters and there's no failure; the splash screen appears and the applications works as expected.

I used the directory name

El Niño é ø and º

which contains a variety of characters and symbols. Dragging an image from that folder into TT worked, as did the File > Save As option to store the image back into that directory.

It sounds like there may be something in the Windows implementation. Note that Windows doesn't have a system-wide encoding setting, while Unix applications typically use the value of environment variable LANG and/or LC_ALL for both locale and encoding info. Perhaps the JVM simply needs the proper options passed to it...?

[xdy]

I just verified that it's a problem on windows (10).
Installed to "C:\Users[user]\AppData\Local\TokenToolåäö\TokenTool.exe" and TokenTool wouldn't start.
Uninstalled and reinstalled at "C:\Users[user]\AppData\Local\TokenTool\TokenTool.exe", which worked.
I can't see anything interesting in windows Event Viewer.

[xdy]

A bit of unstructured playing around later I don't think it even gets into the TokenTool java code.
No logs were produced when the exe is in a directory with diacritics (e.g. TokenTool.åäö), so I added a FileUtils.writeStringToFile as the first line in TokenTool.init, which wasn't written either. (But both logs and the temp file were created when the exe is in a non-diacritic-containing directory.)
Something related to the installer, or javafx itself perhaps? (I used the latest wix and innosetup, so upgrading those won't help. Not about to try downgrades.)
The TokenTool.exe doesn't seem to be a .net assembly meaning dotPeek was no help, so I can't see what's going on in there. (My assembler-reading days are long gone. :)
And everything works fine when running from intellij in a directory that contains diacritics so debugging won't help.
Meaning that I'm out of ideas. 🤷‍♂
(Other than hacking and rebuilding javafx so it pauses enabling me to connect a debugger, but, um, nope. :)

[Azhrei]

That’s some good detective work. 👍

Java has had similar bugs in the past. It wouldn’t surprise me if this is another one. It may even be logged already. For now, we can document it while investigating. And maybe the installer can put up a message in big letters about this issue!

[Azhrei]

Perhaps this bug? https://bugs.openjdk.java.net/browse/JDK-8195129

[xdy]

Yeah, that one sounds likely, not as if we can avoid having System.load() getting called. :)
To no one's surprise, MapTool has the same problem.

Somewhat to my surprise, pcgen (which uses launch4j to make installers) doesn't have a problem with starting from folders containing diacritics.

Might be worth looking into launch4j (for tokentool and maptool both I guess), unless this has been rejected already? (Pcgen is on jdk 11 and uses a mix of swing and javafx, so reasonably similar tech stacks.)

[Azhrei]

Hm. Not that bug, then; System.load() is unavoidable in any JVM, so if PCGen runs, this is not the bug we're looking for. More likely it has to do with a JVM option, such as encoding not being set properly...

Can you try this solution and see if it helps? You can put the env var into your settings and launch TT or MT from the GUI (changes to env vars won't affect command line windows that were already open). https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding#362006

[xdy]

No, that didn't work. I tried setting both JAVA_TOOL_OPTIONS and JAVA_OPTIONS to -Dfile.encoding=UTF8, but got the same behaviour.
Update: Tried with UTF-8 and utf8 too, but that made no difference.

[Phergus]

Both TokenTool and MapTool have executables that read a config file from the app dir. In it is this line:

app.runtime=$APPDIR\runtime

I wonder if it is the expansion of $APPDIR that isn't handling the path correctly?

[Phergus]

This is fixed in 2.2.0.