Korean (unicode ?) signs in path makes tkinter fail when

[deajan]

Hello Kay,

I am progressively deploying a nuitka compiled app on 2500 computers, and had some fun with a Korean one, where the username has korean signs in it.
I've tried to reproduce that problem on my dev computer, and got to the conclusion that the app will fail with a tkinter library path error when the path contains Korean signs.
Of course, running the interpreted program in the same path works.

Could you perhaps have a look ?

• Nuitka version, full Python version and Platform (Windows, OSX, Linux ...)

0.6.7
Python: 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 18 2019, 23:46:00) [MSC v.1916 32 bit (Intel)]
Executable: c:\Python37-32\python.exe
OS: Windows
Arch: x86

• How did you install Nuitka and Python (pip, anaconda, deb, rpm, from source,
  what is a virtualenv ...), this is very important usually.
  Installed with python -m pip install nuitka

My SSCEE:

The minimal source code:

#! /usr/bin/env python
#  -*- coding: utf-8 -*-

import tkinter

print('hello world')

I've compiled the program with

python -m nuitka --enable-plugin=tk-inter --standalone korean_tkinter.py

I've then placed both the compiled and the interpreted version in the same folder:

C:\TEST\会社案\삼성\korean_tkinter.py
C:\TEST\会社案\삼성\korean_tkinter.dist

When I run the interpreted code I get:

When I run the compiled version I get:

Of course, running from the GUI without a cmd prompt triggers the same result.

I've then proceeded to create a second SSCEE:

#! /usr/bin/env python
#  -*- coding: utf-8 -*-

import sys
import os

os.environ['TCL_LIBRARY'] = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tcl')))
os.environ['TK_LIBRARY'] = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tk')))

tcl_path = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tcl')))
print('is_dir=%s' % os.path.isdir(tcl_path))
print('tcl_path=%s' % tcl_path)
print('os_env=%s' % os.environ.get('TCL_LIBRARY, False'))

import tkinter

print('hello world')

Interpreted run:

Compiled run:

Of course, the TCL library is present

Since the LoadLibraryEx function is C related, I think this might not a python related problem.
@JorjMcKie could you please confirm this problem ? The korean signs can just be copy pasted from the browser.

[Update]
I managed to push investigations in a third SSCEE:
Copied the tcl and tk directories directly into the source directory so they would be present for interpreted source below:

#! /usr/bin/env python
#  -*- coding: utf-8 -*-

import sys
import os

os.environ['TCL_LIBRARY'] = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tcl')))
os.environ['TK_LIBRARY'] = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tk')))

tcl_path = os.path.normpath(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), 'tcl')))

print('is_dir=%s' % os.path.isdir(tcl_path))
print('tcl_path=%s' % tcl_path)
print('os_env=%s' % os.environ.get('TCL_LIBRARY', False))

import PySimpleGUI as sg

sg.Popup('hello world')

Compiled or not, this one fails with:

C:\Python37-64\python.exe C:/TEST/会社案/삼성/korean_tkinter3.py
is_dir=True
tcl_path=C:\TEST\会社案\삼성\tcl
os_env=C:\TEST\会社案\삼성\tcl
Traceback (most recent call last):
  File "C:/TEST/会社案/삼성/korean_tkinter3.py", line 18, in <module>
    sg.Popup('hello world')
  File "C:\TEST\会社案\삼성\PySimpleGUI\PySimpleGUI.py", line 11453, in Popup
    button, values = window.Read()
  File "C:\TEST\会社案\삼성\PySimpleGUI\PySimpleGUI.py", line 5768, in Read
    self._Show()
  File "C:\TEST\会社案\삼성\PySimpleGUI\PySimpleGUI.py", line 5637, in _Show
    StartupTK(self)
  File "C:\TEST\会社案\삼성\PySimpleGUI\PySimpleGUI.py", line 9351, in StartupTK
    Window.hidden_master_root = tk.Tk()
  File "C:\Python37-64\lib\tkinter\__init__.py", line 2023, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    {C:\TEST\???\??\tcl} C:/TEST/???/??/tcl8.6 C:/Python37-64/lib/tcl8.6 C:/lib/tcl8.6 C:/lib/tcl8.6 C:/library C:/library C:/tcl8.6.9/library C:/tcl8.6.9/library



This probably means that Tcl wasn't installed properly.

[/UPDATE]

[kayhayen]

My first thought was that maybe this is not UTF-8, some Asian encodings are not that, and that could be the issue. Your last program seems to suggest that is not the case, at least you were able to store it in a UTF-8 encoded script, is it?

Something that we use on Python2, which is holy incapable of working with Unicode paths is the short path name. There is code in Nuitka which does shorten a path. I am pretty confident that would help.

def getWindowsShortPathName(filename):
    """ Gets the short path name of a given long path.

    Args:
        filename - long Windows filename
    Returns:
        Path that is a short filename pointing at the same file.
    Notes:
        Originally from http://stackoverflow.com/a/23598461/200291
    """
    import ctypes.wintypes

    GetShortPathNameW = ctypes.windll.kernel32.GetShortPathNameW
    GetShortPathNameW.argtypes = [
        ctypes.wintypes.LPCWSTR,
        ctypes.wintypes.LPWSTR,
        ctypes.wintypes.DWORD,
    ]
    GetShortPathNameW.restype = ctypes.wintypes.DWORD

    output_buf_size = 0
    while True:
        output_buf = ctypes.create_unicode_buffer(output_buf_size)
        needed = GetShortPathNameW(filename, output_buf, output_buf_size)

        if needed == 0:
            # Windows only code, pylint: disable=I0021,undefined-variable

            raise WindowsError(
                ctypes.GetLastError(), ctypes.FormatError(ctypes.GetLastError())
            )

        if output_buf_size >= needed:
            return output_buf.value
        else:
            output_buf_size = needed

[kayhayen]

Not sure how well that code runs on Python3 though, since Nuitka does it only to hand file names to Scons. It used to be highly problematic to be doing that, since there really is no good way to deal with unicode paths in Python2. But ever since I made that, reports from Asians were gone.

Since TkInter gets its path from the short path binary directory name (this is happening in the C code of Nuitka) into the environment variable, it ought to work with Python2.

Now that I realize that "Tcl" easily is not Unicode aware enough, even in Python3, then it makes sense to apply this or some other form of shortening to it.

Today I wanted to check out your gift, while also day jobbing soon. But I kind of promise you to look into it. My suspect is that using that function in the module preload code of Nuitka in the TkInter will make it work for sure. My only uncertainty is if the ctypes usage above is all that correct. Sometimes it turned out the declarations were not portable.

Question however is, if Nuitka ought to not go for the short path more globally, although this can cause problems on a network share. I didn't get any such reports yet, but compiling with Nuitka or running programs from there, with Python2, might give errors. I am not sure how much of a "use short path if possible" we do, or if we crash if it fails.

All in all, no big thing, and thanks for breaking this down @deajan .

[kayhayen]

So this is what I was talking about:

#if PYTHON_VERSION < 300
    char *binary_directory = getBinaryDirectoryHostEncoded();
    NUITKA_PRINTF_TRACE("Binary dir is %s\n", binary_directory);

    Py_SetPythonHome(binary_directory);
#else
    wchar_t *binary_directory = getBinaryDirectoryWideChars();
    NUITKA_PRINTF_TRACE("Binary dir is %S\n", binary_directory);

    Py_SetPythonHome(binary_directory);

#endif

[kayhayen]

Behind host encoded, there is also a short path usage.

[kayhayen]

And there is also this:

// On Python3, this must be a unicode object, it cannot be on Python2,
// there e.g. code objects expect Python2 strings.
#if PYTHON_VERSION >= 300
#if defined(_WIN32)
    binary_directory = PyUnicode_FromWideChar(getBinaryDirectoryWideChars(), -1);
#else
        binary_directory = PyUnicode_DecodeFSDefault(getBinaryDirectoryHostEncoded());
#endif
#else
    binary_directory = PyString_FromString(getBinaryDirectoryHostEncoded());
#endif

[kayhayen]

This is code that makes excplitely no use of that shorted path for Win32 for Python3. We would have to checkout these places, and try to go with short paths in what becomes path sys.path or the directory used for the environment variables only.

This ought to be easy. As I said, I will look into this more on Saturday if I can.

[deajan]

Can do tests on rc versions whenever you request.

[kayhayen]

I am currently stuck with my Python3, when I am compiling in this kind of sub folder, not copying the Python3 DLLs, which is strange.

[kayhayen]

So, the not copying of the Python3 DLLs is due to a broken conditional clause that probably only works for my setup.

As for the TCL, I think I got that going. However, it seems that strangely, compiling and then running in the Korean path does not work, but compiling above, moving to inside, then works, I will want to try out what that is about too. Hang on.

[kayhayen]

Funny diff it is:

diff -r 会社案삼성/TkInterUsing.dist.working 会社案삼성/TkInterUsing.dist
Only in 会社案삼성/TkInterUsing.dist.working: comctl32.dll
Only in 会社案삼성/TkInterUsing.dist.working: libcrypto-1_1-x64.dll
Only in 会社案삼성/TkInterUsing.dist.working: libssl-1_1-x64.dll
Only in 会社案삼성/TkInterUsing.dist.working: sqlite3.dll
Only in 会社案삼성/TkInterUsing.dist.working: tcl86t.dll
Only in 会社案삼성/TkInterUsing.dist.working: tk86t.dll
Binary files 会社案삼성/TkInterUsing.dist.working/TkInterUsing.exe and 会社案삼성/TkInterUsing.dist/TkInterUsing.exe differ
Only in 会社案삼성/TkInterUsing.dist.working: vcruntime140.dll

[kayhayen]

Ok, got that, dependency walker is to blame here, it doesn't like these kinds of paths, or my parsing is too stupid, anyway, with short paths, it works better. Merely adding cwd = getExternalUsePath(os.getcwd()) will muc better.

I will put a test version out soon and let you know.

[kayhayen]

Please check this out https://nuitka.net/doc/factory.html

This has 3 changes. First, the loading of extension modules uses the short path of the binary directory. Second the plugins get handed the binary directory path in short path form as well. And third, before dependency walker is executed, the current directory is switched to a short path as well.

As I previously mentioned, I am concerned about UNC paths and the error handling in those cases. It might not be possible to shorten there, but I don't know that for sure. And if it's not possible, we ought to not crash at least. I didn't test that yet, but I will. I think I only very recently added error handling, when I found some new use of this file name shortening.

[deajan]

I've tried with latest 0.6.8rc2 factory installed an hour ago with pip from factory.zip
I recompiled first korean_tkinter.py source file and here's what I got:

Any additionnal info I can provide ?

[kayhayen]

You didn't use the tk-inter plugin.

[deajan]

Indeed... doing tests at 1am isn't always productive ;)

I tried with the plugin, but it doesn't make any difference:

The SSCEE korean_tkinter2.py gives the same results.

[kayhayen]

Compiling inside the Korean path still led to unusable results, but compiling outside of it, and running inside did. I pushed the change to factory in the mean time, but currently it has corruptions due to lost references, so it's not usable, but I think that explains it. I will probably fix the corruptions later today and make a pre-release out of it.

[kayhayen]

This is now on develop, and going to be part of the next release. You ought to be able to compile and/or execute in unicode paths as you wish.

[deajan]

I've tried 0.6.8rc4 with both mingw and msvc... Works like a charm.
The only glitch is that it outputs some unicode char while compilation ;)

Thank you for your time.

[JorjMcKie]

The only glitch is that it outputs some unicode char while compilation ;)

Setting the right codepage (chcp) would probably solve that one too 😉

[deajan]

@JorjMcKie I guess so, but this behavior has changed from Nuitka 0.6.7 which hadn't that kind of output.

[kayhayen]

These are supposed to be terminal color codes. Warnings are red. I thought this worked on Windows too, but maybe it does not. Maybe only in my git bash, but not cmd, I will have to try that out.

[kayhayen]

https://stackoverflow.com/questions/16755142/how-to-make-win32-console-recognize-ansi-vt100-escape-sequences

[kayhayen]

Seems this needs an API call specific to Windows, or a bug using os.system('') that will do it for us, probably something I can google code for.

[kayhayen]

The trick with os.system and empty command works, because cmd.exe modifies the terminal for itself, and doesn't undo it afterwards, and we can delay it until it's actually necessary. How it then happens to be that the interactive cmd.exe on the outside doesn't do it, I have no clue, but it works, expect it to be in a next pre-release, I am fixing up develop issues right now anyway.

[kayhayen]

The release 0.6.8 was just made and contains the correction.