Get compiled script directory via sys.executable returns 8.3 filenames (DOS SFN)

[Neo23x0]

I try to get the current directory name of the compiled script by the use of the following statement:
application_path = os.path.dirname(os.path.realpath(sys.executable))

When I run the python script it returns:
_C:\Users\neo\Downloads\loki-master_

When I run the compiled executable, it returns:
_C:\Users\neo\DOWNLO1\LOKI-M1_

This is the old MS DOS 8.3 filename style called SFN. I use the application path to exclude it from a directory walk later on but the path returned by os.walk() is obviously different than the one returned by sys.executable.

Is this a bug? Is there a way to retrieve the executable's full path?
I want to allow users to run the executable from other working directories.

[Neo23x0]

I fixed the problem myself by the use of win32api:

dos_path = os.path.dirname(os.path.realpath(sys.executable))
application_path = win32api.GetLongPathName(dos_path)

[tallforasmurf]

Could this be added to the doc? Where?

[codewarrior0]

Yes, this should be documented as a difference between running in normal Python and running under PyInstaller. PYI explicitly uses GetShortPathName to set the value of sys.executable with an 8.3 format path as its directory component - "to overcome the Python and PyInstaller limitation to run with foreign characters in directory names." (pyi_path.c:157)

In normal Python, sys.executable contains the full path to argv[0]. On Windows, argv[0] is encoded using an MS-DOS codepage. Renaming python.exe to something like pyxěščřžýáíé.exe (or putting it in a folder with that name) will result in a worthless sys.executable and the only way to get a good one is with the Win32 API GetCommandLineW for getting the command line as wide characters (i.e. UTF-16 encoded).

However, there is a bug(?) in pywin32 where there is no way to call GetCommandLineW. Calling win32api.GetCommandLine always calls the ANSI variant which returns the program path encoded with a codepage - the same thing you get from sys.executable in normal Python. So you actually have to go through ctypes to call GetCommandLineW.

I want to be able to change sys.executable to always return a usable filename, but this isn't possible on Windows with Python 2.7. sys.executable is required to be a bytes type, so we can't just change it to a unicode. Also, python's filesystem functions will use the A variants of Windows APIs - which expect a codepage encoding (mbcs) whenever they are passed a bytes type, so we can't change sys.executable to be UTF-16 encoded either.

(On Python 3.3, sys.executable is a unicode type, but I didn't check if it actually contains valid data for the odd filename above.)

It looks like the current behavior of using an MS-DOS 8.3 ShortFileName is the best we can do for now. It has one shortcoming though. The path to the .exe's folder is encoded as an SFN, but the filename of the .exe itself is not - it's codepage-encoded as it comes from argv.

[codewarrior0]

I kind of want to go through all of the bootloader code and add hungarian-notation to all of the string variables to specify the encoding - workpath_u8, argv_mbcs and so on.

[htgoebel]

I'd prefer to have this fixed in pyi_path.c. But as I read your explanation I have the impression that we can't get this fixed as long as we are supporting Python 2.x Is this correct?

[codewarrior0]

we can't get this fixed as long as we are supporting Python 2.x

Yes. sys.executable is broken, for most non-ASCII characters, on Python 2.x for Windows, even for vanilla Python without PyInstaller.

PyI's workaround of using a ShortFileName instead of encoding with mbcs improves support over vanilla Python's, but raises this issue of not being able to compare filenames easily. It is also not available for some filesystems, especially SMB network shares. This difference is responsible for the paths I observed here and may be responsible for that issue in some way also, as he notes that the issue only happens on D:\ and not C:. It also needs to be documented as it deviates from vanilla Python's behavior.

[codewarrior0]

Besides, shouldn't you be using sys._MEIPASS instead of dirname(sys.executable) (or __file__ etc) to find data-files when your app is intended to be frozen?

I wonder what the encoding for _MEIPASS is. It's probably UTF-8, but if it's a bytes type on python2 it will run into the snag of the Windows filesystem functions using the ANSI variants and trying to interpret it as mbcs encoded. Maybe _MEIPASS could be changed to a unicode type?

[Neo23x0]

That's what my code looks like. (still clumsy but working)

        if getattr(sys, 'frozen', False):
            application_path = os.path.dirname(sys.executable)
        elif __file__:
            application_path = os.path.dirname(__file__)
        # unknown if still necessary 
        if application_path == "":
            application_path = os.path.dirname(os.path.realpath(__file__))
        if "~" in application_path:
            application_path = win32api.GetLongPathName(application_path)

[htgoebel]

As codewarrior wrote, you must use _MEIPASS, see https://pythonhosted.org/PyInstaller/#adapting-to-being-frozen

[Neo23x0]

Doesn't work for me. I build my file in one file mode (-F). Using _MAIPASS I get the path of the extracted archive contents but not the path where the executable resides.

C:\Users\neo\AppData\Local\Temp

[codewarrior0]

Oh, you're actually using the app by putting it in a folder where it is going to work. Then what you're doing with GetLongPathName will do what you want, but with the caveat that it won't work on filesystems that don't support 8.3 filenames - you'd have to go directly to GetCommandLineW (as I noted a few comments ago, or perhaps GetModuleFileNameW?) if you need that to work.