How to enable UTF-8 mode for script

[jerray]

I'm using pyinstaller on Windows 10, language is Simplified Chinese. When I generate exe for my script using pyinstaller,
the preferred encoding is always cp936. After some searches, I failed to find a way to enable UTF-8 mode.

Here's an example script main.py:

import locale

print("locale: ", locale.getlocale())
print("default locale: ",  locale.getdefaultlocale())
print("preferred encoding: ", locale.getpreferredencoding())

Run it using Python interpreter without any other options:

> python main.py
locale:  ('Chinese (Simplified)_China', '936')
default locale:  ('zh_CN', 'cp936')
preferred encoding:  cp936

Now enable UTF-8 mode by set PYTHONUTF8 to 1:

> $Env:PYTHONUTF8=1
> python main.py
locale:  ('Chinese (Simplified)_China', '936')
default locale:  ('zh_CN', 'cp936')
preferred encoding:  UTF-8

But after packaging it to an executable, preferred encoding is always cp936 no matter if PYTHONUTF8 is 1.

> $Env:PYTHONUTF8=0
> .\main.exe
locale:  ('Chinese (Simplified)_China', '936')
default locale:  ('zh_CN', 'cp936')
preferred encoding:  cp936

> $Env:PYTHONUTF8=1
> .\main.exe
locale:  ('Chinese (Simplified)_China', '936')
default locale:  ('zh_CN', 'cp936')
preferred encoding:  cp936

In case I didn't miss something, I reached here to ask if there's a way to enable UTF-8 mode.

[rokm]

PYTHONUTF8 support has been added in #6065. So you would need to use either the tip of the v4 branch or the develop branch. And you'd also need to rebuild the bootloaders yourself.

EDIT: actually, it looks like we haven't back-ported #6065 to v4 branch; so checking out develop and rebuilding the bootloaders is the only option.

[diego-hermida]

Hello,

I'm experiencing the same issue with Python UTF-8 mode and PyInstaller.

Following your instructions, I am able to overcome the limitation by building bootloaders/PyInstaller from source (develop branch). However, this is not ideal, as UTF-8 mode is standard since Python 3.7 and the library releases should be compatible with it.

Is there any plan to integrate this feature into the main v4 branch?

Thanks.

[rokm]

I think the plan is for the next release to be 5.0 (so develop branch will become "main" branch again), but I suppose we could back-port it to v4 as well, in case we make any more 4.x releases later on. The PR not being backported in the first place was more an oversight than anything else...