Unable to import "site" in an embedded application

[AlexWMF]

BPO	26998
Nosy	@pfmoore, @tjguk, @bitdancer, @zware, @zooba, @AlexWMF
Superseder	bpo-25824: 32-bit 2.7.11 installer creates registry keys that are incompatible with the installed python27.dll
Files	Clipboard29.png: output of both x86 and x86_64 python interpreters

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-05-13.15:46:12.057>
created_at = <Date 2016-05-11.11:00:28.871>
labels = ['type-bug', 'expert-installation', 'OS-windows']
title = 'Unable to import "site" in an embedded application'
updated_at = <Date 2016-05-13.15:46:12.056>
user = 'https://github.com/AlexWMF'

bugs.python.org fields:

activity = <Date 2016-05-13.15:46:12.056>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2016-05-13.15:46:12.057>
closer = 'steve.dower'
components = ['Installation', 'Windows']
creation = <Date 2016-05-11.11:00:28.871>
creator = 'AlexWMF'
dependencies = []
files = ['42814']
hgrepos = []
issue_num = 26998
keywords = []
message_count = 7.0
messages = ['265303', '265308', '265311', '265313', '265314', '265317', '265478']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'r.david.murray', 'zach.ware', 'steve.dower', 'AlexWMF']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '25824'
type = 'behavior'
url = 'https://bugs.python.org/issue26998'
versions = ['Python 2.7']

[AlexWMF]

Unable to import "site" error occurs in an embedded application.
I have made small research and found why this module isn't visible for python core.

I have run the python 2.7.11 x86 (for all users) at my win 7x64.
The installer writes the registry key "Software\Wow6432Node\Python\PythonCore\%s\", where "%s" is equal to "32" on my machine.
Difference between x86 and x86_64 registry keys is only in "Wow6432Node" subkey in that key path.
Tracing the application, I have got to getpythonregpath(...) function. This function reads the registry key to determine "PythonPath".
The python core tries to open the following key "Software\Wow6432Node\Python\PythonCore\2.7-32\PythonPath" and fails due to absence of that key.

In this case, the DLL was compiled with PyWin_DLLVersionString = "2.7-32".
Because of this, the python core can't open this key and the "sys.path" stays not fully filled with paths from registry.

The PyWin_DLLVersionString is compiles from MS_DLL_ID="$(SysWinVer)", which is equal to "2.7-32" on my system.
It's defined in .\PCbuild\python.props file as
<SysWinVer>$(MajorVersionNumber).$(MinorVersionNumber)</SysWinVer>
<SysWinVer Condition="$(Platform) == 'Win32' or $(Platform) == 'x86'"&gt;$(SysWinVer)-32</SysWinVer>

So, the simplest solution of this bug is to fix installer to store proper key

[AlexWMF]

I've checked the x86_64 version of python27.dll.
It contains "2.7-32" as well as x86 python dll. So strange, seems like "-32" is always there, independently of target bitness.

[bitdancer]

As far as I know the registry key is a supplament and not a requirement for a correctly functioning python. There is an entire PEP about the registry keys and what to do with them...I'm not sure what the current status is on 2.7, but like I said, I believe that lack of registry keys should not keep python from working if it is installed correctly. Python uses other methods to find its library. Since you are in an embedded scenario, this probably means that there's something about the environment and/or disk layout that you haven't set up in the way that python expects it to be set up. You migth try asking on python-list or the #python irc channel on freenode for help.

I'm guessing this means there's something missing from the ebedding docs with respect to Windows, so I'm going to leave this open until that is confirmed or disproven :)

[AlexWMF]

Could you provide link to that PEP?
As I posted in the previous message, both x86 and x86_64 python27.dll have version string "2.7-32" (shown in sys.winver). Is this right? Any of installers (x86 nor x86_64) don't create "2.7-32" subkey.

I understand that python should work without set up of sys.path from the registry, but after start of embedded python instance, sys.path don't have any valid path where 'c:\Python27x64\lib' folder resides. sys.path have ['c:\windows\system32\python27.zip', '.\Dlls', '.\Lib\'] and '.\\' expands to current application directory.
But directory is always different, not points to the python home.
When start the application which embeds x86_64 python, it just exits because can't find the 'site' module when tried to perform Py_InitializeEx(0).

[AlexWMF]

The output of both x86 and x86_64 python interpreters is attached

[AlexWMF]

I found the following PEP https://www.python.org/dev/peps/pep-0514/

According to this PEP, starting from python 2.7.11 the backward compatibility was broken, because .dll can't read from the following registry keys anymore:
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
HKEY_CURRENT_USER\Software\Python\PythonCore\<Tag>
HKEY_CURRENT_USER\Software\Wow6432Node\Python\PythonCore\<Tag>

That means, the python27.dll (x86/x86_64) tries to reads '2.7-32' and never see old keys like '2.7'

[zooba]

The PEP doesn't really apply to 2.7, apart from explicitly expecting tools to still detect 32-bit and 64-bit versions despite sys.winver being the same.

As for sys.winver being incorrect in 2.7.11, that's been fixed (bpo-25824), but wasn't deemed critical enough to release 2.7.12 immediately. The next update will be released soon with the fix.