Wrong Paths for distutils build --plat-name=win-amd64

[rgbecker]

BPO	8170
Nosy	@loewis, @mhammond, @tarekziade, @merwok, @zooba
Files	patch.txt: patch to fix order of cross compile lib dirs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tarekziade'
closed_at = <Date 2021-02-03.18:26:37.973>
created_at = <Date 2010-03-18.11:55:31.861>
labels = ['type-bug', 'library', 'OS-windows']
title = 'Wrong Paths for distutils build --plat-name=win-amd64'
updated_at = <Date 2021-02-03.18:26:37.972>
user = 'https://bugs.python.org/rgbecker'

bugs.python.org fields:

activity = <Date 2021-02-03.18:26:37.972>
actor = 'steve.dower'
assignee = 'tarek'
closed = True
closed_date = <Date 2021-02-03.18:26:37.973>
closer = 'steve.dower'
components = ['Distutils', 'Windows']
creation = <Date 2010-03-18.11:55:31.861>
creator = 'rgbecker'
dependencies = []
files = ['16574']
hgrepos = []
issue_num = 8170
keywords = []
message_count = 10.0
messages = ['101259', '113051', '154214', '154429', '154543', '154548', '154619', '154667', '154686', '386396']
nosy_count = 6.0
nosy_names = ['loewis', 'mhammond', 'rgbecker', 'tarek', 'eric.araujo', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8170'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[rgbecker]

When building extensions on win32 distutils with --plat-name=win-amd64 adds PCBuild/AMD64 in the wrong place.

This patch ensures the AMD64 location comes first

[terryjreedy]

With development of disutils2, distutils issues are not getting much separate attention. This may or may not get folded in to disutils2.

[merwok]

Martin, Mark, can either of you comment on this?

[mhammond]

I don't quite understand how the order will be wrong. Which earlier entry is causing the problem? OTOH though, I also don't quite understand how the build would work at all if a 32bit Python is used to invoke distutils with --plat-name=win-amd64 as the 32bit Python install wont have that PCBuild/AMD64 directory at all - so some more context is needed before I can understand this.

[rgbecker]

Some context. ReportLab windows exe installers for pythons 2.x x=4-7 are built on a single 32bit machine with 32bit pythons using a code that looks like this

set FT_LIB=c:\devel\libs_x86\freetype.lib
\python2x\python setup.py bdist_wininst --bitmap=%BMP% --title="ReportLab"

we build the 64 bit versions for 2.6 & 2.7 only. In those cases an extra step does

set FT_LIB=c:\devel\libs_amd64\freetype.lib
\python2.x\python setup.py bdist_wininst --bitmap=%BMP% --plat-name=win-amd64 --title="ReportLab"

To make this work I have copied from 64bit win32 installs of python2.6/7 various lib/dll/pyds. These live in

C:\Python2x\PCBuild\AMD64

eg
C:\ux\ExeBuilder>ls \Python26\PCBuild\AMD64
_bsddb.lib _socket.lib pyexpat.lib
_bsddb.pyd _socket.pyd pyexpat.pyd
_ctypes.lib _sqlite3.lib python26.dll
_ctypes.pyd _sqlite3.pyd python26.lib
_ctypes_test.lib _ssl.lib select.lib
_ctypes_test.pyd _ssl.pyd select.pyd
_elementtree.lib _testcapi.lib sqlite3.dll
_elementtree.pyd _testcapi.pyd tcl85.dll
_hashlib.lib _tkinter.lib tclpip85.dll
_hashlib.pyd _tkinter.pyd tk85.dll
_msi.lib bz2.lib unicodedata.lib
_msi.pyd bz2.pyd unicodedata.pyd
_multiprocessing.lib py.ico winsound.lib
_multiprocessing.pyd pyc.ico winsound.pyd

I followed most of section 5.4 in http://docs.python.org/distutils/builtdist.html to get this almost working, but got problems related to the library paths which my patch appears to fix (I seem to get working exes out of this approach).

[mhammond]

Can't you just install a 64bit Python in a different directory and use that executable to build the 64bit installer? It seems less error prone and should work without manually copying stuff or changing any code.

[rgbecker]

That would be a solution if we had a separate 64 bit machine & extra versions of Visual Studio, but the actual bug is with the cross-compiling behaviour. If it's not supposed to work then this isn't a bug and section 5.4 should go; otherwise the correct way to do this should be better documented and made to work.

[mhammond]

Those instructions in section 5.4 do seem wrong. On first reading they imply that you need to start with a clean source tree on your 32bit Windows, then build the 64bit version of Python. So far so good. However, then you need to run python.exe - which must be a 32bit version of Python or it wouldn't start.

So I *guess* the instructions really should say to build *both* the 32 and 64bit versions in the same source tree - in which case I can see why your patch would be necessary.

Can you confirm the process you used? I assume you did build both versions in the same tree, but you also mentioned copying the .lib files around - was that an attempt to work around this bug?

But yeah, sorry for the distraction - given 5.4 I think this bug is probably valid, but the patch should probably include updated instructions in 5.4 to make it more clear.

[rgbecker]

I cheated on the building both versions. I had 32 bit python installed and with the help of a colleague got hold of the installed files for the 64 bit version. I noticed that distutils was looking for the 64bit files in new_lib = os.path.join(sys.exec_prefix, 'PCbuild'). However, even though I had the files there the loader was missing them and trying to use the earlier lib specs. Experimenting with the command line led to the desire to prioritize the amd4 lib specification ie use insert instead of append for library_dirs. After this patch things do seem to work, but perhaps there's a better fix that removes/moves the 32 bit libs in some way. However, given that I am building the exes with a 32 bit python I suppose it's natural for distutils to assume we need the 32 bit libs.

[zooba]

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools