Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unfreeze setuptools version again, workaround #1936

Merged
merged 4 commits into from
Aug 28, 2022
+23 −163
Merged

Unfreeze setuptools version again, workaround #1936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
39a06f8
Unfreeze setuptools version again, workaround
kxrob Aug 21, 2022
68ef031
Cleanups and de-lints in setup.py
kxrob Aug 21, 2022
522c03e
Remove obsolete _fixup_sdk_dirs() mechanism in setup.py (#1897)
kxrob Aug 21, 2022
b2110ac
Remove bdist_wininst from CI (gone in setuptools / distutils)
kxrob Aug 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 2 additions & 12 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,7 @@ jobs:
run: |
python --version
pip --version
# XXX With setuptools 63.4.3 .. 64.0.0 the scintilla nmake won't compile (
# setuptools.msvc._msvc14_get_vc_env() won't get the VC\INCLUDE into os.environ)
# Lets see if this will be fixed again in higher versions
pip install --upgrade setuptools<=63.2.0 wheel
pip install --upgrade setuptools wheel

- name: Build and install
run: |
Expand All @@ -46,13 +43,6 @@ jobs:
run: |
python setup.py bdist_wheel --skip-build

# 3.10 dropped support for bdist_wininst. If we had both 3.9 and 3.10 available in a single
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? It means we don't get .exe installers from CI.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bdist_wininst has been completely removed recently from setuptools /distutils which is effective with current version of setuptools. (Commit message of 4. patch: Remove bdist_wininst from CI (gone in setuptools / distutils) -
Removed now in pypa/distutils@e28b6193 )

So its dead until some independent project would continue bdist_wininst or bdist_msi .
On the build machine you may use an older version of setuptools for some time if needed, or somehow fall back to deprecated legacy distutils in python (where the stub .exe were taken from anyway via setup.py/maybe_fixup_exes() ) ... ?
Or end .exe installers completely?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry the the delay and thanks for the explanation! Download stats of the .exe from github are quite minor compared to pypi, so I think I'll open an issue to phase them out, but might as well do that as late as practical. My local environment can still .exe installers, so 👍 and thanks!

# environment we could also build the .exe for 3.10+ - but for now, we can't.
- if: matrix.python-version != '3.10' && matrix.python-version != '3.11-dev'
name: Build installers
run: |
python setup.py bdist_wininst --skip-build --target-version=${{ matrix.python-version }}

- uses: actions/upload-artifact@v2
if: ${{ always() }}
with:
Expand Down Expand Up @@ -83,7 +73,7 @@ jobs:
run: |
python --version
pip --version
pip install --upgrade setuptools<=63.2.0 wheel
pip install --upgrade setuptools wheel

- name: Obtain ARM64 library files
run: |
Expand Down
4 changes: 2 additions & 2 deletions Pythonwin/Scintilla/makefile_pythonwin
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ all: ..\pywin\scintilla\scintillacon.py Scintilla


clean:
cd win32 && $(MAKE) /$(MAKEFLAGS) -f scintilla.mak DIR_BIN=$(SUB_DIR_BIN) DIR_O=$(SUB_DIR_O) clean & cd..
cd win32 && $(MAKE) /$(MAKEFLAGS) -f scintilla.mak DIR_BIN=$(SUB_DIR_BIN) DIR_O=$(SUB_DIR_O) clean && cd ..

Scintilla:
@if not exist $(DIR_O) md $(DIR_O)
Expand All @@ -42,7 +42,7 @@ Scintilla:
@set LINK=$(LINK) /nologo
where cl
set INCLUDE
@cd win32 && $(MAKE) /$(MAKEFLAGS) /f scintilla.mak DIR_BIN=$(SUB_DIR_BIN) DIR_O=$(SUB_DIR_O) LEXCOMPONENT=$(SCINTILLA_DLL) COMPONENT=dummy $(SCINTILLA_DLL) & cd..
@cd win32 && $(MAKE) /$(MAKEFLAGS) /f scintilla.mak DIR_BIN=$(SUB_DIR_BIN) DIR_O=$(SUB_DIR_O) LEXCOMPONENT=$(SCINTILLA_DLL) COMPONENT=dummy $(SCINTILLA_DLL) && cd ..

..\pywin\scintilla\scintillacon.py: Include\Scintilla.h Include\SciLexer.h
@if not exist $(DIR_PYTHON)\tools\scripts\h2py.py echo ***** Can't find h2py.py in '$(DIR_PYTHON)\tools\scripts - please pass DIR_PYTHON to this script *******
Expand Down
168 changes: 19 additions & 149 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
"""
# Originally by Thomas Heller, started in 2000 or so.
import os
import string
import sys
import glob
import re
Expand Down Expand Up @@ -54,7 +53,6 @@

from distutils.dep_util import newer_group
from distutils.filelist import FileList
from distutils.errors import DistutilsExecError, DistutilsSetupError
import distutils.util

build_id_patch = build_id
Expand Down Expand Up @@ -89,90 +87,6 @@
dll_base_address = 0x1E200000


def registry_data():
# Find the win 10 SDKs installed.
root = ""
versions = []
try:
key = winreg.OpenKey(
winreg.HKEY_LOCAL_MACHINE,
r"SOFTWARE\Microsoft\Windows Kits\Installed Roots",
0,
winreg.KEY_READ | winreg.KEY_WOW64_32KEY,
)
root = winreg.QueryValueEx(key, "KitsRoot10")[0]
keyNo = 0
while 1:
try:
versions.append(winreg.EnumKey(key, keyNo))
keyNo += 1
except winreg.error:
break
except EnvironmentError:
pass
return root, versions


# We need to know the platform SDK dir before we can list the extensions.
def find_platform_sdk_dir():
# The user might have their current environment setup for the
# SDK, in which case "MSSDK_INCLUDE" and "MSSDK_LIB" vars must be set.
if "MSSDK_INCLUDE" in os.environ and "MSSDK_LIB" in os.environ:
print("Using SDK as specified in the environment")
return {
"include": os.environ["MSSDK_INCLUDE"].split(os.path.pathsep),
"lib": os.environ["MSSDK_LIB"].split(os.path.pathsep),
}

install_root, installed_versions = registry_data()
if not installed_versions:
print("Can't find a windows 10 sdk")
return None

# We don't want to automatically used the latest as that's going to always
# be a moving target. Github's automation has "10.0.16299.0", so we target
# that if it exists, otherwise we use the earliest installed version.
preferred_ver = "10.0.16299.0"
if preferred_ver not in installed_versions:
print(
"Windows 10 SDK version",
preferred_ver,
"is preferred, but that's not installed.",
)
print("Installed versions are", installed_versions)
else:
installed_versions = [e for e in installed_versions if e != preferred_ver]
installed_versions.insert(0, preferred_ver)
user_mode = "um"
for ver in installed_versions:
print("Attempting", ver)
include_base = os.path.join(install_root, "include", ver)
include = [os.path.join(include_base, user_mode)]
if not os.path.exists(os.path.join(include[0], "windows.h")):
print(
"Found Windows sdk in",
include,
"but it doesn't appear to have windows.h",
)
continue
include.append(os.path.join(include_base, "shared"))
lib = [os.path.join(install_root, "lib", ver, user_mode)]
return {"include": include, "lib": lib}
return None # Redundant (for readability)


sdk_info = find_platform_sdk_dir()
if not sdk_info:
print()
print("It looks like you are trying to build pywin32 in an environment without")
print("the necessary tools installed. It's much easier to grab binaries!")
print()
print("Please read the docstring at the top of this file, or read README.md")
print("for more information.")
print()
raise RuntimeError("Can't find the Windows SDK")


def find_visual_studio_file(pattern):
try:
files = subprocess.check_output(
Expand Down Expand Up @@ -510,61 +424,6 @@ def finalize_options(self):
self.excluded_extensions = [] # list of (ext, why)
self.swig_cpp = True # hrm - deprecated - should use swig_opts=-c++??

def _fixup_sdk_dirs(self):
# Adjust paths etc for the platform SDK - the default paths used by
# distutils don't include the platform SDK.
# Note that just having them in INCLUDE/LIB does *not* work -
# distutils thinks it knows better, and resets those vars (see notes
# below about how the paths are put together)

# Called after the compiler is initialized, but before the extensions
# are built. NOTE: this means setting self.include_dirs etc will
# have no effect, so we poke our path changes directly into the
# compiler (we can't call this *before* the compiler is setup, as
# then our environment changes would have no effect - see below)

# distutils puts the path together like so:
# * compiler command line includes /I entries for each dir in
# ext.include_dir + build_ext.include_dir (ie, extension's come first)
# * The compiler initialization sets the INCLUDE/LIB etc env vars to the
# values read from the registry (ignoring anything that was there)

# We are also at the mercy of how MSVC processes command-line
# includes vs env vars (presumably environment comes last) - so,
# moral of the story:
# * To get a path at the start, it must be at the start of
# ext.includes
# * To get a path at the end, it must be at the end of
# os.environ("INCLUDE")
# Note however that the environment tweaking can only be done after
# the compiler has set these vars, which is quite late -
# build_ext.run() - so global environment hacks are done in our
# build_extensions() override)
#
# Also note that none of our extensions have individual include files
# that must be first - so for practical purposes, any entry in
# build_ext.include_dirs should 'win' over the compiler's dirs.
assert self.compiler.initialized # if not, our env changes will be lost!

for extra in sdk_info["include"]:
# should not be possible for the SDK dirs to already be in our
# include_dirs - they may be in the registry etc from MSVC, but
# those aren't reflected here...
assert extra not in self.include_dirs
# and we will not work as expected if the dirs don't exist
assert os.path.isdir(extra), "%s doesn't exist!" % (extra,)
self.compiler.add_include_dir(extra)
# and again for lib dirs.
for extra in sdk_info["lib"]:
extra = os.path.join(extra, self.plat_dir)
assert os.path.isdir(extra), extra
assert extra not in self.library_dirs # see above
assert os.path.isdir(extra), "%s doesn't exist!" % (extra,)
self.compiler.add_library_dir(extra)

log.debug("After SDK processing, includes are %s", self.compiler.include_dirs)
log.debug("After SDK processing, libs are %s", self.compiler.library_dirs)

def _why_cant_build_extension(self, ext):
# Return None, or a reason it can't be built.
# Exclude exchange 32-bit utility libraries from 64-bit
Expand Down Expand Up @@ -615,6 +474,7 @@ def _why_cant_build_extension(self, ext):
look_dirs = common_dirs + ext.library_dirs
found = self.compiler.find_library_file(look_dirs, lib, self.debug)
if not found:
print("-- LIB NOT FOUND:", lib, look_dirs)
log.debug("Looked for %s in %s", lib, look_dirs)
return "No library '%s'" % lib
self.found_libraries[lib.lower()] = found
Expand Down Expand Up @@ -671,14 +531,9 @@ def _build_scintilla(self):

cwd = os.getcwd()
old_env = os.environ.copy()
os.chdir(path)
print("-- _build_scintilla INCLUDE old:", os.environ.get("INCLUDE"))
if not self.compiler.initialized:
print("-- _build_scintilla compiler.initialize()")
self.compiler.initialize()
os.environ["INCLUDE"] = os.pathsep.join(self.compiler.include_dirs)
os.environ["LIB"] = os.pathsep.join(self.compiler.library_dirs)
print("-- _build_scintilla INCLUDE new:", os.environ.get("INCLUDE"))
os.chdir(path)
try:
cmd = [nmake, "/nologo", "/f", makefile] + makeargs
self.compiler.spawn(cmd)
Expand Down Expand Up @@ -706,7 +561,23 @@ def build_extensions(self):
if not self.compiler.initialized:
self.compiler.initialize()

self._fixup_sdk_dirs()
# XXX this distutils class var peek hack should become obsolete
# (silently) when https://github.com/pypa/distutils/pull/172 is
# resolved.
# _why_cant_build_extension() and _build_scintilla() at least need
# complete VC+SDK inspectable inc / lib dirs.
classincs = getattr(self.compiler.__class__, "include_dirs", [])
if classincs:
log.warn("distutils hack to expose complete inc/lib dirs")
print("-- orig compiler.include_dirs:", self.compiler.include_dirs)
print("-- orig compiler.library_dirs:", self.compiler.library_dirs)
self.compiler.include_dirs += classincs
self.compiler.__class__.include_dirs = []
classlibs = getattr(self.compiler.__class__, "library_dirs", [])
self.compiler.library_dirs += classlibs
self.compiler.__class__.library_dirs = []
else:
print("-- FIX ME ! distutils may expose complete inc/lib dirs again")

# Here we hack a "pywin32" directory (one of 'win32', 'win32com',
# 'pythonwin' etc), as distutils doesn't seem to like the concept
Expand Down Expand Up @@ -1174,7 +1045,6 @@ def key_reverse_mc(a):
def spawn(self, cmd):
is_link = cmd[0].endswith("link.exe") or cmd[0].endswith('"link.exe"')
is_mt = cmd[0].endswith("mt.exe") or cmd[0].endswith('"mt.exe"')
_want_assembly_kept = getattr(self, "_want_assembly_kept", False)
if is_mt:
# We don't want mt.exe run...
return
Expand Down