building: attempt to preserve parent directories for pywin32 extensions #7627
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
At the end of the analysis process, post-process the `binaries` TOC and apply two work-arounds for the `pywin32` package on Windows. First, we now attempt to preserve the parent directory layout for `pywin32` extensions that are originally located in `win32` and `pythonwin` directories. The `pywin32.pth` places these directories into `sys.path`, so we end up seeing them as top-level modules, and collect them into `sys._MEIPASS`. However, some of these extensions might be linked to other extensions, and for those our binary dependency analysis would preserve the parent directory layout. So to keep things consistent, we now explicitly attempt to preserve the parent directory of extensons that were discovered by the modulegraph as well. This is done as post-processing step on the `binaries` TOC, and also requires us to add the sub-directories in question to `sys.path` in our bootstrap module for `pywin32`. The second work-around is specific to Anaconda `pywin32` package, which installs three copies of `pywintypes3X.dll` and ` pythoncom3X.dll`, in the following locations: - Library/bin - Lib/site-packages/pywin32_system32 - Lib/site-packages/win32 So if, due to order of imports and library location search order, we end up collecting from `Library/bin` (into `sys._MEIPASS`) or from `Lib/site-packages/win32` (into `sys._MEIPASS/win32`), we need to explicitly change the destination directory to `sys._MEIPASS/pywin32_system32` to match the layout we expect.
Remove the special handling for `pywin32` from dependency scanner's parent directory preservation mechanism. We had code in place that explicitly prevented directory preservation for dependent binaries from `pythonwin` directory. This was because we were collecting `win32uiole` extension from that directory into `sys._MEIPASS` (due to it being seen as top-level module, thanks to `pywin32.pth` adding the directory to `sys.path`); but the extension is linked against `win32ui` extension, so when discovered by dependency analysis of `win32uiole`, we would place it in the `pythonwin` directory. Thus, the two extensions would end up in different place, and `win32uiole` would refuse to load due to missing dependency. But now, our analysis/build process explicitly forces collection of extensions from `pythonwin` and `win32` to their original subdirectories (by means of `binaries` TOC post-processing), so it does not matter if we collect `win32ui` into the subdirectory; the `win32uiole` will also be placed next to it in the end.
bwoodsend
approved these changes
May 13, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Attempt to preserve the parent directories for
pywin32extensions that originate fromwin32andpythonwindirectories, rather than collecting them to the top-level application directory (which happens by default because those directories are added tosys.pathbypywin32.pth). This is achieved by post-processing thebinariesTOC at the end of the analysis process.This brings us closer to the original package layout, and allows us to remove a
pywin32-specific hack from binary dependency analysis, where we earlier had to force collection of dependent binaries from thepythonwindirectory to the top-level directory (so that an extension from that directory, discovered via modulegraph, and dependent extension from that directory, discovered via binary dependency scanner, ended up in the same place).With a similar post-processing step, we also attempt to mitigate the issues caused by Anaconda
pywin32package that installs three copies ofpywintypes3X.dllandpythoncom3X.dllin different locations, all of which are visible to us. So we have no control over which one we collect, which is problematic because we expect the copy to end up inpywin32_system32directory. But with post-processing, we can bend the reality to our will, and divert those two files into desired destination directory. See #7624 (comment).Our
pywin32extension import tests now all seem to pass also under Anaconda environments with Anacondapywin32package installed.