pip install pywin32 inside Conda Python 3.8/3.9 env causes DLL module not found error

[eddiebergman]

Hello, thanks for the awesome library :)

# Sys info
Python 3.7 | 3.8
Conda
pywin32==303

I apologies if this is a dupe but I didn't find a clean answer given the various installation methods.

When using conda with Python 3.8 or 3.9 and install pywin32 with pip install pywin32, the following error is raised:

# Inside a conda env
# ./mypackage, a source folder which requires `pywin32`
$ cat mypackage/src/__init__.py
> import win32api

$ pip install mypackage
$ python -c "import mypackage"
> ImportError: DLL load failed while importing win32api: The specified module could not be found.

Simple Solution for others reading

The easiest fix for most cases is to simply:

• pip uninstall pywin32; conda install pywin32.

Failing that, please see the README.md about the post_install script:

• https://github.com/mhammond/pywin32#installing-via-pip

This was not required in every other case

---

However we are developing a cross-platform package pynisher that uses pywin32 for limiting memory of processes for compatibility on windows platforms. This package is mostly a dependency for research tools which are mostly just source code hosted on github and installed using git clone <x>; pip install <x>, we also don't have experience with releasing to conda. I would imagine many of these users will also be using conda environments. Unfortunately, as far as I'm aware conda install <x> will not work on local source code and so recommending git clone <x>; conda install <x> will not work and requires <x> to be built first. As a further step, part of our CI is testing installation of packages in conda envs with pip install. All dependant tools would require updating the CI to perform the above fix.

While the above solution works, it would require all dependent tools depending on pynisher to include explicit steps on how to fix this issue. Ideally, this could be resolved within pynisher's installation but I'm not sure how and what the best solution is.

1. Have all dependant tools be forced to include this extra fix step.
2. Have a post_install inside setup.py for pynisher which calls the post_install script.
   • If this is the case, would there be side effects left over after something like pip uninstall pywin32 is run or the virtual env destroyed?
   • As a side note this comment seems a bit too strong as doing dist installs and normal python virtualenv setups work fine with pip install pywin32. I've tested the following setups in our github actions CI and the issue is only with the described setup of conda and 3.8 or 3.9.
     

     pywin32/pywin32_postinstall.py

     Lines 696 to 702 in f53d207

     	> python pywin32_postinstall.py -install
     	If you installed pywin32 via a .exe installer, this should be run
     	automatically after installation, but if it fails you can run it again.
     	If you installed pywin32 via PIP, you almost certainly need to run this to
     	setup the environment correctly.

Is this something that is even possible to fix within pywin32 or is the scope of the problem related to conda and python 3.8/3.9?

Apologies for the long post, I understand it's an error of interaction and I hope this could hopefully serve as something informative to any other people facing this particular packaging issue!

[vernondcole]

This is probably a side-effect of the unfortunate fact that anaconda python comes with an ancient version (228) of pywin32 pre-installed. That is my principle reason for not using the anaconda release of Python -- they like to hang on to older versions of libraries. Nevertheless, in order to confirm your problem, I have downloaded and installed (on the machine where I am paid to do data science things) the latest copy of anaconda. I was able to get "python -m pip install pywin32==303 --user" to appear to work, but it, also, seems to need postinstall.py, so, even though it claims to work, that's a lie. So, your failure is confirmed, using Anaconda Python 3.9. Now, what to do about it. . . Have you determined that pywin32 version 228 will in fact not work for your application? Your easiest answer would be not to attempt to pip install a newer version. There is a pretty good chance that the functionality you need is present in v 228. If that does not work, the ultimate solution lies in getting Anaconda to upgrade their installed version. They build their own custom version of Python, and there is no guarantee that packages written in C will work with their version. The .h files could be incompatible in some way which would cause failure. Ideally, C modules should be compiled using the same toolchain used to build the Python interpreter itself. Probably the most sweeping changes in the last few versions of pywin32 have been in the build procedure itself, so building from source (such as Anaconda would want to do) is easier now.

…

On Fri, Apr 22, 2022 at 2:37 PM Eddie Bergman ***@***.***> wrote: Hello, thanks for the awesome library :) # Sys info Python 3.7 | 3.8 Conda pywin32==303 # Inside a conda env# ./mypackage, a source folder which requires `pywin32` pip install mypackage python -c "import my package" ImportError: DLL load failed while importing win32api: The specified module could not be found. I apologies if this is a dupe but I didn't find a clean answer given the various installation methods. When using conda with Python 3.8 or 3.9 and install pywin32 with pip install pywin32, the following error is raised: The easiest fix for most cases is to simply pip uninstall pywin32; conda install pywin32. This is not an issue with Python 3.7 or Python 3.10 from a conda env. However we are developing a cross-platform package pynisher <automl/pynisher#18> that uses pywin32 for limiting memory of processes for compatibility on windows platforms. This package is mostly a dependency for research tools which are mostly just source code hosted on github and installed using pip install <x>, we also don't have experience with releasing to conda. I would imagine many of these users will also be using conda environments. Unfortunately, as far as I'm aware conda install <x> will not work on local source code and requires <x> to be built first. As a further step, part of our CI is testing installation of packages in conda envs with pip install. All dependant tools would require updating the CI to perform the above fix. While the above solution works, it would require all dependent tools depending on pynisher to include explicit steps on how to fix this issue. Ideally, this could be resolved within pynisher's installation but I'm not sure how and what the best solution is. 1. Have all dependant tools be forced to include this extra fix step. 2. Have a post_install inside setup.py for pynisher which calls the post_install script. - If this is the case, would there be side effects left over after something like pip uninstall pywin32 is run or the virtual env destroyed? - As a side not this comment seems a bit too strong, I've tested the following setups <https://github.com/automl/pynisher/runs/6134570105?check_suite_focus=true> in our github actions CI <https://github.com/automl/pynisher/blob/f64046ae744c84674bd450a2283a54b9bd5c3547/.github/workflows/pytest.yml#L88> and the issue is only with the described setup. https://github.com/mhammond/pywin32/blob/f53d207b28108d6c070fb9163102436364023324/pywin32_postinstall.py#L696-L702 Is this something that is even possible to fix within pywin32 or is the scope of the problem related to conda and python 3.8/3.9? Apologies for the long post, I understand it's an error of interaction and I hope this could hopefully serve as something informative to any other people facing this particular packaging issue! — Reply to this email directly, view it on GitHub <#1865>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZOBICWSUU4A6DKLAONJTVGMEXNANCNFSM5UDKKAFA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[eddiebergman]

Hi @vernondcole,

Thanks for the pointers one possible solution I tried was:

setup(
    install_requires=[
        "pywin ; platform==Windows ; python_version==3.7 or python_version >= 3.10",
        "pywin32==228 ; platform==Windows ; python_version==3.8 or python_version==3.9",
    ]
)

Unfortunately, it doesn't seem like there's a straightforward way to specify requirements based on the env it's running in, i.e. virtaulenv or conda, so this will needlessly effect non conda users.

I also see some issues with this with respect to dependencies, i.e. some package requires pywin32 >= 300. Our university makes machine learning tools which are often used in upstream applications or Jupyter environments, giving a dependency chain such as app_package -> machine_learning_package -> optimizer_package -> pynisher. Locking a version could provide quite a few upstream issues.

The pip documentation for pip install says it shouldn't try to update pywin32 if it can find it already, so I assume anaconda is somehow hiding this from pip which causes it to install the newest compatible version, instead of using the one that is there.

It doesn't seem like there's anything you can do at the pywin32 level and I don't think it's something conda will fix anytime soon. My solution is to disable the feature relying on pywin32 if it fails to import and just include documentation which effected users will have to read and do, i.e. pip uninstall pywin32; conda install pywin32.

Please feel free to close this issue, I hope it can help anyone else encountering this issue and it will be useful to link in documentation!

Please feel free to close :)

[skewty]

In order to get services working I had to make these changes:

cd /d C:\Program Files\Python310\lib\site-packages

# fix for python windows service .DLL errors
copy pywin32_system32\*.dll ..\..
copy win32\pythonservice.exe ..\..

# fix for .DLL naming issue with logging.handlers.NTEventLogHandler
copy win32\win32service.*.pyd win32\win32service.pyd

This doesn't quite seem to be enough though. My Windows Event Error log is still showing entries for my service using Event ID numbers I don't understand / recognize (aren't in my code or in pywin32 code (example servicemanager.py). Any ideas about other fixes / tweaks required?

[yezhengli-Mr9]

Is pip no longer supported? Or these relevant repo only support on Windows operation system?

python -m pip install --upgrade pywin32

[vernondcole]

This will be a bit of a non-answer -- but hopefully it may help with understanding. ... Ananconda Inc. makes an alternate distribution of Python which includes a number of packages which are useful for some professions such as data science. The packages they include are selected and tested to interoperate correctly without surprises, so that it is not necessary to download a number of large, hard to install packages before you can get useful work done. I have worked with data scientists who use it every day and love it. One of the reasons that I personally do not appreciate conda, is because they tend to include very old versions of some packages -- pywin32 being a frequent example of that happening. The conda developers do intend for it to support installations using pip, and that works most of the time. However, some packages can only be installed with the conda package manager, using "conda install" rather than "pip install". You might want to try that. See https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html for their documentation. I find that in my work of packaging code written by data scientists so that it can be distributed and used by other people, that I am better off to use a recent version of Python downloaded directly from https://python.org and write a suitable `requirements.txt` file <https://pip.pypa.io/en/stable/reference/requirements-file-format/> to install all of the packages I need from pypi <https://pypi.org> using pip. I assume that somewhere there is a conda discussion forum where you might be able to get a more definitive answer than this about your exact situation. I have not found it. (If someone who reads this can share a link to that forum -- thank you!)

…

On Fri, Oct 27, 2023 at 11:53 AM Yezheng Li ***@***.***> wrote: Are pip no longer supported python -m pip install --upgrade pywin32 — Reply to this email directly, view it on GitHub <#1865 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZOBL3FGU2QSH3662Z5N3YBPYLBAVCNFSM5UDKKAFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZYGMZDSNRRGQ2Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Avasam]

setup(
    install_requires=[
        "pywin ; platform==Windows ; python_version==3.7 or python_version >= 3.10",
        "pywin32==228 ; platform==Windows ; python_version==3.8 or python_version==3.9",
    ]
)

[...]
The pip documentation for pip install says it shouldn't try to update pywin32 if it can find it already, so I assume anaconda is somehow hiding this from pip which causes it to install the newest compatible version, instead of using the one that is there.

AFAIK Conda doesn't support PEP 508 Environment markers. (see conda/conda#11198)
It does however have something called Preprocessing selectors, but I never looked into it much. (conda/conda#676 (comment))

(maybe also related: conda/conda#8089 and conda/conda#7258)

[Avasam]

Sending the ball in Conda's camp for this one, given that the issue seems to be caused by an older version of pywin32 being shipped with Conda, with the solution always being "reinstall or upgrade pywin32":

• https://github.com/mhammond/pywin32#troubleshooting
• ImportError: DLL load failed while importing shell: Can not find procedure. conda/conda#11503
• MS Learn Space Rock Exercise - conda reports "Import Error", apparently from win32 library conda/conda#10525
• UNDIAGNOSED ERROR: "environment variables: conda info could not be constructed. KeyError('pkgs_dirs')" conda/conda#9356 (comment)
• Cannot load conda in command prompt; conda/conda#10577 (comment)
• Windows machines without COM support fail to install packages conda/conda#12006

We can re-open if there's something pywin32 can do with a concrete plan to alleviate issues with Conda (and isn't duplicating another issue)