[Windows] Psutil module not working on Windows server with no internet

[nitishnambiar]

Hi @giampaolo/All ,
I am trying to run python(3.6) script to get service statuses using psutil module. It is working fine on my laptop. But the same script is not working on the prod windows server. Getting this error

Traceback (most recent call last):
  File "<<script_name>>.py", line 45, in <module>
    import <<script_name>>
  File "<<script_name>>", line 47, in <module>
    import psutil
  File "packages\psutil\__init__.py", line 147, in <module>
    from . import _pswindows as _psplatform
  File "packages\psutil\_pswindows.py", line 35, in <module>
    from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS
ModuleNotFoundError: No module named 'psutil._psutil_windows'

Note : Do not have internet access on this server, hence I have added all the required modules in a folder and using them. I cannot use pip install psutil on this machine.

[altendky]

Are you running 32 or 64 bit? I'm getting this on 32-bit in GitHub Actions installed via PyPI.

[altendky]

I'm able to import locally in a Windows 10 running in VirtualBox with Python 3.8.2 32-bit.

Below is the error from GitHub Actions. It also works in the same build's 64-bit jobs.

https://github.com/altendky/pyqt5-tools/runs/1157662519?check_suite_focus=true#step:8:5794

    Traceback (most recent call last):
      File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 280, in <module>
        main()
      File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\site-packages\pip\_vendor\pep517\_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-1z8kg9rr\overlay\Lib\site-packages\setuptools\build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-1z8kg9rr\overlay\Lib\site-packages\setuptools\build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 10, in <module>
        import build_new
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-axbf0sj_\build_new.py", line 22, in <module>
        import psutil
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-1z8kg9rr\normal\Lib\site-packages\psutil\__init__.py", line 145, in <module>
        from . import _pswindows as _psplatform
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-1z8kg9rr\normal\Lib\site-packages\psutil\_pswindows.py", line 35, in <module>
        from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS
    ModuleNotFoundError: No module named 'psutil._psutil_windows'

[altendky]

So I skipped over psutil for now in my build and now I get the following traceback. Obviously this isn't something for anyone here to help with, but it does seem to be a similar thing in that it's a failure on importing a 'native' module. So maybe there's just something generally broken with these systems that we both happened to hit via psutil (first).

  Traceback (most recent call last):
    File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\runpy.py", line 185, in _run_module_as_main
      mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
    File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\runpy.py", line 144, in _get_module_details
      return _get_module_details(pkg_main_name, error)
    File "c:\hostedtoolcache\windows\python\3.8.5\x86\lib\runpy.py", line 111, in _get_module_details
      __import__(pkg_name)
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\aqt\__init__.py", line 25, in <module>
      from aqt.cli import Cli
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\aqt\cli.py", line 35, in <module>
      from aqt.installer import QtInstaller
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\aqt\installer.py", line 32, in <module>
      import py7zr
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\py7zr\__init__.py", line 30, in <module>
      from py7zr.py7zr import ArchiveInfo, FileInfo, SevenZipFile, is_7zfile, pack_7zarchive, unpack_7zarchive
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\py7zr\py7zr.py", line 38, in <module>
      from py7zr.archiveinfo import Folder, Header, SignatureHeader
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\py7zr\archiveinfo.py", line 35, in <module>
      from py7zr.compressor import SevenZipCompressor, SevenZipDecompressor
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\py7zr\compressor.py", line 32, in <module>
      from Crypto.Cipher import AES
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\Crypto\Cipher\__init__.py", line 27, in <module>
      from Crypto.Cipher._mode_ecb import _create_ecb_cipher
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\Crypto\Cipher\_mode_ecb.py", line 35, in <module>
      raw_ecb_lib = load_pycryptodome_raw_lib("Crypto.Cipher._raw_ecb", """
    File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\Crypto\Util\_raw_api.py", line 308, in load_pycryptodome_raw_lib
      raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
  OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.cp38-win32.pyd': Could not find module 'C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\Crypto\Cipher\_raw_ecb.cp38-win32.pyd' (or one of its dependencies). Try using the full path with constructor syntax., Trying '_raw_ecb.pyd': Could not find module 'C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-2yqif93p\normal\Lib\site-packages\Crypto\Cipher\_raw_ecb.pyd' (or one of its dependencies). Try using the full path with constructor syntax.

[altendky]

Uh... 32 bit Python compatible with 64 bit wheels and also selecting the 64 bit wheels? I guess I'll be taking this up with... GitHub Actions? Python? pip? I'll have to look around.

(from pip debug --verbose)

https://github.com/altendky/pyqt5-tools/pull/53/checks?check_run_id=1158319323

https://gist.github.com/altendky/bd0895a97f9b45b02055e0b59346dd2f#file-gistfile1-txt-L601-L671

2020-09-24T02:44:09.3992704Z pip version: pip 20.2.3 from c:\hostedtoolcache\windows\python\3.8.5\x86\lib\site-packages\pip (python 3.8)
2020-09-24T02:44:09.3999078Z sys.version: 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:43:08) [MSC v.1926 32 bit (Intel)]
2020-09-24T02:44:09.4000061Z sys.executable: c:\hostedtoolcache\windows\python\3.8.5\x86\python.exe

2020-09-24T02:44:09.5991854Z Compatible tags: 30
2020-09-24T02:44:09.5993032Z   cp38-cp38-win_amd64

https://gist.github.com/altendky/bd0895a97f9b45b02055e0b59346dd2f#file-gistfile1-txt-L2097

2020-09-24T02:44:16.2831520Z     Downloading sip-5.4.0-cp38-cp38-win_amd64.whl (509 kB)

[altendky]

pypa/packaging#327

@nitishnambiar, maybe check if you have VSCMD_ARG_TGT_ARCH set in your environment.

[nitishnambiar]

@altendky , thanks for the update.... I could not find VSCMD_ARG_TGT_ARCH environment variable..

[altendky]

Maybe still do a verbose install and check what wheels are getting installed. If they look like the correct bit width, then maybe try https://github.com/lucasg/Dependencies. Are you running 32 or 64 bit wheels python?

[nitishnambiar]

Hi @altendky , these are prod servers and has no internet access, therefore I cannot do pip install. I am basically copying the installed modules from "site-packages" folder in a mirror server to this prod server. (Both servers are having exact configurations i.e. 64 bit )
P.S. : I am new to python and this is the first time I came across this kind of scenario where the servers have no permission to internet and installations.

[altendky]

Copying files isn't installing. Sure, sometimes you can get away with it, but it's not something I would recommend. There's pex and shiv for building a single-file to be run with Python and pyinstaller that does the same but includes Python. You can also just pip wheel to collect wheel files for all the packages, copy those onto the server, and do a real installation process there. In case you aren't, you should also be working in a virtual environment usually. https://bit.ly/py-env gives an intro. They are less necessary on Windows than in Linux but they are easy enough to create and definitely still a thing to use anywhere you try to work with two separate projects or even just two copies of a project. pip-tools or poetry could be relevant to keeping track of what versions of packages you want to be using.

But, it sounds like probably something isn't the same... maybe the linked dependencies program can help identify what.

[nitishnambiar]

Thanks for the detailed response... I will surely go through the official documentations for the above mentioned options.

[giampaolo]

Copying source source files (is that what you're doing?) cannot work because there are C files involved, and those require compilation. If you're copying (previously compiled) pyd files it may work but it's not precisely recommended. =)
Can't you simply use wheel files https://pypi.org/project/psutil/#files? pip is also able to install those from the disk, without accessing internet.
Anyway, I'm closing this as it's not a psutil issue per-se.