Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

distutils not included with latest virtualenv (16.4.0) #4064

Closed
lsoica opened this issue Feb 12, 2019 · 33 comments

Comments

@lsoica
Copy link

commented Feb 12, 2019

This was already reported in #4031. The issue was closed without a fix so I'm creating this one.

With virtualenv 16.4.0, pyinstaller reports :

3583 INFO: distutils: retargeting to non-venv dir '/usr/lib64/python3.6/distutils/__init__.py'

and then during "Loading module hook" sequence, the hook-distutils.py is missing and distutils modules are not included into the final executable binary.

When executing the binary the error is:

ModuleNotFoundError: No module named 'distutils'
[10373] Failed to execute script <name here>

With virtualenv 16.1.0, pyinstaller reports :

3157 INFO: Processing pre-find module path hook   distutils
5053 INFO: Loading module hook "hook-distutils.py"...

and distutils modules are included into the final executable binary.

@lsoica lsoica changed the title distutils not included with latest virtualenv (16.0.4) distutils not included with latest virtualenv (16.4.0) Feb 12, 2019

@gsemet

This comment has been minimized.

Copy link

commented Feb 13, 2019

I also have a pbl with disutils on 16.3. Pyinstaller does not says anything:

5055 INFO: Processing pre-find module path hook   distutils
5055 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.6/distutils/__init__.py'

But later on

16793 WARNING: Hidden import "distutils.command.build_ext" not found!
...
17570 DEBUG: Collecting submodules for distutils
17676 DEBUG: collect_submodules - Found submodules: {'distutils'}

But at execution I still get:

ModuleNotFoundError: No module named 'distutils'

Worked fine on 16.1

Running pyi-archive_viewer on the executable does not list distutils package.
On an executable that worked, pyi-archive_viewer shows the following packages:

 'distutils': (1, 1487595, 267),
 'distutils._msvccompiler': (0, 1487862, 7159),
 'distutils.archive_util': (0, 1495021, 3352),
 'distutils.ccompiler': (0, 1498373, 12557),
 'distutils.cmd': (0, 1510930, 6328),
 'distutils.command': (1, 1517258, 326),
 'distutils.command.bdist': (0, 1517584, 2030),
 'distutils.command.build_ext': (0, 1519614, 8083),
 'distutils.command.build_scripts': (0, 1527697, 2375),
 'distutils.command.install': (0, 1530072, 6322),
 'distutils.command.install_lib': (0, 1536394, 2602),
 'distutils.command.install_scripts': (0, 1538996, 1128),
 'distutils.command.sdist': (0, 1540124, 6183),
 'distutils.config': (0, 1546307, 1895),
 'distutils.core': (0, 1548202, 3246),
 'distutils.debug': (0, 1551448, 137),
 'distutils.dep_util': (0, 1551585, 1430),
 'distutils.dir_util': (0, 1553015, 3048),
 'distutils.dist': (0, 1556063, 13966),
 'distutils.errors': (0, 1570029, 1966),
 'distutils.extension': (0, 1571995, 3282),
 'distutils.fancy_getopt': (0, 1575277, 5261),
 'distutils.file_util': (0, 1580538, 2989),
 'distutils.filelist': (0, 1583527, 4493),
 'distutils.log': (0, 1588020, 1080),
 'distutils.msvc9compiler': (0, 1589100, 8918),
 'distutils.spawn': (0, 1598018, 2749),
 'distutils.sysconfig': (0, 1600767, 6128),
 'distutils.text_file': (0, 1606895, 3793),
 'distutils.util': (0, 1610688, 7966),
 'distutils.version': (0, 1618654, 3110),
 'distutils.versionpredicate': (0, 1621764, 2346),
@gsemet

This comment has been minimized.

Copy link

commented Feb 13, 2019

Solution for the moment: force the system's virtualenv to be 16.1 (freeze the virtualenv package version inside the virtualenvironment is not enough)

rtrbt added a commit to Tinkerforge/brickv that referenced this issue Feb 15, 2019

Add work-around for pyinstaller not including distutils.
pyinstaller/pyinstaller#4064

As a work-around the red brick plugin uses it's own copy of
distutils.version.

rtrbt added a commit to Tinkerforge/brickv that referenced this issue Feb 20, 2019

Add work-around for pyinstaller not including distutils.
pyinstaller/pyinstaller#4064

As a work-around the red brick plugin uses it's own copy of
distutils.version.

rtrbt added a commit to Tinkerforge/brickv that referenced this issue Feb 26, 2019

Add work-around for pyinstaller not including distutils.
pyinstaller/pyinstaller#4064

As a work-around the red brick plugin uses it's own copy of
distutils.version.
@tappi287

This comment has been minimized.

Copy link

commented Mar 2, 2019

Just to pinpoint the problem a bit further: <virtual_env>/Lib/distutils/__init__.py attribute distutils_path returns a path to a file <system_interpreter>/Lib/distutils/__init__.py from virtualenv 16.4.0 on, but previously returned a path to the directory <system_interpreter>/Lib/distutils/.

PyInstaller hook hook-distutils uses os.path.dirname to expand the search directorys which will result in <system_interpreter>/Lib/distutils/ with virtualenv 16.4.0 which would result with <system_interpreter>/Lib/ with previous version of virtualenv.

@cpbotha

This comment has been minimized.

Copy link

commented Mar 8, 2019

Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.

@Ziqi-Li

This comment has been minimized.

Copy link

commented Mar 9, 2019

I also had this problem with virtualenv. So i switch to use python's built in venv.

rtrbt added a commit to Tinkerforge/brickv that referenced this issue Mar 13, 2019

Add work-around for pyinstaller not including distutils.
pyinstaller/pyinstaller#4064

As a work-around the red brick plugin uses it's own copy of
distutils.version.
@jce94

This comment was marked as off-topic.

Copy link

commented Mar 31, 2019

Does anyone know something? I am working with jenkins. I build a virtualenv with this packages:

  • altgraph==0.16.1
  • cx-Freeze==5.1.1
  • cycler==0.10.0
  • distutils-pytest==0.1
  • future==0.17.1
  • kiwisolver==1.0.1
  • macholib==1.11
  • matplotlib==3.0.2
  • numpy==1.15.0
  • pandas==0.23.4
  • pefile==2018.8.8
  • PyInstaller==3.4
  • pyparsing==2.3.1
  • python-dateutil==2.7.5
  • pytz==2018.9
  • pywin32-ctypes==0.2.0
  • six==1.12.0
  • tornado==6.0.1
  • virtualenv==16.2.0

But my problem is that in jenkins fails becaouse my app said: "ModuleNotFoundError: No module named 'distutils' after buids the my app. I exetue this app (is .exe to Windows) but the console said that.

If i build the app wit pyinstaller in a console with virtualenv and the same packacts as befere, the build app NOT FAILD. Help me please!

@Bernardoow

This comment was marked as off-topic.

Copy link

commented Apr 2, 2019

Same probleam here.

@jce94

This comment was marked as off-topic.

Copy link

commented Apr 2, 2019

Same probleam here.

Hi @Bernardoow . My problem appear when I am executing pyinstaller to generate my app in jenkins job into a virtualenv. I don´t know because it happens.
If I execute the pyinstaller in my local machine into a virtualenv the promblem not appear and my app is built successfully. I appreciate any help

@gsemet

This comment has been minimized.

Copy link

commented Apr 2, 2019

Did you tried installing virtualenv 16.1 on your system (pip install —user virtualenv==16.1)?

@Bernardoow

This comment has been minimized.

Copy link

commented Apr 2, 2019

Before I sent the message I was testing on the linux and pyinstaler not running after compiled.

Now I'm testing on the windows and pyinstaler is working normally.

I'm Using virtualenv 16.1.0 in the boths os and the same requirements.txt. Python 3.6.8

i'm using this code suggested by @cpbotha : import distutils if distutils.distutils_path.endswith('__init__.py'): distutils.distutils_path = os.path.dirname(distutils.distutils_path)

@maxslimmer

This comment has been minimized.

Copy link

commented Apr 5, 2019

How about a small variation on @cpbotha's suggestion and putting the workaround directly in the hook?

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)
            
        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]
            
        logger.info('distutils: retargeting to non-venv dir %r' % distutils_dir)

NB: reverting system virtualenv AND re-creating the venv also worked for me.

@meidachen

This comment has been minimized.

Copy link

commented Apr 5, 2019

virtualenv 16.1 worked for me, thanks!

@jce94

This comment was marked as off-topic.

Copy link

commented Apr 5, 2019

Thanks to all, I am going to try revet my virtualevn 16.1, when I finish I explain here the results. Maybe be util to all. But I think that the problem is being create virtualenv into Jenkins 😄

@j9ac9k

This comment has been minimized.

Copy link

commented Apr 5, 2019

Ran into this error, and can confirm the change to virtualenv 16.1.0 was required, but it had to be the virtualenv module that was used by pipenv in my case, which is in line with @gsemet 's comment earlier.

@jce94

This comment was marked as off-topic.

Copy link

commented Apr 8, 2019

Hi all!
It's impossible generate into a virtualen one app with pyinstaller in a job jenkins with this packages:
#4064 (comment)
I changes to virtualenv==16.1.0 but the problem persists.

"ModuleNotFoundError: No module named 'distutils', I feel frustrated 🤦‍♂
If someone could give me some idea I would be enormously grateful. Thanks to all!

@j9ac9k

This comment has been minimized.

Copy link

commented Apr 8, 2019

Hi all!
It's impossible generate into a virtualen one app with pyinstaller in a job jenkins with this packages:
#4064 (comment)
I changes to virtualenv==16.1.0 but the problem persists.

"ModuleNotFoundError: No module named 'distutils', I feel frustrated 🤦‍♂️
If someone could give me some idea I would be enormously grateful. Thanks to all!

you probably need to change the virtualenv on the system level of your jenkins build environment, not the virtualenv version of whatever package you're developing

@JaySoyer

This comment has been minimized.

Copy link

commented Apr 11, 2019

I'm still getting this issue with virtualenv 16.4.3. Down grading to 16.2.0 has worked in the interim.

willcl-ark added a commit to willcl-ark/node-launcher that referenced this issue Apr 12, 2019

work-around for pyinstaller/pyinstaller#4064
Signed-off-by: willcl-ark <will8clark@gmail.com>
@jce94

This comment was marked as off-topic.

Copy link

commented Apr 20, 2019

Hi guys! it was impossible to solve my problem with
"ModuleNotFoundError: No module named 'distutils'. (I was trying to build a .exe through pyinsteller). I found a new option to build my python app: Azure DevOps Pipelines. It is similar to Jenkins (I was building my app in a Jenkins server in my local machine but it was impossible for that problem). The .exe works perfect!

Thanks to all for try to help me with problem! 👍

@oberstet

This comment has been minimized.

Copy link

commented Apr 20, 2019

@maxslimmer 's solution (thanks!) worked for me (on a cpy373 based venv), as in (it needs to be in a pre_find_module_path subdir within the hooks dir):

(cpy373_1) oberstet@intel-nuci7:~/scm/crossbario/crossbarfx$ cat pyinstaller/pre_find_module_path/hook-distutils.py 
import distutils
import os

from PyInstaller.utils.hooks import logger

# https://github.com/pyinstaller/pyinstaller/issues/4064
# https://pythonhosted.org/PyInstaller/hooks.html#the-pre-find-module-path-pfmp-api-method

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)

        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]

        logger.info('>>>>>>> CUSTOM >>>>>>>>> distutils: retargeting to non-venv dir %r' % distutils_dir)

3ll3d00d added a commit to 3ll3d00d/beqdesigner that referenced this issue Apr 22, 2019

@Hassanzadeh-sd

This comment was marked as off-topic.

Copy link

commented Apr 27, 2019

create exe without virtualenv , worked for me.

@AlmogCohen

This comment has been minimized.

Copy link

commented May 7, 2019

Same exception while running pipenv + python 3.7. Couldn't quite understand how to solve that yet...

This is the pyinstaller output if anybody happen to understand how to solve that:

INFO:project_packer.packer:Removing 0 old egg files.
INFO:project_packer.packer:creating egg file for ../project-framework with: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/python3.7 setup.py bdist_egg --dist-dir /Users/username/Clients/company/code/project/project-packer
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

INFO:project_packer.packer:egg file created successfully for project-framework
INFO:project_packer.packer:creating egg file for ../pyucwa with: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/python3.7 setup.py bdist_egg --dist-dir /Users/username/Clients/company/code/project/project-packer
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

INFO:project_packer.packer:egg file created successfully for pyucwa
Running pyinstaller command: /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/pyinstaller --clean --name project --additional-hooks-dir=/Users/username/Clients/company/code/project/project-packer/project_packer/pyinstaller_hooks --distpath ./project --onefile --paths pyucwa-1.2.1-py3.7.egg --paths project_framework-1.1.1-py3.7.egg /Users/username/Clients/company/code/project/project-packer/project_packer/project.py
53 INFO: PyInstaller: 3.4
53 INFO: Python: 3.7.2
60 INFO: Platform: Darwin-18.5.0-x86_64-i386-64bit
60 INFO: wrote /Users/username/Clients/company/code/project/project-packer/project.spec
64 INFO: UPX is not available.
65 INFO: Removing temporary files and cleaning cache in /Users/username/Library/Application Support/pyinstaller
84 INFO: Extending PYTHONPATH with paths
['/Users/username/Clients/company/code/project/project-packer',
 '/Users/username/Clients/company/code/project/project-packer/pyucwa-1.2.1-py3.7.egg',
 '/Users/username/Clients/company/code/project/project-packer/project_framework-1.1.1-py3.7.egg',
 '/Users/username/Clients/company/code/project/project-packer']
84 INFO: checking Analysis
84 INFO: Building Analysis because Analysis-00.toc is non existent
84 INFO: Initializing module dependency graph...
87 INFO: Initializing module graph hooks...
88 INFO: Analyzing base_library.zip ...
3123 INFO: running Analysis Analysis-00.toc
3131 INFO: Caching module hooks...
3134 INFO: Analyzing /Users/username/Clients/company/code/project/project-packer/project_packer/project.py
3568 INFO: Processing pre-find module path hook   distutils
3568 INFO: distutils: retargeting to non-venv dir '/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/__init__.py'
4011 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
5045 INFO: Processing pre-safe import module hook   six.moves
8187 INFO: Loading module hooks...
8187 INFO: Loading module hook "hook-pkg_resources.py"...
8703 INFO: Processing pre-safe import module hook   win32com
8706 INFO: Loading module hook "hook-babel.py"...
8766 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
8766 INFO: Loading module hook "hook-pytz.py"...
8797 INFO: Loading module hook "hook-cryptography.py"...
9170 INFO: Loading module hook "hook-encodings.py"...
9291 INFO: Loading module hook "hook-certifi.py"...
9293 INFO: Loading module hook "hook-sysconfig.py"...
9304 INFO: Loading module hook "hook-xml.py"...
9305 INFO: Loading module hook "hook-pydoc.py"...
9305 INFO: Loading module hook "hook-websocket.py"...
9377 INFO: checking Tree
9377 INFO: Building Tree because Tree-00.toc is non existent
9377 INFO: Building Tree Tree-00.toc
9383 INFO: checking Tree
9384 INFO: Building Tree because Tree-01.toc is non existent
9384 INFO: Building Tree Tree-01.toc
9384 INFO: Looking for ctypes DLLs
9402 WARNING: library user32 required via ctypes not found
9413 INFO: Analyzing run-time hooks ...
9425 INFO: Including run-time hook 'pyi_rth_pkgres.py'
9426 INFO: Including run-time hook 'pyi_rth_multiprocessing.py'
9453 INFO: Looking for dynamic libraries
9567 INFO: Looking for eggs
9567 INFO: Using Python library /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/bin/../.Python
9575 INFO: Warnings written to /Users/username/Clients/company/code/project/project-packer/build/project/warn-project.txt
9661 INFO: Graph cross-reference written to /Users/username/Clients/company/code/project/project-packer/build/project/xref-project.html
9757 INFO: checking PYZ
9758 INFO: Building PYZ because PYZ-00.toc is non existent
9758 INFO: Building PYZ (ZlibArchive) /Users/username/Clients/company/code/project/project-packer/build/project/PYZ-00.pyz
10958 INFO: Building PYZ (ZlibArchive) /Users/username/Clients/company/code/project/project-packer/build/project/PYZ-00.pyz completed successfully.
10990 INFO: checking PKG
10990 INFO: Building PKG because PKG-00.toc is non existent
10990 INFO: Building PKG (CArchive) PKG-00.pkg
19258 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
19308 INFO: Bootloader /Users/username/.local/share/virtualenvs/project-packer-iDyFrN8L/lib/python3.7/site-packages/PyInstaller/bootloader/Darwin-64bit/run
19308 INFO: checking EXE
19308 INFO: Building EXE because EXE-00.toc is non existent
19308 INFO: Building EXE from EXE-00.toc
19308 INFO: Appending archive to EXE ./project/project
19334 INFO: Fixing EXE for code signing ./project/project
19338 INFO: Building EXE from EXE-00.toc completed successfully.
@AlmogCohen

This comment has been minimized.

Copy link

commented May 7, 2019

@oberstet You rock!!!!!! Your solution made it! Would it be assimilated into pyinstaller officially?

@truuuuegate

This comment has been minimized.

Copy link

commented May 27, 2019

Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.

Thank you! This solution solve my problem in pipenv+python3.7+pyinstaller3.4

@jmbruges

This comment was marked as off-topic.

Copy link

commented May 27, 2019

@truuuuegate can you tell me where excatly did you made the change? I have my app.spec and I add these lines. However, nothing change.
I am using the same solution (pipenv+P3+Pyinst)
Should I add these lines in my app.py?

@truuuuegate

This comment has been minimized.

Copy link

commented May 27, 2019

@jmbruges
I add that code at the beginning of the *.spec file.
Then the pyinstaller process in pipenv could find the distutils module path that in native enviroment.

@jmbruges

This comment has been minimized.

Copy link

commented May 27, 2019

@jmbruges
I add that code at the beginning of the *.spec file.
Then the pyinstaller process in pipenv could find the distutils module path that in native enviroment.

Thanks, I tried that and did not work. I will see if cx_freeze can resolve it but so far I have other issues trying to install that package.

zacker150 added a commit to zacker150/DownloaderForReddit that referenced this issue Jun 28, 2019

paulmueller added a commit to ZELLMECHANIK-DRESDEN/ShapeOut2 that referenced this issue Jul 3, 2019

@asakiasako

This comment has been minimized.

Copy link

commented Jul 12, 2019

@maxslimmer 's solution (thanks!) worked for me (on a cpy373 based venv), as in (it needs to be in a pre_find_module_path subdir within the hooks dir):

(cpy373_1) oberstet@intel-nuci7:~/scm/crossbario/crossbarfx$ cat pyinstaller/pre_find_module_path/hook-distutils.py 
import distutils
import os

from PyInstaller.utils.hooks import logger

# https://github.com/pyinstaller/pyinstaller/issues/4064
# https://pythonhosted.org/PyInstaller/hooks.html#the-pre-find-module-path-pfmp-api-method

def pre_find_module_path(api):
    # Absolute path of the system-wide "distutils" package when run from within
    # a venv or None otherwise.
    distutils_dir = getattr(distutils, 'distutils_path', None)
    if distutils_dir is not None:
        # workaround for https://github.com/pyinstaller/pyinstaller/issues/4064
        if distutils_dir.endswith('__init__.py'):
            distutils_dir = os.path.dirname(distutils_dir)

        # Find this package in its parent directory.
        api.search_dirs = [os.path.dirname(distutils_dir)]

        logger.info('>>>>>>> CUSTOM >>>>>>>>> distutils: retargeting to non-venv dir %r' % distutils_dir)

Thanks for your tip, I searched pyinstaller docs and finally understand how it works

@htgoebel

This comment has been minimized.

Copy link
Member

commented Aug 7, 2019

Since there is a solution pointed out here, anybody going to create a pull-request? See our Development Guide on how to create pull-requests for PyInstaller.

@tangm

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2019

I ran into this, but I'm not sure continuing to rely on the internal distutils_path is a good idea... maybe the hook should be doing its own system distutils detection instead? Something like:

def pre_find_module_path(api):
    # opcode is not a virtualenv module, so we can use it to find the stdlib
    # Technique taken from virtualenv's "distutils" package detection at
    # https://github.com/pypa/virtualenv/blob/0ab032ce1a/virtualenv_embedded/distutils-init.py#L5
    import opcode

    system_module_path = os.path.normpath(os.path.dirname(opcode.__file__))
    loaded_module_path = os.path.normpath(os.path.dirname(distutils.__file__))
    if system_module_path != loaded_module_path:
        # Find this package in its parent directory.
        api.search_dirs = [system_module_path]
        logger.info('distutils: retargeting to non-venv dir %r' % system_module_path)

tangm added a commit to tangm/pyinstaller that referenced this issue Aug 11, 2019

@htgoebel

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

@tangm I'm still waiting for someone to provide a pull-request. Then we can discuss teh code there.

@htgoebel

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

Also I'd appreciate a summary of the caus for this issue, so I can understand and comment on the solution.

@tangm

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2019

@htgoebel I've created #4372 , with a description of the problem.

@mxnsrengler

This comment has been minimized.

Copy link

commented Aug 14, 2019

Thanks @tangm for your pull request. I was able to implement your code myself and fix the distutils issue for me!

@htgoebel htgoebel closed this in 46286a1 Aug 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.