Some hooks should get fixed #6118

stormbeforesunsetbee · 2021-08-13T05:58:07Z

Description of the issue

It returns ModuleNotFoundError: No module named '_sysconfigdata__win32_' at from sklearn.cluster import MiniBatchKMeans for some reason

Context information (for bug reports)

Output of pyinstaller --version: 4.5.1
Version of Python: Python 3.9.6
Platform: MSYS2 MINGW64
How you installed Python: from MSYS2, the MINGW64 one
Did you also try this on another platform? Does it work there? Nope
follow all the instructions in our "If Things Go Wrong" Guide
(https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

start with clean installation
use the latest development version
Run your frozen program from a command window (shell) — instead of double-clicking on it But it didn't catch the program, like it runs it in separate process. I use windowed mode, anyways
Package your program in --onedir mode
Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

window.py

import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gio
import os
import gc
import numpy as np
import pandas as pd
from sklearn.cluster import MiniBatchKMeans
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

@Gtk.Template(resource_path='/org/example/App/window.ui')
class ClusterifyWindow(Gtk.ApplicationWindow):
    __gtype_name__ = 'ClusterifyWindow'
    full_initialized = False

    ...

    def __init__(self, log_error, **kwargs):
        super().__init__(**kwargs)
        self.log_error = log_error

    @Gtk.Template.Callback()
    def on_ClusterifyWindow_focus_in_event(self, cb_widget, data):
        if not self.full_initialized:
            import matplotlib
            matplotlib.use("GTK3Cairo")
            import matplotlib.pyplot as plt
            import seaborn as sns
            sns.set()
            from matplotlib.backends.backend_gtk3cairo import FigureCanvasGTK3Cairo as FigureCanvas

            self.fi_clusters = plt.figure()
            self.cv_clusters = FigureCanvas(self.fi_clusters)
            self.cv_clusters.show_all()

            ...

        self.full_initialized = True

clusterify.spec

# -*- mode: python ; coding: utf-8 -*-

typelib_path = "C:/msys64/mingw64/lib/girepository-1.0"
block_cipher = None


a = Analysis(['C:/msys64/mingw64/bin/clusterify'],
             pathex=['C:/msys64/home/rivan/Clusterify'],
             binaries=[(os.path.join(typelib_path, tl), 'gi_typelibs') for tl in os.listdir(typelib_path)],
             datas=[],
             hiddenimports=['numpy',
                            'pandas',
                            'sklearn.cluster', 'sklearn.preprocessing', 'sklearn.decomposition',
                            'matplotlib', 'matplotlib.backends.backend_gtk3cairo',
                            'seaborn'],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,  
          [],
          name='clusterify',
          debug=True,
          bootloader_ignore_signals=False,
          strip=False,
          upx=False,
          upx_exclude=[],
          runtime_tmpdir=None,
          console=False,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )

Stacktrace / full error message

Traceback (most recent call last):
  File "clusterify", line 51, in <module>
  File "C:\msys64/mingw64/share/clusterify\clusterify\main.py", line 33, in <module>
    from .window import ClusterifyWindow
  File "C:\msys64/mingw64/share/clusterify\clusterify\window.py", line 36, in <module>
    from sklearn.cluster import MiniBatchKMeans
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/__init__.py", line 82, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/base.py", line 20, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/utils/__init__.py", line 24, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/utils/_joblib.py", line 7, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "joblib/__init__.py", line 113, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "joblib/memory.py", line 16, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "pydoc.py", line 458, in <module>
  File "pydoc.py", line 488, in Doc
  File "sysconfig.py", line 554, in get_path
  File "sysconfig.py", line 544, in get_paths
  File "sysconfig.py", line 179, in _expand_vars
  File "sysconfig.py", line 594, in get_config_vars
  File "sysconfig.py", line 460, in _init_posix
ModuleNotFoundError: No module named '_sysconfigdata__win32_'

The text was updated successfully, but these errors were encountered:

stormbeforesunsetbee · 2021-08-13T06:26:09Z

The #3198 solution doesn't work either. Here's the build output:

10124 INFO: Analyzing hidden import '_sysconfigdata'
10124 ERROR: Hidden import '_sysconfigdata' not found

I think it's related to msys2/MINGW-packages#9075

rokm · 2021-08-13T09:37:10Z

The hiddenimport should be _sysconfigdata__win32_ (i.e., the verbatim name of the reported missing module).

This should be picked up automatically by our sysconfig hook, except we skip the collection on all Windows flavors (presumably because sysconfig._get_sysconfigdata_name() call fails on regular Windows python):

pyinstaller/PyInstaller/hooks/hook-sysconfig.py

Lines 17 to 23 in cde2448

    
           if not is_win and hasattr(sysconfig, '_get_sysconfigdata_name'): 
        
               # Python 3.6 uses additional modules like 
        
               # `_sysconfigdata_m_linux_x86_64-linux-gnu`, see 
        
               # https://github.com/python/cpython/blob/3.6/Lib/sysconfig.py#L417 
        
               # Note: Some versions of Anaconda backport this feature to before 3.6. 
        
               # See issue #3105 
        
               hiddenimports = [sysconfig._get_sysconfigdata_name()]

I suppose we should omit the platform check and try to call sysconfig._get_sysconfigdata_name() within a try-except block, which will also help if this extra data module ever becomes valid on regular Windows python...

stormbeforesunsetbee · 2021-08-14T10:24:03Z

Thank you so much! It works.

I don't know much about this tool, but I think you guys should make a "hook" for sklearn.

EDIT:
Also, make some documentation / tutorial about this :D

bwoodsend · 2021-08-14T11:10:42Z

We'll keep this open until that hook has been fixed. Otherwise it'll just get forgotten about.

stormbeforesunsetbee · 2021-08-14T11:16:15Z

Oh, thanks! Anyways, I still have a problem with this hook: 342ecac

My theme icons isn't loaded although I add this snippet already:

             hooksconfig = {
             "gi": {
                 "icons": ["Fluent"],
                 "themes": ["Fluent"],
                 "languages": ["en_US"],
             }},

Here's my GTK config (C:\msys64\mingw64\etc\gtk-3.0\settings.ini)

[Settings]
gtk-theme-name=Fluent
gtk-icon-theme-name=Fluent

EDIT:
Fixed by adding the above file :D

a = Analysis(['C:/msys64/mingw64/bin/clusterify'],

...

      datas=[('C:\msys64\mingw64\etc\gtk-3.0', '/etc/gtk-3.0'),],

...

)

Under MSYS2/MINGW, we need to collect the platform-specific data module obtained via sysconfig._get_sysconfigdata_name(). That call fails under other python flavors on Windows, but that is better handled by try/except block anyway. Fixes pyinstaller#6118.

) Under MSYS2/MINGW, we need to collect the platform-specific data module obtained via sysconfig._get_sysconfigdata_name(). That call fails under other python flavors on Windows, but that is better handled by try/except block anyway. Fixes #6118.

stormbeforesunsetbee · 2021-08-21T13:38:23Z

Guys! Please reopen this issue... It still doesn't work: https://github.com/rivanfebrian123/Clusterify/actions/runs/1153665274

Traceback (most recent call last):
  File "clusterify", line 59, in <module>
  File "C:\Users\rivan\AppData\Local\Programs\Clusterify\share\clusterify\clusterify\main.py", line 34, in <module>
    from .window import ClusterifyWindow
  File "C:\Users\rivan\AppData\Local\Programs\Clusterify\share\clusterify\clusterify\window.py", line 33, in <module>
    from .data import Data
  File "C:\Users\rivan\AppData\Local\Programs\Clusterify\share\clusterify\clusterify\data.py", line 33, in <module>
    from sklearn.cluster import MiniBatchKMeans
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/__init__.py", line 82, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/base.py", line 20, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/utils/__init__.py", line 24, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "sklearn/utils/_joblib.py", line 7, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "joblib/__init__.py", line 113, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "joblib/memory.py", line 16, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 546, in exec_module
  File "pydoc.py", line 458, in <module>
  File "pydoc.py", line 488, in Doc
  File "sysconfig.py", line 554, in get_path
  File "sysconfig.py", line 544, in get_paths
  File "sysconfig.py", line 179, in _expand_vars
  File "sysconfig.py", line 594, in get_config_vars
  File "sysconfig.py", line 460, in _init_posix
ModuleNotFoundError: No module named '_sysconfigdata__win32_'

The previous commits before I remove _sysconfigdata__win32_ worked just fine

rokm · 2021-08-21T13:45:00Z

https://github.com/rivanfebrian123/Clusterify/blob/65c50b83683152b264d37fe79e8857b9dce95635/msys2_deploy.sh#L28

You're using last release of PyInstaller, and the fix is not part of a release yet...

stormbeforesunsetbee · 2021-08-21T13:49:35Z

Oh... got it. Thanks!

stormbeforesunsetbee added the triage Please triage and relabel this issue label Aug 13, 2021

rokm added bug and removed triage Please triage and relabel this issue labels Aug 13, 2021

stormbeforesunsetbee closed this as completed Aug 14, 2021

bwoodsend reopened this Aug 14, 2021

stormbeforesunsetbee changed the title ~~ModuleNotFoundError: No module named '_sysconfigdata__win32_' at from sklearn.cluster import MiniBatchKMeans for some reason~~ Some hooks should get fixed Aug 14, 2021

rokm mentioned this issue Aug 20, 2021

hooks: sysconfig: fix collection of data module under MSYS2/MINGW #6144

Merged

rokm closed this as completed in #6144 Aug 20, 2021

github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some hooks should get fixed #6118

Some hooks should get fixed #6118

stormbeforesunsetbee commented Aug 13, 2021

stormbeforesunsetbee commented Aug 13, 2021

rokm commented Aug 13, 2021

stormbeforesunsetbee commented Aug 14, 2021 •

edited

Loading

bwoodsend commented Aug 14, 2021 •

edited

Loading

stormbeforesunsetbee commented Aug 14, 2021 •

edited

Loading

stormbeforesunsetbee commented Aug 21, 2021

rokm commented Aug 21, 2021

stormbeforesunsetbee commented Aug 21, 2021

Some hooks should get fixed #6118

Some hooks should get fixed #6118

Comments

stormbeforesunsetbee commented Aug 13, 2021

Description of the issue

Context information (for bug reports)

Make sure everything is packaged correctly

A minimal example program which shows the error

window.py

clusterify.spec

Stacktrace / full error message

stormbeforesunsetbee commented Aug 13, 2021

rokm commented Aug 13, 2021

stormbeforesunsetbee commented Aug 14, 2021 • edited Loading

bwoodsend commented Aug 14, 2021 • edited Loading

stormbeforesunsetbee commented Aug 14, 2021 • edited Loading

stormbeforesunsetbee commented Aug 21, 2021

rokm commented Aug 21, 2021

stormbeforesunsetbee commented Aug 21, 2021

stormbeforesunsetbee commented Aug 14, 2021 •

edited

Loading

bwoodsend commented Aug 14, 2021 •

edited

Loading

stormbeforesunsetbee commented Aug 14, 2021 •

edited

Loading