Scoop Python is not supported yet

[bersbersbers]

• This is a regression, not already reported or fixed. Same thing with develop (08979ec).

2.0rc1
Commercial: None
Python: 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit 
(AMD64)]
Flavor: Unknown
Executable: c:\Build\project\.venv\Scripts\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 10
Version C compiler: ~\AppData\Local\Nuitka\Nuitka\Cache\downloads\gcc\x86_64\13.2.0-16.0.6-11.0.1-msvcrt-r1\mingw64\bin\gcc.exe (gcc 13.2.0).

• How did you install Nuitka and Python

pip install nuitka in an activated .venv

PySide6-Essentials==6.6.0

bug.py

# nuitka-project: --standalone
# nuitka-project: --static-libpython=no
# nuitka-project: --enable-plugin=pyside6
# nuitka-project: --nofollow-import-to=tkinter
# nuitka-project: --nofollow-import-to=pytest

from PySide6.QtWidgets import QApplication

QApplication()
print("Done")

Then, nuitka bug.py && bug.dist\bug.exe:

PySide6/__init__.py: Unable to import Shiboken from C:\Build\project\bug.dist, C:\Build\project\BUG~1.DIS
Traceback (most recent call last):
  File "C:\Build\project\BUG~1.DIS\bug.py", line 7, in <module>
  File "C:\Build\project\BUG~1.DIS\PySide6\__init__.py", line 124, in <module PySide6>
  File "C:\Build\project\BUG~1.DIS\PySide6\__init__.py", line 64, in _setupQtDirectories
  File "C:\Build\project\BUG~1.DIS\shiboken6\__init__.py", line 27, in <module shiboken6>
ImportError: LoadLibraryExW 'C:\Build\project\BUG~1.DIS\shiboken6\Shiboken.pyd' 
failed: The specified module could not be found.

However:

dir bug.dist\shiboken6\Shiboken.pyd

 Directory of C:\Build\project\bug.dist\shiboken6

11/23/2023  05:58 PM            31,888 Shiboken.pyd
               1 File(s)         31,888 bytes

Process Manager says it is looking for python3.dll, which is missing in bug.dist. When building with 1.8.6, that file is present.

[kayhayen]

That's strange, it worked just now for me. Maybe a strange form of a bug with DLL dependency caching, but it considered the Nuitka version as part of the key, or so I am thinking. Works for me with 3.11 CPython and 6.5 and 6.6 just fine, maybe it's somehow related to the venv you created, although I think Nuitka-Watch, where this has a record of working with a late 1.9rc with no meaningful changes afterwards, is run in one.

[bersbersbers]

Can I help somehow? Compilation reports, or a tree of the venv? I install python via scoop now, if that matters:

C:\Scoop>dir python3.dll /s
 Directory of C:\Scoop\apps\python\3.12.0

10/02/2023  01:27 PM            68,376 python3.dll
               1 File(s)         68,376 bytes

 Directory of C:\Scoop\apps\python\current

10/02/2023  01:27 PM            68,376 python3.dll
               1 File(s)         68,376 bytes

 Directory of C:\Scoop\apps\python311\3.11.6

10/02/2023  03:18 PM            67,352 python3.dll
               1 File(s)         67,352 bytes

 Directory of C:\Scoop\apps\python311\current

10/02/2023  03:18 PM            67,352 python3.dll
               1 File(s)         67,352 bytes

But I would think that this does not matter once the venv is created (which I do via VS Code).

[kayhayen]

I install python via scoop now, if that matters:

Maybe it does. Did you check if Nuitka supports that? You can identify the differences to CPython and make PR to add support for it. First step would be how to identify it and not have Flavor: Unknown which I overlooked.

[bersbersbers]

Did you check if Nuitka supports that?

Not explicitly, but since Nuitka 1.8.6 works fine, I wonder if that should be enough of an indication. I will try again tomorrow: maybe git bisect is the best approach here since the problem reproduces in less than a minute.

[bersbersbers]

This is how I bisected:

# del bug.dist /s /q & del bug.build /s /q & pip install -e . && nuitka bug.py && bug.dist\bug.exe

# nuitka-project: --standalone
# nuitka-project: --static-libpython=no
# nuitka-project: --enable-plugin=pyside6
# nuitka-project: --enable-console

from PySide6.QtWidgets import QApplication

QApplication()
print("Done")

And this is the result:

git bisect start
# status: waiting for both good and bad commits
# bad: [1a6fc1e0e15fb8150269dcd38568c648ae1cf8ba] New pre-release.
git bisect bad 1a6fc1e0e15fb8150269dcd38568c648ae1cf8ba
# status: waiting for good commit(s), bad commit known
# good: [de0987cea49ce85fc3031438a72be4a931e003a6] Merge branch 'hotfix/1.8.6'
git bisect good de0987cea49ce85fc3031438a72be4a931e003a6
# good: [ac07b4b560107e0c61b9c2de2a99e1642f6f900d] Changelog update
git bisect good ac07b4b560107e0c61b9c2de2a99e1642f6f900d
# bad: [64b9ac41840467e2a88c9a9b628786345a4c33cd] New pre-release.
git bisect bad 64b9ac41840467e2a88c9a9b628786345a4c33cd
# bad: [32d7c5c3c684559f9e5af18aa284e6301c8a821a] Anti-Bloat: Avoid using "IPython" in "gradio" package
git bisect bad 32d7c5c3c684559f9e5af18aa284e6301c8a821a
# good: [507fecd5da744a3f70926b46c906f1aded4f3cd6] UI: Output clang and gcc versions in help output
git bisect good 507fecd5da744a3f70926b46c906f1aded4f3cd6
# bad: [52a472d19d167438d6baeb568c9de4d480be10cb] Standalone: Massive DLL inclusion cleanups
git bisect bad 52a472d19d167438d6baeb568c9de4d480be10cb
# skip: [55217d99f9b3455b9c5dc54ea4f3abb45df9c53e] Compatibility: Added support for metadata from pip "pkg_resources"
git bisect skip 55217d99f9b3455b9c5dc54ea4f3abb45df9c53e
# skip: [7cb7f1f52a6c8543af20e0b261d41ab4563b84c2] Python3.11: Pylint fixups
git bisect skip 7cb7f1f52a6c8543af20e0b261d41ab4563b84c2
# skip: [cedcef1a4467c3ad4a4cc6057a564a24ce6353c6] Python3.11: Added support for deactivating frozen modules at accelerated runtime
git bisect skip cedcef1a4467c3ad4a4cc6057a564a24ce6353c6
# good: [5b19cda096b38556284cb0e1902b20764d6e2de3] Release: Added ability to be version about container build
git bisect good 5b19cda096b38556284cb0e1902b20764d6e2de3
# skip: [f6cc0c2b9466b6493c74f4dafdcd40c49f7cfeb5] UI: Do not consider aliases of options for ambiguous option error
git bisect skip f6cc0c2b9466b6493c74f4dafdcd40c49f7cfeb5
# only skipped commits left to test
# possible first bad commit: [52a472d19d167438d6baeb568c9de4d480be10cb] Standalone: Massive DLL inclusion cleanups
# possible first bad commit: [f6cc0c2b9466b6493c74f4dafdcd40c49f7cfeb5] UI: Do not consider aliases of options for ambiguous option error
# possible first bad commit: [cedcef1a4467c3ad4a4cc6057a564a24ce6353c6] Python3.11: Added support for deactivating frozen modules at accelerated runtime
# possible first bad commit: [7cb7f1f52a6c8543af20e0b261d41ab4563b84c2] Python3.11: Pylint fixups
# possible first bad commit: [55217d99f9b3455b9c5dc54ea4f3abb45df9c53e] Compatibility: Added support for metadata from pip "pkg_resources"

I had to skip a couple of commits close to the end due to this error:

Traceback (most recent call last):
  File "C:\Build\Nuitka\nuitka\__main__.py", line 163, in <module>
    main()
  File "C:\Build\Nuitka\nuitka\__main__.py", line 147, in main
    from nuitka import MainControl  # isort:skip
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Build\Nuitka\nuitka\MainControl.py", line 123, in <module>
    from .reports.Reports import writeCompilationReports
  File "C:\Build\Nuitka\nuitka\reports\Reports.py", line 52, in <module>
    from nuitka.utils.Distributions import (
ImportError: cannot import name 'getDistributionVersion' from 'nuitka.utils.Distributions' (C:\Build\Nuitka\nuitka\utils\Distributions.py)

But that's not too bad, I think, given that [52a472d] Standalone: Massive DLL inclusion cleanups looks very related.

[bersbersbers]

You can identify the differences to CPython and make PR to add support for it. First step would be how to identify it and not have Flavor: Unknown which I overlooked.

Okay, this is a two-fold issue.

1. By default, scoop does not modify the registry. At all. Meaning, your detection using registry keys must fail, explaining the "Unknown" flavor.
2. Scoop does prepare an install-pep-514.reg file for each installed version of Python which, after merging, makes nuitka detect Python as "CPython Official":
   See https://github.com/ScoopInstaller/Main/blob/master/bucket/python.json or here:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Python\PythonCore]
"DisplayName"="Official Python installed with Scoop"
"SupportUrl"="https://www.python.org/"

[HKEY_CURRENT_USER\Software\Python\PythonCore\3.11]
"DisplayName"="Python 3.11 (64-bit)"
"SupportUrl"="https://www.python.org/"
"Version"="3.11.6"
"SysVersion"="3.11"
"SysArchitecture"="64bit"

[HKEY_CURRENT_USER\Software\Python\PythonCore\3.11\Help\Main Python Documentation]
@="C:\\Scoop\\apps\\python311\\3.11.6\\Doc\\python3116.chm"
"DisplayName"="Python 3.11.6 Documentation"

[HKEY_CURRENT_USER\Software\Python\PythonCore\3.11\InstallPath]
@="C:\\Scoop\\apps\\python311\\3.11.6"
"ExecutablePath"="C:\\Scoop\\apps\\python311\\3.11.6\\python.exe"
"WindowedExecutablePath"="C:\\Scoop\\apps\\python311\\3.11.6\\pythonw.exe"

[HKEY_CURRENT_USER\Software\Python\PythonCore\3.11\PythonPath]
@="C:\\Scoop\\apps\\python311\\3.11.6\\Lib\\;C:\\Scoop\\apps\\python311\\3.11.6\\DLLs\\"

There is potential to detect this using "DisplayName"="Official Python installed with Scoop", for example, but that is only if the user chooses to merge that registry file.

I will not be able to dig much deeper than this, unfortunately. I can add that my .venv does not have a python3.dll - not sure if it should have one. In the Scoop directory, I see that file just next to python.exe:

And this path is what I see as sys_prefix as well when debugging nuitka --version:

Final bit of information: Scoop uses junctions named current to provide stable paths to executables. This has been a problem elsewhere before (pyinstaller/pyinstaller#8023) - and indeed, my .venv refers to the Python installation under two different paths.

 Directory of C:\Scoop\apps\python311

11/23/2023  02:13 PM    <DIR>          .
11/23/2023  02:13 PM    <DIR>          ..
11/23/2023  02:13 PM    <DIR>          3.11.6
11/23/2023  02:13 PM    <JUNCTION>     current [\??\C:\Scoop\apps\python311\3.11.6]

home = c:\Scoop\apps\python311\current
include-system-site-packages = false
version = 3.11.6
executable = C:\Scoop\apps\python311\3.11.6\python.exe
command = c:\Scoop\apps\python311\current\python.exe -m venv c:\Build\project\.venv

I guess nuitka might be susceptible to seeing these paths as different. Indeed:

>>> from nuitka.utils import FileOperations
>>> FileOperations.areSamePaths(r"C:\Scoop\apps\python311\3.11.6", r"C:\Scoop\apps\python311\current")
False

By adding

    path1 = os.path.realpath(path1)
    path2 = os.path.realpath(path2)

to areSamePaths, that comparison operation can be fixed (since Python 3.8).

That does not seem to fix the python3.dll problem, but maybe it gives you an idea where else to use realpath.

[bersbersbers]

Haha, that was the right track! This fixes the issue:

diff --git a/nuitka/utils/FileOperations.py b/nuitka/utils/FileOperations.py
index dba50455c..a1934ed78 100644
--- a/nuitka/utils/FileOperations.py
+++ b/nuitka/utils/FileOperations.py
@@ -1121,6 +1121,8 @@ def getExternalUsePath(filename, only_dirname=False): 

     filename = os.path.abspath(filename)

+    filename = os.path.realpath(filename)
+
     if os.name == "nt":
         if filename not in _external_use_path_cache:
             asked_filename = filename

[kayhayen]

Nuitka supports links to the python prefix for a while already, I used to used them myself a lot, for a default between 64 and 32 bits and getDirectoryRealPath makes it work with older Python even, some of which do not have proper symlink support.

23.02.2023  18:56    <SYMLINK>      Python39 [C:\Python39_64]
25.01.2023  11:35    <DIR>          Python39_32
25.01.2023  11:33    <DIR>          Python39_64

Now I am not sure if it's that (will try) or if junctions are not supported by islink tests done, or if with the recent fix to use external path for python_prefix towards scons something sneaked in, or is still missing for dependency walker, which is also possible.

[kayhayen]

btw, if you have a pointer to the commits you needed to skip, that would be nice. I cherish bitseting a lot, and would have be victim of it myself later, so I would want to rectify this as soon as possible.

[kayhayen]

So, just making the Python prefix for 3.9 a junction, and clearing all caches, does indeed reproduce the issue, which is absolutely interesting. When googling up, I also noticed that symlinks are not nearly recommendable.

I do not want to mix symlink resolution and external use in this way, unless where it's necessary. For the python prefix, it absolutely is, for other things, maybe not so much.

In terms of fixit it, I think I need to make compilation reports capable of pointing out the difference between using the junction and not. From my experiment, python_prefix is resolved, but I guess, sys.path probably is not, that kind of thing.

[bersbersbers]

a pointer to the commits you needed to skip

Sure - all that info is in the bisect log in #2568 (comment).

[kayhayen]

So, I think I noticed what's going on, the sys.path retaining the original filename with the junction is fully desired, and wanted, and scan directories already are properly covered, but when dependency walker it being called on a DLL, the original filename of that DLL (e.g. extension module) is used on the command line, which is a small issue by itself, this could e.g. be a Chinese module name, that then wouldn't work.

The external use path indeed can be using Junctions, and my suspect is that dependency walker just doesn't handle that, and our function to resolve symlinks is not really good at it, only doing the last level, and not all the way up, which os.path.realpath does. I think I will want to add a recursive usage of it, inside getExternalUsePath such that it is going to work independent of the level of symlink support in the used Python, that shouldn't be too hard.

[kayhayen]

Unfortunately, e.g. macOS hates symlinks to be resolved, it will not find its stuff anymore. Need to further refine this to be limited to Windows I guess, since every other system has ideas that symlinks are the real thing.

[kayhayen]

Updated factory with a version that limits the symlink resolution for external paths to Windows only. Not so easy to have the "correct" path everywhere.

I also rebased develop and main to not have the issue found during your bisect, no change from that. I still intend to put this into 1.9.2 but it feels a bit risky I guess.

[kayhayen]

Part of the 1.9.3 release I just made.

[kayhayen]

@ManPython posting below closed issues is a sure fire way to get ignored