how to include dll that lives in ~\Lib

[KRRT7]

Hi kay, you asked me to open an issue for this
regarding #2649
gmsh requires an DLL that lives in ~\Lib next to site-packages which is problematic since gmsh-4.12.dll normally would be in it's own module namespace.

 Directory of C:\Users\Divine\Desktop\OSS contributions\nuitka-2649-gmsh\venv\Lib

01/15/2024  02:37 PM    <DIR>          .
01/15/2024  02:37 PM    <DIR>          ..
01/15/2024  02:37 PM        86,433,792 gmsh-4.12.dll
01/15/2024  02:37 PM           356,182 gmsh.jl
01/15/2024  02:37 PM        41,059,108 gmsh.lib
01/31/2024  02:26 AM    <DIR>          site-packages

[kayhayen]

Really terrible. I think, the plugin code ought to protect for relative paths to go outside. I believe we would have to use by_code here, or maybe the new variables. Once thing, though, the directory of the current module or package, it probably should be added by variable, such that then we get to do get_variable("module_directory") + "/.." in there if you know what I mean. I will have something for that shortly I hope, and even 2.0 capable.

[kayhayen]

I mean something that can be a in a hotfix.

[kayhayen]

I might be drawing false conclusion, I guess I need to first check that the code actually uses ".." to get there, and doesn't place it in the DLL lib folder, meaning a DLL folder, but I think for that it would be wrong, DLLs are not searched in Lib or so I am thinking.

[kayhayen]

So on Linux it adds those gmsh.jl and libgmsh.so.4.12 in the virtualenv lib folder.

What a beauty:

# Searching lib in various subfolders
libpath = None
possible_libpaths = [os.path.join(moduledir, libname),
                     os.path.join(moduledir, "lib", libname),
                     os.path.join(moduledir, "Lib", libname),
                     # first parent dir
                     os.path.join(parentdir1, libname),
                     os.path.join(parentdir1, "lib", libname),
                     os.path.join(parentdir1, "Lib", libname),
                     # second parent dir
                     os.path.join(parentdir2, libname),
                     os.path.join(parentdir2, "lib", libname),
                     os.path.join(parentdir2, "Lib", libname)
                     ]

for libpath_to_look in possible_libpaths:
    if os.path.exists(libpath_to_look):
        libpath = libpath_to_look
        break

That probably means, if we put it top level below lib it's going to be good. Any idea what that gmsh.jl file is, do we have to include it as well? From the looks of the code, I would say, they place it at ../.. during install, which is oh so ugly. I am tempted to query the code for that path, such that if they change there mind ever, we don't rely on that, and also use the libname value, which is probably subject to change.

[kayhayen]

- module-name: gmsh
  dlls:
    - by_code:
        setup_code: 'import gmsh'
        filename_code: "gmsh.libpath"
      dest_path: 'lib'

That ought to work, it is an expression that gives the actual DLL filename, and the target path is then lib where it should be picked up. In that way, we don't care at all about their madness.

[KRRT7]

gmsh.jl is a julia lang source code file, and i get the feeling it is needed by the dll.

[kayhayen]

@KevinRodriguez777 maybe give that a shot, I am positive, it would have to work, all platforms, etc.

[kayhayen]

Try placing it in the lib folder as well. I didn't see any code for using it in the Python. If it's used by the DLL, probably loads it from next to itself. For data files, we kind of are in a pit there too. I don't think for data files, going outside of a module space is considered OK, and if not now, it should be added as a bug, to go outside of an installation potentially. Maybe that's where we have to do variables rather than the DLL code. I have the plan to replace by_code with variables eventually anyway, but I thought it's maybe not needed.

[KRRT7]

- module-name: gmsh
  dlls:
    - by_code:
        setup_code: 'import gmsh'
        filename_code: "gmsh.libpath"
      dest_path: 'lib'

That ought to work, it is an expression that gives the actual DLL filename, and the target path is then lib where it should be picked up. In that way, we don't care at all about their madness.

works correctly on windows.
on linux however
(venv) rizesquad@bww:~/gmsh$ python -m nuitka --standalone main.py Traceback (most recent call last): File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/Plugins.py", line 82, in withPluginProblemReporting yield File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/Plugins.py", line 652, in considerExtraDlls for entry_point in _iterateExtraBinaries( File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/Plugins.py", line 641, in _iterateExtraBinaries for val in value: File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/standard/DllFilesPlugin.py", line 252, in getExtraDlls for dll_entry_point in self._handleDllConfig( File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/standard/DllFilesPlugin.py", line 211, in _handleDllConfig for result in self._handleDllConfigByCode( File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/standard/DllFilesPlugin.py", line 178, in _handleDllConfigByCode filename = self.queryRuntimeInformationMultiple( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/plugins/PluginBase.py", line 1185, in queryRuntimeInformationMultiple feedback = check_output([sys.executable, "-c", cmd], env=env) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/nuitka/utils/Execution.py", line 141, in check_output raise NuitkaCalledProcessError(exit_code, cmd, output=output, stderr=stderr) nuitka.utils.Execution.NuitkaCalledProcessError: Command '['/home/rizesquad/gmsh/venv/bin/python', '-c', '\\\nfrom __future__ import print_function\nfrom __future__ import absolute_import\n\ntry:\n import gmsh\nexcept ImportError:\n import sys\n sys.exit(38)\nprint(repr(gmsh.libpath))\nprint("-" * 27)\n']' returned non-zero exit status 1. Error was b'Traceback (most recent call last):\n File "<string>", line 6, in <module>\n File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/gmsh.py", line 87, in <module>\n lib = CDLL(libpath)\n ^^^^^^^^^^^^^\n File "/home/rizesquad/.pyenv/versions/3.11.7/lib/python3.11/ctypes/__init__.py", line 376, in __init__\n self._handle = _dlopen(self._name, mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\nOSError: libGLU.so.1: cannot open shared object file: No such file or directory'. FATAL: dll-files: Plugin issue while working on 'module 'gmsh''. Please report the bug with the above traceback included. Nuitka-Reports: Compilation report written to file 'nuitka-crash-report.xml'. Nuitka-Reports: Compilation crash report written to file 'nuitka-crash-report.xml'.
nuitka-crash-report.zip

[kayhayen]

I wonder if the correct way to phrase that kind of DLL data files, I hope it's not needed.

Regarding your crash, does gmsh work at all if imported on your Linux, or are there OS DLLs missing?

[KRRT7]

oh my bad, i tried compiling without seeing if the file itself runs.

(venv) rizesquad@bww:~/gmsh$ python main.py
Traceback (most recent call last):
  File "/home/rizesquad/gmsh/main.py", line 1, in <module>
    import gmsh
  File "/home/rizesquad/gmsh/venv/lib/python3.11/site-packages/gmsh.py", line 87, in <module>
    lib = CDLL(libpath)
          ^^^^^^^^^^^^^
  File "/home/rizesquad/.pyenv/versions/3.11.7/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libGLU.so.1: cannot open shared object file: No such file or directory

[KRRT7]

turns out, when i install gmsh on my linux machine using sudo apt install gmsh and then compile main.py using nuitka, everything works properly. i'll open a PR for this then.

[kayhayen]

From a Debian package it's allowed to take in system DLLs, but you opted out of portability to outside of that Debian pretty much for the largest part. The debian warning page probably needs an update for that kind of thing.