Problem with virtualenv: "Could not open python 310.dll"

[danieleteti]

This project is simply awesome! Good job guys.

I'm having a problem while using virtualenvs

I did the same test using python 3.9 and the problem is the same.
The problem doesn' happens using the system Python installation (without venvs).

Thanks

[lmbelo]

Hi, @danieleteti!
Thanks for the report and for the accolades.
Python4Delphi requires Python to be compile with the "--enable-shared" symbol, once it consumes the Python interpreter .dll.
I'd suggest you to try again using miniconda or conda itself.

[pyscripter]

@lmbelo
P4D supports venvs.
The issue is that when P4D is used to develop extension modules such as delphivcl, the standard search for the python dll should not be applied (looking in the registry etc.). The python dll is already loaded in the execution space and you want to work with the very same dll. It should be enough to call LoadLibrary('pythonxy.dll') without specifying a directory. Also a lot of the stuff that initializes python (SetHomeDirectory etc.) I think they are not needed. We probably need a different version of LoadDLL for use in extension modules.

[lmbelo]

@pyscripter have a look at the following link:

https://github.com/Embarcadero/python4delphi/blob/master/Modules/DelphiFMX/ModuleSpecs.pas

We are doing something similar on Linux. It is still not supported on delphivcl and delphifmx (non-Linux).

[pyscripter]

@lmbelo

I don't think any of that stuff is needed at least not in Windows and quite likely not in Linux either. I don't know about Android.
See my latest commit. I have added a new method LoadDLLInExtensionModule. It skips all stuff related to python initialization, since python is already initialized. It also bypasses the registry on Windows and GetDLLPath returns ''.

As a proof of concept I have modified the DemoModule project to use the LoadDLLInExtensionModule. It works well with all python versions and venvs as well.

If you look at the documentation of LoadLibrary it says that if the DLL is already loaded (which is this case here), then it is used and you do not need the path.

[lmbelo]

Ok, cool. I will have a look. Thank you.

[pyscripter]

Another approach would be to use EnumProcessModules to find out which python dll is loaded and load that file.

In linux one could use dl_iterate_phdr.

The above methods should be more reliable if you need to have one extension module for all python versions.

[lmbelo]

The example I sent you before works with a sort of Python library on Linux that doesn't provide a shared library, the symbols are on the host executable.

[pyscripter]

I am not sure what you mean. I thought you are trying to use delphivcl/fmx with standard python distributions. In windows and linux python.exe is using a shared library pythonxy.dll or libpythonxy.so or something.

I understand in Android you are making your won stuff so it could be different.

[lmbelo]

Have you tried delphifmx with miniconda on Linux? It doesn't load a shared library.
Run the delphifmx and enumerate the loaded libraries, there is no shared library loaded, but it exports the API symbols in the host executable.
Try to install using apt on Ubuntu, as well. There is no shared library, neither exported symbols in host.

[pyscripter]

Have you tried delphifmx with miniconda on Linux?
No I haven't.

[lmbelo]

Regarding the host exported symbols, same may occur on macOS.
The current way we are setting up the Python environment works well, and doesn't affect the original P4D library, which was indeed our main concern.
Venv is still not supported by the delphivcl because I haven't introduced the same changes we made in the delphifmx here in the delphivcl yet.

[pyscripter]

Anyway, LoadDLLInExtensionModule makes sense in all systems. Python is already initialized. There is no need to initialize it again. And you need to make sure you are using the same DLL as python.exe.

Try the approach in DemoModule with Delphivcl on Windows. Is incredibly simple and it works great.

[lmbelo]

Cool. Thanks!

[pyscripter]

Python4Delphi requires Python to be compile with the "--enable-shared" symbol, once it consumes the Python interpreter .dll.
I'd suggest you to try again using miniconda or conda itself.

I see what you meant! This is an issue on Linux and not on Windows that the original report was referring to. From what I see the standard python3 on Ubuntu and other distributions is statically linking libpython. But I tried miniconda and this also statically links libpython.

I had a closer look at ModuleSpecs.pas and I see that on Linux you resolve the imports from the executable (clever!).
I am still not sure what is moduledefs.json. I cannot find such a file on Linux.

On my Windows Ubundu WSL terminal delphifmx works with miniconda but not with the standard python3. Why is that?

[lmbelo]

@danieleteti we have introduced venv support. Can you try again, please?

[Priyatham10]

We now successfully got it working with all;

• System-level Python
• System-level Python virtualenv
• VIrtualenv installed under conda
• Conda environments

Please see the supporting screenshots:

Sytem level Python virtualenv:

Python virtualenv created inside conda:

[danieleteti]

@danieleteti we have introduced venv support. Can you try again, please?

Hi, t seems to work.
I did the following

C:\CORSI\PythonWithDelphiUI
(venv) λ ls
samples/  venv/

C:\CORSI\PythonWithDelphiUI
(venv) λ pip install -U delphivcl
Requirement already satisfied: delphivcl in c:\corsi\pythonwithdelphiui\venv\lib\site-packages (0.1.29)
Collecting delphivcl
  Downloading delphivcl-0.1.37-cp310-cp310-win_amd64.whl (6.4 MB)
     |████████████████████████████████| 6.4 MB 6.8 MB/s
Installing collected packages: delphivcl
  Attempting uninstall: delphivcl
    Found existing installation: delphivcl 0.1.29
    Uninstalling delphivcl-0.1.29:
      Successfully uninstalled delphivcl-0.1.29
Successfully installed delphivcl-0.1.37
WARNING: You are using pip version 21.2.3; however, version 22.0.3 is available.
You should consider upgrading via the 'C:\CORSI\PythonWithDelphiUI\venv\Scripts\python.exe -m pip install --upgrade pip' command.

C:\CORSI\PythonWithDelphiUI
(venv) λ ls
samples/  venv/

C:\CORSI\PythonWithDelphiUI
(venv) λ cd samples\

C:\CORSI\PythonWithDelphiUI\samples
(venv) λ ls
helloworldvcl.py

C:\CORSI\PythonWithDelphiUI\samples
(venv) λ python helloworldvcl.py

And the simple form correctly shows up.

Thank you guys. From my POV this issue can be closed. ASAP I'll do more extensive tests.

[lmbelo]

@danieleteti I'm closing this issue once it is resolved.
Thank you!