Should I take care of "could not determine Graal.Python's *** path"?

[eomasters-repos]

When I enable verbose logging, I see the following lines:

[python::Python3Core] FINE: Initial locations:
	Language home: null
	SysPrefix: 
	BaseSysPrefix: 
	CoreHome: 
	StdLibHome: 
	CAPI: 
	JNI library: 
	Home candidate: C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin
[python::Python3Core] FINE: Updated locations:
	Language home: C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin
	SysPrefix: 
	BaseSysPrefix: 
	CoreHome: 
	StdLibHome: 
	Executable: C:\Users\marco\snap_code\GraalPyTest\venv\Scripts\python.exe
	CAPI: 
	JNI library: 
[python::PythonContext] WARNING: could not determine Graal.Python's core path - you may need to pass --python.CoreHome.
[python::PythonContext] WARNING: could not determine Graal.Python's sys prefix path - you may need to pass --python.SysPrefix.
[python::PythonContext] WARNING: could not determine Graal.Python's standard library path. You need to pass --python.StdLibHome if you want to use the standard library.
[python::PythonContext] WARNING: could not determine Graal.Python's C API library path. You need to pass --python.CAPI if you want to use the C extension modules.

I wonder If I should worry about and set the paths explicitly? And which paths are those?

[msimacek]

It's definitely a problem, but you shouldn't set the paths manually, GraalPy should be able to find them. How do you execute GraalPy and how did you create the virtualenv? And what GraalPy version is this?

[eomasters-repos]

Thanks. Good to know. I already thought that graalpy should be able to figure out the paths.
I'm using graalpy-community-24.2.2-windows-amd64.zip in my maven pom I use

• org.graalvm.polyglot:polyglot:24.2.2
• org.graalvm.polyglot:python:24.2.2
• org.graalvm.python:python-language:24.2.2

And I execute the Java project with graalvm-ce-23.0.2.

I setup the engine and the context manually in my project.
It could be that something went wrong. I did quite some experiments.
I'll extract the essential code of my project and try to reproduce it. Maybe I fix it on the way, or I can provide a better report.

[eomasters-repos]

Meanwhile I'm confused and a bit desperate. Probably I'm doing something wrong, but I don't see it.
Sorry for the lengthy report but I don't know how to make it short. And thanks to everyone who is looking into it.

I have now created a simple projecte where I do basically the same as in my real project.
https://github.com/eomasters-repos/GraalPySetup

I'm using, as said

• org.graalvm.polyglot:polyglot:24.2.2
• org.graalvm.polyglot:python:24.2.2
• org.graalvm.python:python-language:24.2.2

As JVM I'm using the latest graalvm-jdk-24.0.2+11.1
And my Graalpython is Python version: 3.11.7 [Graal, Oracle GraalVM, Java 24.0.2 (amd64)]
All is executed on Win11

First, I should say that now in the sample project are initialised, but still not in my real project even though I have the same configuration, I think:

[python::Python3Core] FINE: Initial locations:
	Language home: null
	SysPrefix: 
	BaseSysPrefix: 
	CoreHome: 
	StdLibHome: 
	CAPI: 
	JNI library: 
	Home candidate: C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin
[python::Python3Core] FINE: Initial locations:
	Language home: null
	SysPrefix: 
	BaseSysPrefix: 
	CoreHome: 
	StdLibHome: 
	CAPI: 
	JNI library: 
	Home candidate: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c
[python::Python3Core] FINE: Updated locations:
	Language home: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c
	SysPrefix: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c
	BaseSysPrefix: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c
	CoreHome: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c\lib-graalpython
	StdLibHome: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c\Lib
	Executable: C:\Users\marco\IdeaProjects\GraalPySetup\example\venv\Scripts\python.exe
	CAPI: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c\lib-graalpython
	JNI library: C:\Users\marco\AppData\Local\org.graalvm.polyglot\python\python-home\86aae279f9357aa0b2b83ce56850e28c7b84fe1c\lib-graalpython

But still can't use numpy. The only numpy version I can install using pip is 1.26.4. But when importing it I get the error:

Traceback (most recent call last):
  File "C:\Users\marco\IdeaProjects\GraalPySetup\example\src\code.py", line 12, in <module>
    import numpy as np
  File "C:\Users\marco\IdeaProjects\GraalPySetup\example\venv\Lib\site-packages\numpy\__init__.py", line 148, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.
Exception in thread "main" org.graalvm.polyglot.PolyglotException
	at org.graalvm.polyglot.Context.eval(Context.java:416)
	at Main.main(Main.java:50)

Installation of later numpy versions like 2.2.4, already the package installation fails.

auto-patching C API usages in C:\Users\marco\AppData\Local\Temp\pip-install-wv5pzjaq\numpy_fa5565c7103d4b8495c575fb841aaeb1\numpy\_core\include\numpy\ndarrayobject.h
<shortened>
auto-patching C API usages in C:\Users\marco\AppData\Local\Temp\pip-install-wv5pzjaq\numpy_fa5565c7103d4b8495c575fb841aaeb1\numpy\_core\src\umath\_rational_tests.c
  Looking for GraalPy patches for numpy
  Patching package numpy using numpy-2.2.4.patch
ERROR: Could not install packages due to an OSError: [Errno 5] I/O error: 'patch.exe'

I have the folder where the patch.exe is located on the PATH. I'm testing this with the following command and the patch window is shown.

new ProcessBuilder("cmd", "/c", "patch.exe").start()

Strangely I don't get the _add_dll_directory error here.

[msimacek]

Strangely I don't get the #493 (comment) here.

You do get in the case in 1.26.4 (the prebuilt wheel), but you don't see it because numpy catches it and wraps it into their own exception.

You won't get it when you build numpy from source, which is the case of the later versions (there are no prebuilt wheels for those atm). I tried your example and I got a slightly different error from patch:

Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354

That's a bug in the GNU32 windows port of patch that apparently happens when the file has unix line endings (the bug was probably fixed, but the GNU32 port hasn't been updated for 22 years). The patches that are shipped with GraalPy have windows line endings, but our pip also checks for post-release patches on github and it seems that this is how we end up with unix line endings there. As a workaround, you can set PIP_GRAALPY_PATCHES_URL variable to something nonexistent which will disable the post-release patches. Then you can build numpy from source with pip install numpy --no-binary numpy. The maximum supported numpy version without the post-release patches is 2.0.2

[eomasters-repos]

Thanks for your suggestions. However, I get other issues now. I give up one trying to get it to run on windows.
I just tried my example on Linux, and it just worked.
Do you have a rough estimate of when Windows will be supported as well as Linux?

[msimacek]

I'll try to fix all the packaging issues you encountered for 25.0.1 release. There are no Windows-specific plans beyond that for now. Note you can run the Linux version of GraalPy on WSL.

[eomasters-repos]

Sounds good. WSL is not an option for the final application. Average users need to run this. Setting up WSL is a bit too much. But thanks for the hint.

[eomasters-repos]

Meanwhile I think that the NetBeans platform is the culprit. The paths (SysPrefix, BaseSysPrefix, ...) are initialsed when I use a simple main.
But when initialising graalpython from the running NetBeans platform application, it is not.
Could it be a ClassLoader issue? Because Netbeans uses its own ClassLoader for the module system.
I tried to do the context.eval() from a special thread and setting ContextClassLoader, but till now without success.

[msimacek]

Yes, that could be a problem. The standard library is looked up via resources, which might break when you have unusal classloader setup. You could try to set resourceLoadingClass.

[eomasters-repos]

Thanks. for the quick reply. Unfortunately, I don't use the VirtualFileSystem. I only use the Engine and Context builder. Both don't provide this functionality. I don't use the embedding at all because the sources and the venv are user provided.

Using hostClassLoader() on the context builder does not help.

[eomasters-repos]

Finally the graalpython context is properly initialised.
The problem arose from the fact that when initialising a venv with graalpy the home attribute in the cfg file points to the bin folder.
With a standard python it is the python folder.

Graalpy:

home = C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin
include-system-site-packages = false
version = 3.11.7
executable = C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin\python.exe
command = C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin\python.exe -m venv C:\Users\marco\AppData\Local\Temp\venvTest4693457791775089874\graalVenv
venvlauncher_command = C:\Users\marco\.snap\auxdata\eomtbxpro\graalpy\bin\python.exe

python:

home = C:\Users\marco\AppData\Local\Programs\Python\Python39
include-system-site-packages = false
version = 3.9.13

The option 'python.PythonHome' for the context needs the parent folder of the bin.

[msimacek]

The home option in pyvenv.cfg needs to point to the directory where the executable is. In case of GraalPy, it is in the bin directory, so having the bin there is correct. We strip the bin away when setting PythonHome from pyvenv.cfg when running the standalone launcher:

graalpython/graalpython/com.oracle.graal.python.shell/src/com/oracle/graal/python/shell/GraalPythonMain.java

Line 981 in cee059b

if (graalpyHome.endsWith("bin") || graalpyHome.endsWith("Scripts")) {

But nothing parses that file when embedding. If you're setting python.PythonHome manually then you indeed need to remove the bin. But you shouldn't need to set that. The recommended way to set us a context with externally managed resources (venv etc) is:

        try (Context context = GraalPyResources.contextBuilder(path).build()) {
            context.eval("python", "...");
        }

The path you pass to the builder is the resources root that should contain a venv directory. It can also contain a src directory that will be added to the pythonpath.

[eomasters-repos]

Ah. Thanks. I will give it a try.