How can I install a Python module from Java?

[ajh123]

In my Java project (which uses graalvm polyglot) I want to use a Python script which depends on the requests library - so I want to install it using pip.

Not sure if this is right place to post this, however I've not found a place for polyglot

This is what I tried so far:

version 1

	public static void installModule(Context context, String module) {
		context.eval("python", "import os\n" // line 32
				+ "import pip\n"
				+ "def install(package):\n"
				+ "    if hasattr(pip, 'main'):\n"
				+ "        pip.main(['install', package])\n"
				+ "    else:\n"
				+ "        pip._internal.main(['install', package])\n"
				+ "install('"+ module +"')");
	}

version 2

	public static void installModule(Context context, String module) {
		context.eval("python", "import subprocess\n" // line 32
				+ "import sys\n"
				+ "def install(package):\n"
				+ "    subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", package])\n"
				+ "install('"+ module +"')");
	}

In version 1 I got this error:

Exception in thread "main" ModuleNotFoundError: No module named 'pip'
	at <python> <module>(Unknown)
	at org.graalvm.polyglot.Context.eval(Context.java:428)
	at uk.minersonline.gns34j.PythonConnector.installModule(PythonConnector.java:32)
	at uk.minersonline.gns34j.PythonConnector.main(PythonConnector.java:16)

In version 2 I got this error:

Exception in thread "main" OSError: (5, I/O error)
	at <python> __init__(C:\Users\samro\AppData\Local\org.graalvm.polyglot\python\python-home\7d0c656f1dac055b3b202d151a09908bf97221ff\Lib\subprocess.py:1687-1880:64732-73792)
	at <python> call(Unknown)
	at <python> check_call(Unknown)
	at <python> install(Unknown)
	at <python> <module>(Unknown)
	at org.graalvm.polyglot.Context.eval(Context.java:428)
	at uk.minersonline.gns34j.PythonConnector.installModule(PythonConnector.java:32)
	at uk.minersonline.gns34j.PythonConnector.main(PythonConnector.java:16)

This is how I am calling both versions:

	public static void main(String[] args) {
		Context context = Context.newBuilder().allowAllAccess(true).build();
		installModule(context,"requests"); # line 16
	}

What I want to know is what is the best way to install a Python module from Java.

[timfel]

The best way is currently (for the 23.1 release) to start with a project generated from graalpy's distribution:

Download a graalpy release and run this:

graalpy -m standalone polyglot_app -o graalpy_java_app

It will generated a skeleton project for you, check the pom.xml for how to install packages and the Java code for how to include them.

With the upcoming release 24.0, we have a maven plugin. Then it'll just be

    <plugins>
      <plugin>
        <groupId>org.graalvm.python</groupId>
        <artifactId>graalpy-maven-plugin</artifactId>
        <version>24.0.0</version>
        <executions>
          <execution>
            <configuration>
              <!-- specify python packages as if used with pip -->
              <packages>
                <package>termcolor==2.2</package>
              </packages>
              <pythonHome>
                <!-- java-like regular expression what file paths should be included into venv/home;
                     default is all -->
                <includes>
                  <include>.*</include>
                </includes>
                <!-- java-like regular expression what file paths should be excluded from venv/home;
                     default is none -->
                <excludes>
                </excludes>
              </pythonHome>
            </configuration>
            <goals>
              <goal>process-graalpy-resources</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>

[timfel]

FYI: Both of these solutions install the package into the Java resources at build time

[ajh123]

Thanks, I will have a look later. Once the maven plugin is released how could I use it from Gradle?

[timfel]

Thanks, I will have a look later. Once the maven plugin is released how could I use it from Gradle?

Our Gradle plugin is still under development and didn't make it for this release. We are planning to have a dedicated Gradle plugin in autumn

[ajh123]

@timfel
On Windows 10 when I tried to run:
graalpy -m standalone polyglot_app -o graalpy_java_app

I got this error:

Mar 04, 2024 2:32:14 PM org.graalvm.shadowed.org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
Error processing line 7 of C:\Users\samro\AppData\Roaming\Python\Python310\site-packages\pywin32.pth:

  Traceback (most recent call last):
    File "D:\Program Files (x86)\graalpy-23.1.2-windows-amd64\Lib\site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Users\samro\AppData\Roaming\Python\Python310\site-packages\win32\lib\pywin32_bootstrap.py", line 23, in <module>
      os.add_dll_directory(path)
    File "D:\Program Files (x86)\graalpy-23.1.2-windows-amd64\Lib\os.py", line 1118, in add_dll_directory
      cookie = nt._add_dll_directory(path)
  AttributeError: module 'nt' has no attribute '_add_dll_directory'

Remainder of file ignored

GrealPy version

graalpy --version
Mar 04, 2024 2:33:25 PM org.graalvm.shadowed.org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
GraalPy 3.10.8 (Oracle GraalVM Native 23.1.2)

After doing this I uninstalled my existing Python 3.10.2 and still get the same errors. GraalPy's "bin" directory is on my path.

[ajh123]

However it looks like the graalpy_java_app folder was still created with its contents, so its a very random error.

[timfel]

After doing this I uninstalled my existing Python 3.10.2 and still get the same errors. GraalPy's "bin" directory is on my path.

Uninstalling Python 3.10.2 will not remove globally installed packages, and GraalPy will pick them up. You should remove "C:\Users\samro\AppData\Roaming\Python\Python310" before running graalpy

[ajh123]

I think I know where I am going now. Thanks for the help.

[timfel]

I think I know where I am going now. Thanks for the help.

You're welcome! I'd be interested to hear about what you're doing if it is sth you want to share later on :)

[ajh123]

I can send a link to my repository once I get it working

[wangxi761]

pretty good,it slove my problem. but I don't see any relevant steps in the document about install packge from embedded java. Do I need to provide a pr to improve the document ？

[ajh123]

@wangxi761 It would be nice if you do, it would solve the problem for many more people.

Lack of documentation was the main reason why I submitted this issue ...

[timfel]

Does the documentation in the Maven archetype help? https://github.com/oracle/graalpython/blob/master/docs/user/README.md#maven