Is Python installation required on the host system + shipping pip packages

[fcunilim]

I want to ship a Java application which currently uses bytedeco's CPython through a maven dependency. I have no problem with my code so far. For now I use version 3.6-1.5. Python 3.6.8 is installed on my development computer.

I have two questions:

• do I need to install Python on the target (user) system? Or will Python be completely "shipped" in the JARs? Do I need to upgrade to the latest version (3.7.7-1.5.3) for this to happen?

• I would like to ship a few Python pip packages as well. What is the recommended way of doing that? Shall I just create a venv, ship it with my application, and add the site-packages folder to the path, i.e.:

Py_AddPath(cachePackages());
Py_AddPath( my venv site-packages folder path );

Thank you.

[saudet]

CPython is being bundled since 1.5.1, see issue #712, so no need to install anything, no.

Ideally everything should be packaged as JARs as with the presets for NumPy, OpenCV, SciPy, and Gym, but we can install packages with pip and have them loaded in the way you describe, yes. It works just fine in the case of, for example, the optional dependencies of Gym, as shown on issue deeplearning4j/deeplearning4j#8688 (comment).

[fcunilim]

I could get the venv to run by adding site-packages through Py_AddPath. As a side note, I noticed 'pip install -e'-installed packages (i.e. egg links) are not visible though. This is not very important.

Ideally everything should be packaged as JARs as with the presets for NumPy, OpenCV, SciPy, and Gym

Is there an howto for this?
I understand that the various cachePackage(s) functions copy JAR resources over to the local filesystem under the '.javacpp' folder in the home directory. The Python runtime can then load these files. The only problem is when there are many files (i.e. tensorflow, cuda) but I guess the wait is bearable.

Is there any tutorial available, or is there any piece of code that can be reused from your side to assist in implementing cachePackage for home-baked libraries? I think this would be very beneficial for the project.

[saudet]

We basically need to create new presets, and all the information to get started with that is here:
https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets
In the case of Python libraries, they don't typically come with a C API (although some like NumPy do), so we don't need to implement InfoMap. SciPy is a good example of that, and it can be used as a template to add more packages like this:
https://github.com/bytedeco/javacpp-presets/tree/master/scipy
Let me know if you have any questions regarding that or need any help understanding something in particular. It would be great indeed if we started having more Python packages, such as Pandas (#788), Matplotlib (#755), etc.

/cc @penetest

BTW, caching files should occur only once, so even CUDA or TensorFlow should load fast after the first load. Let me know if this isn't the case on your system, which may indicate there is a bug somewhere. Thanks

[saudet]

BTW, here's an example that works fine using pip:

        // Extract to JavaCPP's cache CPython and obtain the path to the executable file
        String python = Loader.load(org.bytedeco.cpython.python.class);

        // Install in JavaCPP's cache GluonNLP and MXNet to download and import BERT model
        new ProcessBuilder(python, "-m", "pip", "install", "gluonnlp", "mxnet", "pytest").inheritIO().start().waitFor();

https://github.com/bytedeco/javacpp-presets/blob/master/tvm/samples/DeployBERT.java