Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is Python installation required on the host system + shipping pip packages #871

Closed
fcunilim opened this issue Apr 20, 2020 · 4 comments
Closed
Labels

Comments

@fcunilim
Copy link

fcunilim commented Apr 20, 2020

I want to ship a Java application which currently uses bytedeco's CPython through a maven dependency. I have no problem with my code so far. For now I use version 3.6-1.5. Python 3.6.8 is installed on my development computer.

I have two questions:

  • do I need to install Python on the target (user) system? Or will Python be completely "shipped" in the JARs? Do I need to upgrade to the latest version (3.7.7-1.5.3) for this to happen?

  • I would like to ship a few Python pip packages as well. What is the recommended way of doing that? Shall I just create a venv, ship it with my application, and add the site-packages folder to the path, i.e.:

Py_AddPath(cachePackages());
Py_AddPath( my venv site-packages folder path );

Thank you.

@fcunilim fcunilim changed the title Is Python installation on the host system or is CPython completely bundled? Is Python installation required on the host system or is CPython completely bundled? Apr 20, 2020
@fcunilim fcunilim changed the title Is Python installation required on the host system or is CPython completely bundled? Is Python installation required on the host system + shipping pip packages Apr 20, 2020
@saudet
Copy link
Member

saudet commented Apr 21, 2020

CPython is being bundled since 1.5.1, see issue #712, so no need to install anything, no.

Ideally everything should be packaged as JARs as with the presets for NumPy, OpenCV, SciPy, and Gym, but we can install packages with pip and have them loaded in the way you describe, yes. It works just fine in the case of, for example, the optional dependencies of Gym, as shown on issue deeplearning4j/deeplearning4j#8688 (comment).

@fcunilim
Copy link
Author

fcunilim commented Apr 27, 2020

I could get the venv to run by adding site-packages through Py_AddPath. As a side note, I noticed 'pip install -e'-installed packages (i.e. egg links) are not visible though. This is not very important.

Ideally everything should be packaged as JARs as with the presets for NumPy, OpenCV, SciPy, and Gym

Is there an howto for this?
I understand that the various cachePackage(s) functions copy JAR resources over to the local filesystem under the '.javacpp' folder in the home directory. The Python runtime can then load these files. The only problem is when there are many files (i.e. tensorflow, cuda) but I guess the wait is bearable.

Is there any tutorial available, or is there any piece of code that can be reused from your side to assist in implementing cachePackage for home-baked libraries? I think this would be very beneficial for the project.

@saudet
Copy link
Member

saudet commented Apr 28, 2020

We basically need to create new presets, and all the information to get started with that is here:
https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets
In the case of Python libraries, they don't typically come with a C API (although some like NumPy do), so we don't need to implement InfoMap. SciPy is a good example of that, and it can be used as a template to add more packages like this:
https://github.com/bytedeco/javacpp-presets/tree/master/scipy
Let me know if you have any questions regarding that or need any help understanding something in particular. It would be great indeed if we started having more Python packages, such as Pandas (#788), Matplotlib (#755), etc.

/cc @penetest

BTW, caching files should occur only once, so even CUDA or TensorFlow should load fast after the first load. Let me know if this isn't the case on your system, which may indicate there is a bug somewhere. Thanks

@fcunilim fcunilim closed this as completed May 3, 2020
@saudet
Copy link
Member

saudet commented Mar 9, 2021

BTW, here's an example that works fine using pip:

        // Extract to JavaCPP's cache CPython and obtain the path to the executable file
        String python = Loader.load(org.bytedeco.cpython.python.class);

        // Install in JavaCPP's cache GluonNLP and MXNet to download and import BERT model
        new ProcessBuilder(python, "-m", "pip", "install", "gluonnlp", "mxnet", "pytest").inheritIO().start().waitFor();

https://github.com/bytedeco/javacpp-presets/blob/master/tvm/samples/DeployBERT.java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants