Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

numpy support does not seem to be activated when installing from PyPi #405

Closed
tduberne opened this issue Apr 23, 2019 · 7 comments

Comments

Projects
None yet
2 participants
@tduberne
Copy link
Contributor

commented Apr 23, 2019

Not sure whether this is intended, but I understood from the documentation that numpy support should be enabled if numpy is found on the system, even when installing from PyPi.

In a fresh virtualenv, I did:

pip install numpy
pip install JPype1

And tested numpy support using:

import numpy as np
import jpype

jpype.startJVM(jpype.getDefaultJVMPath())
a = np.random.random(100).astype(np.byte)
jarr = jpype.JArray(jpype.JByte)(a)

It resulted in the error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-eb80e17c6103> in <module>
----> 1 jarr = jpype.JArray(jpype.JInt)(a)

~/Documents/code/fast-python-java-api/python/venv/lib64/python3.6/site-packages/jpype/_jarray.py in _jarrayInit(self, *args)
    108 
    109         _JavaArrayClass.__init__(
--> 110             self, _jpype.newArray(self.__class__.__javaclass__, sz))
    111 
    112         if values is not None:

TypeError: only size-1 arrays can be converted to Python scalars

using

pip install git+https://github.com/jpype-project/jpype@master

to install JPype instead solves the issue. I understand this should be the same version as on PyPi. I noticed no compilation happens on my machine when installing from PyPi, so this might be the problem?

My actual problem is more complicated to describe, but was solved as well: passing a Java byte array calling an interface implemented by JProxy resulted in a list, and now in a numpy array.

I am on Fedora Linux, in case it matters.

Thanks a lot for the fantastic library in any case!

@Thrameos

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

I don't think this is intended behavior. It should find numpy if it is installed when doing the install procedure. I think there is an option to force it to compile with numpy, but if it didn't find it it is likely not going to work. Perhaps numpy has both a runtime and developer package on your system?

@tduberne

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Thank you for the answer. I did some experiments in docker containers to better understand here, and it appears that the problem I mention does not come from a difference between the modes of installation, but that the master branch does not correspond to the version on PyPi... Installing from the GitHub release tag fails as well. It might be my test tests something else than "is numpy support enabled": I just copied what I found in the JPype unit tests.

Sorry for the misleading issue. I guess my solution for the moment will be to use the development version.

Is there any planned release in a close future? Depending on an official release would certainly be better.

For the record, what I actually am trying to achieve is to implement a java interface that looks like this:

public interface ByteConsumer {
    void consume(byte[] bytes);
}

with the development head, my solution looks like this:

class Consumer:
	def consume(self, bytes):
		np_array = bytes[:]
		# use the numpy byte array

consumer = JProxy(ByteConsumer, inst=Consumer())
@Thrameos

This comment has been minimized.

Copy link
Contributor

commented May 2, 2019

The devel version will hopefully become release next month. The JProxy should work for you. The devel version should also support creating interfaces using annotations, but that it one the last patches that has a problem in the CI before I was rudely pulled off the project due to work issue for the last few months.

@tduberne

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Thank you, that sounds great! Looking forward to the new release.

Closing this issue as (1) the title is misleading, and (2) a solution was found.

@tduberne tduberne closed this May 3, 2019

@tduberne

This comment has been minimized.

Copy link
Contributor Author

commented Jul 5, 2019

Coming again with new information. Here is what just happened:

  • I just upgraded to 0.7.0 from PyPi (previously, I had 0.6.4 from github)
  • getting slices as numpy array failed again

After quite some research, I could solve the issue using the "no binary" option:

pip install --no-binary :all: JPype1

This might be platform dependent, if I trust a comment posted under this stackoverflow answer. I am using fedora, which is one of the two distributions mentioned in the comment (the other one being ubuntu, which seems to be used for JPype's CI).

I am not sure what the best fix is for this. I would propose at least mentioning that issue in the "installation" page (PR coming).

Thrameos added a commit that referenced this issue Jul 8, 2019

Merge pull request #507 from tduberne/binary_install_doc
document issue with PyPi install and numpy support (see issue #405)

tduberne added a commit to tduberne/jpype that referenced this issue Jul 18, 2019

run tests installing from binary distribution rather than source dist…
…ribution

new tests seem to indicate jpype-project#405 is related to differences between python
versions: this aims at testing that behavior.
@tduberne

This comment has been minimized.

Copy link
Contributor Author

commented Jul 19, 2019

For the record, I finally understood the reason for this behavior:

  • the binary distributions on PyPi are in manylinux2010 format, which is only understood by pip>=19.0.
  • With such a pip version, installing the binary package leads to numpy support being deactivated, and JPype needs to be installed explicitly from the source distribution.
  • with pip<=18.1, pip cannot find a binary distribution it understands, and silently reverts to trying the source distribution, which makes numpy support active if numpy is found.

So it seems the problem is in general that if one needs numpy support, one needs to install from a source distribution.

Is there a strong argument for distributing in wheel format? As far as my tests go, it seems that (contrary to installing directly from source), installing from the source distribution on PyPi does not require having e.g. ant installed.

If there are good reasons to offer binaries as well, I will prepare a new PR such that the documentation is more precise as to the sources of the problem.

@Thrameos

This comment has been minimized.

Copy link
Contributor

commented Jul 19, 2019

Thanks for the good info. marscher is the one building the Linux distributions, so he will need to deal with this particular problem.

The current state of the source distribution is

  1. Building directly from git requires ant because it needs to create the jpype jar.
  2. Source distributions built from the git repo cache the jar into the distribution to avoid the ant dependency.
  3. Binary distributions obviously have no requirements for ant.

In general, I am moving toward requiring numpy for JPype. Although numerical support is not necessarily a requirement for Java integration, there is a strong need to provide a raw access to arrays from Java in order to get reasonable performance from JPype and numpy is currently the only good option. Integration of streams, byte channels, pickle support and access to primitive arrays all need some method of rapidly converting Java arrays to C arrays and back.

If you can put in a pull request with updated instruction as documentation for the install it would be a great help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.