Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NumPy 1.14+ compatibility #22

Open
Stewori opened this issue Jan 17, 2018 · 8 comments
Open

NumPy 1.14+ compatibility #22

Stewori opened this issue Jan 17, 2018 · 8 comments

Comments

@Stewori
Copy link
Owner

Stewori commented Jan 17, 2018

... is currently broken:

Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 68, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.0/numpy/__init__.py", line 168, in <module>
    from . import ma
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/__init__.py", line 44, in <module>
    from . import core
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6333, in <module>
    masked = masked_singleton = MaskedConstant()
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6270, in __new__
    cls.__singleton = MaskedArray(data, mask=mask).view(cls)
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 2790, in __new__
    _data = ndarray.view(_data, cls)
AttributeError: 'type$1' object has no attribute 'update'

Stick to NumPy 13.3 until this is solved...

@Stewori Stewori changed the title NumPy 1.14.0 compatibility NumPy 1.14 compatibility Jun 30, 2018
@Stewori
Copy link
Owner Author

Stewori commented Jun 30, 2018

I just investigated newer NumPy versions. 14.1 and 14.2 result in the same error as 14.0 above.
NumPy 14.3, 14.4, 14.5 give this new error:

('1.8.0_171', 'Oracle Corporation', ('OpenJDK 64-Bit Server VM', '25.171-b11', 'Oracle Corporation'), ('Linux', '4.4.0-128-generic', 'amd64'))
Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 73, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 166, in <module>
    from . import random
  File "mtrand.pyx", line 1, in init mtrand
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 118, in <module>
    __NUMPY_SETUP__
TypeError: unhashable type: 'str'

@Stewori
Copy link
Owner Author

Stewori commented Jul 1, 2018

So, I located the first type of failure to NumPy's source file multiarray/ctors.c.
Specifically in the function PyArray_NewFromDescr_int the line res = PyObject_Call(func, args, NULL); fails, i.e. returns zero, triggering the subsequent goto fail;. Next step is to identify why this call fails. For now I only know that it emits a bunch of Python-exceptions, where the message that we see, i.e. 'type$1' object has no attribute 'update' is only the tip of the iceberg. The full sequence of error messages is:

'numpy.ndarray' object has no attribute '_fill_value'
'numpy.ndarray' object has no attribute '_hardmask'
'numpy.ndarray' object has no attribute '_sharedmask'
'numpy.ndarray' object has no attribute '_isfield'
'numpy.ndarray' object has no attribute '_baseclass'
'type$1' object has no attribute 'update'

So far I conclude that somehow attribute (write-)access of numpy.ndarray got messed up.

@Stewori
Copy link
Owner Author

Stewori commented Jul 2, 2018

Tracked it down a bit further:
It fails in the method MaskedArray._update_from in ma/core.py.
Specifically the line self.__dict__.update(_dict) fails. Here _dict contains a bunch of updates for attributes that shall be written to self.__dict__. It seems that for some reason self.__dict__ might not be an ordinary dictionary, which somehow causes the issue.
type(self.__dict__) results in <type 'type$1'>, which is familiar from our error message 'type$1' object has no attribute 'update'. print(self.__dict__) outputs <attribute '__dict__' of 'MaskedArray' objects>. So maybe it's an issue with attribute wrappers while the dictionary itself is fine. It seems like the key lies in understanding what type$1 actually is.

The MaskedArray class extends ndarray and I remember that I once observed Astropy to fail with JyNI because of ndarray subclassing. Maybe before version 14.0 the NumPy import worked without subclassing ndarray. Unfortunately the commit history of ma/core.py is insanely complex and it is challenging to track down the breaking commit. I thought about using git's bisect feature, but that's probably tedious because of all the rebuilds required. For now I will just continue investigation. If I can fix ndarray subclassing in JyNI, then it hopefully also improves usecases like in Astropy.

@Stewori
Copy link
Owner Author

Stewori commented Jul 4, 2018

Note that the CPython output is as one would naively expect:
Instead of <type 'type$1'> we get <type 'dict'> and instead of <attribute '__dict__' of 'MaskedArray' objects> we get {}. So somehow JyNI, or maybe Jython already, messes up this attribute. Maybe it just misses to evaluate the attribute wrapper properly.

@Stewori
Copy link
Owner Author

Stewori commented Jul 5, 2018

The "dict" object of type <type 'type$1'> is an instance of PyDataDescriptor, see PyDataDescriptor.java. type$1 is a generated type (as I already expected actually). PyDataDescriptor is abstract and Jython generates concrete implementations at runtime, resulting in these strange type names. Now the real question is why JyNI does not evaluate __get__. Maybe the case of extending natively defined builtin-like types is not yet properly covered, an observation that is consistent with the way how Astropy fails.
I will review the way how new-style classes are handled by JyNI. Originally the described case should have been workable, but that appears to be incomplete. Maybe this will require something like PyCPeerDerived. We'll see...

@Stewori
Copy link
Owner Author

Stewori commented Jul 5, 2018

Quick experiment:
Inserted this code snippet into PyCPeer.__findattr_ex__ right before the return statement:

if (er != null && er.implementsDescrGet() && er.isDataDescr()){
    er = er.__get__(this, objtype);
}

Now the descriptor is evaluated, but so far this doesn't fix anything because it just evaluates to None. It results now in AttributeError: 'NoneType' object has no attribute 'update' instead of AttributeError: 'type$1' object has no attribute 'update' (Is this an improvement?).
It seems like the generated implementation of PyDataDescriptor does not know what to do with a PyCPeer...

@Stewori Stewori changed the title NumPy 1.14 compatibility NumPy 1.14+ compatibility Jul 26, 2018
@Stewori
Copy link
Owner Author

Stewori commented Jul 26, 2018

The recently released NumPy 1.15.0 gives the following error on import:

/usr/lib/jvm/java-8-openjdk-amd64/bin/java:
symbol lookup error:
/data/workspace/linux/numpy/1.15.0/numpy/core/multiarray.so:
undefined symbol:
PyStructSequence_InitType

However, I suppose this is easier to fix. Adding a PyStructSequence_InitType implementation is not a big deal. However I suspect there will be subsequent issues.

@ShaharZivan
Copy link

Did you find any solution for this yet? Even a temporary hardcoded patch would do. I need to use sequitur-g2p with Jython and sadly it requires NumPy 1.14.2 or higher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants