Objects in Map<String,Object> and other complex containers sometimes disappear

[vslotman]

I am using pyjnius to pass nested Python dictionaries and lists to Java
Sometimes the content of Map<String,Object> and ArrayList disappears.

It usually goes okay with simple one-dimensional lists and dictionaries filled with strings and integers. But when structures get more complex sometimes data goes missing. It doesn't always go wrong, one day a certain Python object maps fine to Java, the other day the resulting Java-objects misses items or is completely empty.

I'm using the following versions:
Python 2.7.6
Cython 0.23.4
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
Tried jnius 1.0.2 and 1.1-dev

import jnius
from collections import Iterable, Mapping
import gc

# Disable the Python garbage-collector, just to be sure
gc.disable()

# Java DataTypes
jMap       = jnius.autoclass('java.util.HashMap')
jArrayList = jnius.autoclass('java.util.ArrayList')
jInt       = jnius.autoclass('java.lang.Integer')
jLong      = jnius.autoclass('java.lang.Long')
jFloat     = jnius.autoclass('java.lang.Float')
jDouble    = jnius.autoclass('java.lang.Double')
jString    = jnius.autoclass('java.lang.String')

class JavaNumber(object):
    '''
    Convert int/float to their corresponding Java-types based on size
    '''
    def __call__(self, obj):
        if isinstance(obj, int):
            if obj <= jInt.MAX_VALUE:
                return jInt(obj)
            else:
                return jLong(obj)
        elif isinstance(obj, float):
            if obj < jFloat.MAX_VALUE:
                return jFloat(obj)
            else:
                return jDouble(obj)

# Map between Python types and Java
javaTypeMap = { int:   JavaNumber(),
                str:   jString,
                float: JavaNumber() }


def mapObject(data):
    '''
    Recursively convert Python object to Java Map<String, Object>
    :param data:
    '''
    try:
        if type(data) in javaTypeMap:
            # We know of a way to convert type
            return javaTypeMap[type(data)](data)
        elif isinstance(data, jnius.MetaJavaClass):
            # It's already a Java thingy, or None
            return data
        elif isinstance(data, Mapping):
            # Object is dict-like
            map = jMap()
            for key, value in data.iteritems():
                map.put(str(key), mapObject(value))
            return map
        elif isinstance(data, Iterable):
            # Object is list-like
            array = jArrayList()
            for item in data:
                i = mapObject(item)
                array.add(i)
            return array
        else:
            # Convert it to a String
            return jString(str(data))
    except:
        print 'Failed to map Python-object to Java!'
        print str(data)
        raise



goes_okay = {'list': {3: 4, 5: 6}, 
              4      : 5 }

misses_dict_item = {'list': [1, 2, 7], 
                      'int': 3, 
                      4: 5, 
                      'dict': {2: 3} }

empty = {'access_log': [{'stored_proc': 'getsomething'},
                        {'uses': [{'usedin': 'some->bread->crumb'},
                                {'usedin': 'something else here'},
                                {'stored_proc': 'anothersp'}]},
                        {'uses': [{'usedin': 'blahblah'}]}],
        'reporting': [{'stored_proc': 'reportingsp'},
                    {'uses': [{'usedin': 'breadcrumb'}]}]}

print mapObject(goes_okay).toString()
print mapObject(misses_dict_item).toString()
print mapObject(empty).toString()

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/32616881-objects-in-map-string-object-and-other-complex-containers-sometimes-disappear?utm_campaign=plugin&utm_content=tracker%2F77133&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F77133&utm_medium=issues&utm_source=github).

[ctrueden]

If you change the line:

map.put(str(key), mapObject(value))

to:

k = mapObject(key)
v = mapObject(value)
map.put(k, v)

Then it works.

Here is a more minimal example demonstrating the issue:

HashMap = autoclass('java.util.HashMap')

def new_key():
    return 'stuff'

def new_val():
    map_value = HashMap()
    map_value.put('foo', 'bar')
    return map_value

hm_good = HashMap()
k = new_key()
v = new_val()
hm_good.put(k, v)
print('good: ' + hm_good.toString())

hm_mid = HashMap()
k = new_key()
hm_mid.put(k, new_val())
print('val only: ' + hm_mid.toString())

hm_bad = HashMap()
hm_bad.put(new_key(), new_val())
print('bad: ' + hm_bad.toString())

Which outputs:

good: {stuff={foo=bar}}
val only: {}
bad: {}

If you change HashMap to ConcurrentHashMap or LinkedHashMap (or ArrayList, or Object, etc., with appropriate adjustments) the problem goes away in the minimal example. But still fails in the recursive function.

My Python-fu is not strong enough to understand why there could be a difference between including the function calls within the expression, versus assigning them to intermediate output variables. (I thought it might be a race condition, but adding a sleep call before returning the newly populated HashMap objects makes no difference.)

[AKuederle]

I ran into this issue multiple times now, and it is really annoying. Could we get this resolved somehow?

[ctrueden]

@AKuederle I implemented conversion methods similar to those discussed here, and published them on PyPI as part of the scyjava Python module. Could you give it a try and see if any of your conversions suffer from this issue? I wrote quite a few unit tests and was unable to find any failing cases. I avoided the issue by splitting the function calls across multiple lines of code, as shown by the above minimal example; additionally, I use LinkedHashMap rather than HashMap, which does not seem to suffer from the problem regardless.

I'd love to hear any ideas from others on why the following works:

HashMap = autoclass('java.util.HashMap')
hm_good = HashMap()
k = new_key()
v = new_val()
hm_good.put(k, v)
print('good: ' + hm_good.toString())

But this does not:

hm_bad = HashMap()
hm_bad.put(new_key(), new_val())
print('bad: ' + hm_bad.toString())

I am not a Python expert, but I find it hard to believe that a bug like this could really be on the Pyjnius side.

[AKuederle]

That's awesome! I will have a look. From my naive knowledge about Python, it appears to an issue with the garbage collection. Basically, the second you leave the current scope the object gets deleted. It feels like this might be connected to the bug with the refcounter increase reported in another bug. I am currently at mobile but I will link the issue tomorrow.

…

On Wed, Nov 28, 2018, 21:03 Curtis Rueden ***@***.*** wrote: @AKuederle <https://github.com/AKuederle> I implemented conversion methods similar to those discussed here, and published them on PyPI as part of the scyjava <https://pypi.org/project/scyjava/> Python module. Could you give it a try and see if any of your conversions suffer from this issue? I wrote quite a few unit tests and was unable to find any failing cases. I avoided the issue by splitting the function calls across multiple lines of code, as shown by the above minimal example; additionally, I use LinkedHashMap rather than HashMap, which does not seem to suffer from the problem regardless. I'd love to hear any ideas from others on why the following works: HashMap = autoclass('java.util.HashMap') hm_good = HashMap() k = new_key() v = new_val() hm_good.put(k, v)print('good: ' + hm_good.toString()) But this does not: hm_bad = HashMap() hm_bad.put(new_key(), new_val())print('bad: ' + hm_bad.toString()) I am not a Python expert, but I find it hard to believe that a bug like this could really be on the Pyjnius side. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#217 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKRwuyFVEMSsNRfmQFdHRX7F24x_MaaJks5uzuwNgaJpZM4IACro> .

[AKuederle]

here is the reference to the other issue: #345

[ctrueden]

@AKuederle Ahh, that makes sense! Then probably this issue should be closed in favor of #345, no?

[AKuederle]

I am not 100% sure. I have no way of testing it and also I think there is no proper solution. When passing things to a java method, there is no way of knowing if you should keep a reference to the object or not, right? If the method is just a simple function, then you don't need to increase the ref counter, but if the method actually stores the object on java side, you should increase the refcounter.

But I could be totally wrong here. For sure not my field of expertise.

[AKuederle]

Btw. This might be related as well: #59