[Python-Java-Interop] default type conversions

[revintec]

Context context=Context.newBuilder().allowAllAccess(true).build();
Value value=context.eval("python","[\"1\",2,3]");
Object obj=value.as(Object.class);List list=value.as(List.class);
System.out.println(obj.getClass()+": "+((Map)obj).size());
System.out.println(list.getClass()+": "+list.size());

as the code above shown, ["1",2,3] is a Python <class 'list'>
however when converted to Java object, it became a PolyglotMap(with zero size)
instead of Object[] or List

is this behavior intended? if so, is there any way to customize it to return Object[] or List?
when used like this: JavaMap getMap(){...}; Pythondict=getMap();dict.put("key",["1",2,3])
it corrupts the data inside the map values

---

and another related Java-Python-Interop question:

context.getBindings("python").putMember("getMap",(Supplier<Map>)HashMap::new);
try{context.eval("python","dict=getMap();dict[\"a\"]");}catch(Throwable e){e.printStackTrace();}
try{context.eval("python","dict=getMap();dict[\"a\"]=1");}catch(Throwable e){e.printStackTrace();}

is there a way to convert Java Map to PolyglotMap so we can use it like a dict in Python?
currently I'm recursively wrapping them to ProxyObject, but it's cumbersome and pollutes map values

KeyError: invalid key a
	at <python> <module 'Unnamed'>(Unnamed:1:14-22)
	at org.graalvm.polyglot.Context.eval(Context.java:340)
	at BugReport.main(BugReport.java:21)
AttributeError: JavaObject[{} (java.util.HashMap)] instance has no attribute '__setitem__'
	at <python> <module 'Unnamed'>(Unknown)
	at org.graalvm.polyglot.Context.eval(Context.java:340)
	at BugReport.main(BugReport.java:22)

[timfel]

Hi,
unfortunately, default conversions are not possible at this point. All objects from other languages (Java or other Truffle languages) are treated through the same API, and if an object says it has keys, it is a Map.

We also cannot implement both the List and Map interfaces for a PolyglotMapList, unfortunately. You will have to use the Value.as api to convert the object.

We also don't support accessing java Maps like dicts directly. You can use the Map.get method to read from the Java map.

[timfel]

when used like this: JavaMap getMap(){...}; Pythondict=getMap();dict.put("key",["1",2,3])
it corrupts the data inside the map values

I'm not entirely sure what you mean by that. The data should be inserted as Value types.

[revintec]

@timfel sorry for the ambiguity, basically I meant two things:

1. default conversion for Python <class 'list'> is unintuitive:
   <class 'list'> should be converted to Object[] or Collection/List by default
   but instead it was converted to a Map, and that is because <class 'list'> reports true for hasKeys
   it's indeed the same behavior for Javascript, considering you can do array=[];array["key"]=1
   1.1) but the I think returning Object[] or Collection/List by default would be better
   since we rarely need it to be a Map, and if we do, we can always convert it manually
   1.2) if instead we got a Map, it shouldn't has no data in it, that's the problem

the following code is run on GVM-1.0.0-RC6, records empty PolyglotMap instead of a Value

        Context context=newBuilder().allowAllAccess(true).build();
        Map<String,Object>map=new HashMap<>(); // or other collections
        context.getBindings("python").putMember("map",map);
        context.eval("python","map.put(\"key\",[1,2,3])");

relevant code: com.oracle.truffle.polyglot.ToHostNode#convertToObject

---

2. making Map interoperable between Python and Java would make things smoother
   thus we can write map["key"]="value" instead of map.put("key","value")
   and Map would resembles more closely a Python dict

if GraalPython team decided to not implement this, how can I implement it myself?
I think I can use com.oracle.truffle.api.instrumentation.Instrumenter to rewrite the AST
change __setitem__ related node to try Map.put if it's a HostObject of type Map
but I haven't pulled it off, could you help and give me a code snippet?

[timfel]

Hi, sorry for the delay, we looked again at this:

Re 1: A Java object cannot implement both the List and the Map interface at the same time, they are incompatible. So in the presence of an object with keys (which all Python object have), Java always picks Map. There is a bug here in Python that we don't report inherited keys, which we will fix.

Re 2: One problem is that this is currently not possible to implement with the API we have. There is only one message Python can send to "foreign" (that includes Java) objects to get at it's contents: KEYS. In order to merge the namespaces of the Map with the object attribute and method namespace, this always returns the object attributes for Host objects, which is not something that the Python language controls. There is no way us to access the map directly (e.g. through []), because from the Python world, we cannot actually know what method we'd have to call to get to an item. The only thing you can do is write your own Python class and wrap incoming objects.

[revintec]

@timfel Hi, thanks for the reply, please correct me if I misunderstood something

1. you've decided to convert [1,2,3] to Java map, either {"1":1,"2":2,"3",3} or {"0":1,"1":2,"2":3}, because all Python objects have keys, so only a Java map is suited for a lossless conversion
2. I don't quite understand, if we discover that we are accessing a Java map, could we rewrite that specific AST node to use Map.put/get instead of __setitem__/__getitem__ messages?

[timfel]

re 1: a list will be a Java map but with the list methods as keys ("append", "extend", etc). To get the items, you have to use e.g. "as(Object[])" or some such. You can also read single items using a READ with integer indices, but to get all items in a Java array, you have to do the conversion.

re 2: we can probably do something to access Java maps a little more conveniently. But in general we won't be able to offer automatic mappings for arbitrary types. The simplest way is probably to just change the TruffleObjectBuiltin we have for __getitem__/__setitem__ do check if we're dealing with a Java Map object

[revintec]

for 1: I still think converting to a list is better, please see my original post, emphasising map.put(...):
#21 (comment)

        Context context=newBuilder().allowAllAccess(true).build();
        Map<String,Object>map=new HashMap<>(); // or other collections
        context.getBindings("python").putMember("map",map);
        context.eval("python","map.put(\"key\",[1,2,3])");

for 2: sounds great, if it won't hinder performance, would this feature be added?

[timfel]

Re 1: You can always use get with an integer key and it will work, it just doesn't report the indices as KEYS (because that's not allowed by the API).

[revintec]

thx, I'll try when new versions are released