Function for converting Python values to an explicit Java type #128

rbasralian · 2024-01-10T16:22:04Z

Fix a bug in cast
Add an as_jobj method to convert a Python value to a Java object of the given type. This leverages the existing JPy_AsJObjectWithType method.
Update some tests to match current behavior (e.g. returning Byte instead of Integer)
More logging during tests
Fixed failing hashNegativeOne test
Replace incorrect return Py_NONEs with Py_RETURN_NONE (for correct reference counting)

…type

…for extracting a JPy_JObject's declared type.

…nd as_jobj_internal.

# Conflicts: # jpyutil.py

devinrsmith · 2024-01-10T20:56:22Z

src/main/c/jpy_module.c

@@ -57,10 +59,19 @@ static PyMethodDef JPy_Functions[] = {
                    "get_type(name, resolve=True) - Return the Java class with the given name, e.g. 'java.io.File'. "
                    "Loads the Java class from the JVM if not already done. Optionally avoids resolving the class' methods."},

+    {"get_type_name",    (PyCFunction) JPy_get_type_name, METH_VARARGS,


Do we need this? You can currently do type(jobj).jclass.getName(). This does involve a boundary crossing though.

If we think it's important to have w/o boundary crossing, and explicitly meant for jobj types, instead of adding jpy top level method, we can attach an attribute to the JPy_JType. From end user perspective, it would look something like:

# Sources from JPy_JType->javaName type(jobj).jclassname

would be appropriate. See logic in JType_AddClassAttribute.

I'm actually surprised that

type(jobj).__name__

isn't the full classname; https://docs.python.org/3/c-api/typeobj.html#tp-slots hints that that should be tp_name.

And our code has this:

typedef struct JPy_JType { ... // typeObj.tp_name is this type's fully qualified Java name (same as 'javaName' field). PyTypeObject typeObj; // The Java type name. char* javaName;

Something screwy is going on though:

>>> jpy.get_type('java.lang.CharSequence').__name__ 'CharSequence' >>> jpy.get_type('java.math.BigInteger').__name__ 'BigInteger'

I'll do some further digging, b/c this seems odd...

https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_name

Looks like

>>> f"{type.__module__}.{type.__name}"

is the way to get the java class name; this doesn't quite work for primitives (not sure what we expect to happen w/ primitives).

I don't think primitives apply — I don't think you can wrap an unboxed primitive value in a JPy_JObj.

I got rid of this method and added type(obj).jclassname.

jpyutil.py

devinrsmith · 2024-01-10T21:25:53Z

src/main/c/jpy_module.c

+    if (resultObj == NULL) {
+        return NULL;
+    }


We should delete the global ref if this happens.

Thanks — fixed

devinrsmith · 2024-01-10T21:30:57Z

src/main/c/jpy_module.c

    {"cast",        JPy_cast, METH_VARARGS,
                    "cast(obj, type) - Cast the given Java object to the given Java type (type name or type object). "
                    "Returns None if the cast is not possible."},

+    {"as_jobj",     JPy_as_jobj, METH_VARARGS,


This is cool - previously, we could do this, it just required explicit java code:

public class MyClass { public static MyJavaType identity(MyJavaType x) { return x; } }

jobj = jpy.get_type('MyClass').identity(pobj)

I'm assuming these are the semantics you are after?

Yeah, basically I had two parallel lists, one of java types and one of python values (in practice strings/numbers), and I wanted to stick the python values into a java Object[] with the given Java type, but numbers I wanted as Long were getting converted to Byte. I could have just used the constructors explicitly but that's not as clean, and it seems worthwhile to expose JType_ConvertPythonToJavaObject anyway.

I am not too crazy about the name as_jobj, would conv be better?
From what I can gather, I do like that I can now create a org.jpy.PyObject in Python directly.

I'm definitely open to changing it. I went with as_jobj because it ultimately calls JPy_AsJObjectWithType, and I thought having 'obj' in the name explicitly could be helpful because conversion in Java is a distinct concept (though a related one) that also involves Java primitives.

@devinrsmith what do you think?

I think "as" gives off the wrong semantic notion, somewhat akin to zero-cost. While the same could be said about JPy_AsJObjectWithType, that's lower level and not exposed to the end user. Also, it calls down into JType_ConvertPythonToJavaObject.

I'm also not the biggest fan of trying to save a few characters; my vote would be for jpy.convert instead of jpy.conv / jpy.as_jobj.

devinrsmith · 2024-01-10T21:42:32Z

src/test/python/jpy_typeconv_test.py

+        self.assertEqual(fixture.stringifyObjectArg(my_jcharseq2), 'String(testStr)')
+
+
+    def test_ToObjectConversionTyped(self):


I think trying to exercise some of the other logic from JType_ConvertPythonToJavaObject would be nice; arrays, boolean, char, PyObject, String.

Added more tests. The test for converting explicitly to PyObject doesn't run, apparently due to an issue with the classpath

…n why it doesn't work.

devinrsmith

I don't think your changes to the JVOID / JNULL macros match with their indented use cases. While technically the results will be the same, the usage semantics are wrong. I would revert all these changes.

As you've probably noticed, the jpy tests aren't very robust. Ideally, we'd get them to run in GH CI here, but that hasn't been done yet.

devinrsmith · 2024-01-16T19:44:44Z

src/main/c/jpy_jtype.c

+    } else if (JPy_IS_CLONG(pyArg) && (*jenv)->IsAssignableFrom(jenv, JPy_Long_JClass, type->classRef)) {
        return JType_CreateJavaLongObject(jenv, type, pyArg, objectRef);
-    } else if (PyFloat_Check(pyArg) && (type == JPy_JObject || ((*jenv)->IsAssignableFrom(jenv, JPy_Double_JClass, type->classRef)))) {
+    } else if (PyFloat_Check(pyArg) && (type == JPy_JObject || (*jenv)->IsAssignableFrom(jenv, JPy_Double_JClass, type->classRef))) {
        return JType_CreateJavaDoubleObject(jenv, type, pyArg, objectRef);
-    } else if (PyFloat_Check(pyArg) && (type == JPy_JObject || ((*jenv)->IsAssignableFrom(jenv, JPy_Float_JClass, type->classRef)))) {
+    } else if (PyFloat_Check(pyArg) && (*jenv)->IsAssignableFrom(jenv, JPy_Float_JClass, type->classRef)) {


Can you comment on the effect of this? It looks like you removed the JPy_JObject condition from Long / Float, but not Double?

the cases I changed were:

} else if (JPy_IS_CLONG(pyArg) && (type == JPy_JObject || (*jenv)->IsAssignableFrom(jenv, JPy_Number_JClass, type->classRef))) { return JType_CreateJavaNumberFromPythonInt(jenv, type, pyArg, objectRef); } else if (JPy_IS_CLONG(pyArg) && ((*jenv)->IsAssignableFrom(jenv, JPy_Integer_JClass, type->classRef))) { return JType_CreateJavaIntegerObject(jenv, type, pyArg, objectRef); } else if (JPy_IS_CLONG(pyArg) && (type == JPy_JObject || ((*jenv)->IsAssignableFrom(jenv, JPy_Long_JClass, type->classRef)))) { return JType_CreateJavaLongObject(jenv, type, pyArg, objectRef);

and:

} else if (PyFloat_Check(pyArg) && (type == JPy_JObject || ((*jenv)->IsAssignableFrom(jenv, JPy_Double_JClass, type->classRef)))) { return JType_CreateJavaDoubleObject(jenv, type, pyArg, objectRef); } else if (PyFloat_Check(pyArg) && (type == JPy_JObject || ((*jenv)->IsAssignableFrom(jenv, JPy_Float_JClass, type->classRef)))) { return JType_CreateJavaFloatObject(jenv, type, pyArg, objectRef); }

in both case, the second type == JPy_JObject test will always be false — for the first case, we will have already landed in JType_CreateJavaNumberFromPythonInt and in the second case we'd hit JType_CreateJavaDoubleObject.

devinrsmith · 2024-01-16T19:52:16Z

src/main/c/jpy_module.c

@@ -451,7 +457,7 @@ PyObject* JPy_create_jvm(PyObject* self, PyObject* args, PyObject* kwds)
    if (JPy_JVM != NULL) {
        JPy_DIAG_PRINT(JPy_DIAG_F_JVM + JPy_DIAG_F_ERR, "JPy_create_jvm: WARNING: Java VM is already running.\n");
        JPy_DECREF(options);
-        return Py_BuildValue("");
+        return JPy_FROM_JVOID();


JPy_FROM_JVOID is meant to be used in combination with void.class type; it's not appropriate to use it in these contexts.

Replaced the handful of Py_BuildValue()s I changed with a new JPy_PY_NONE()

devinrsmith · 2024-01-16T19:53:44Z

src/main/c/jpy_module.c

        return NULL;
    }

    if (obj == Py_None) {
-        return Py_BuildValue("");
+        return JPy_FROM_JNULL();


This is not appropriate to use here unless we are converting from a null java object.

devinrsmith · 2024-01-16T19:54:07Z

src/main/c/jpy_module.c

    } else {
-        return Py_BuildValue("");
+        return JPy_FROM_JNULL();


devinrsmith · 2024-01-16T20:03:14Z

src/test/java/org/jpy/PyObjectTest.java

+         */
+//        System.out.println(obj.getObjectValue());
+
+//        assertEquals(-1, obj.getIntValue()); // TODO: should this be 'result.getIntValue()'?


Yes; this should be result.getIntValue()

Thanks — updated & reenabled the test

devinrsmith · 2024-01-16T20:05:21Z

src/test/python/jpy_typeconv_test.py

+
+    def test_AsJobjToPyObject(self):
+        print('Starting test_AsJobjToPyObject')
+        # TODO: "Java class 'org.jpy.PyObject' not found"???


This makes sense; PyObject is only necessary if the user needs to reference python from java. At least with this test scaffolding, the jpy jar is not on the classpath.

Ok — I moved this to a separate file (jpy_typeconv_test_pyobj.py).

…`FROM_JNULL`)

…spath that works).

devinrsmith

LGTM. Adding Jianfeng for final pass.

jmao-denver

The changes LGTM, not sure if I like the name very much.

jmao-denver · 2024-01-18T16:02:16Z

src/main/c/jpy_module.c

    {"cast",        JPy_cast, METH_VARARGS,
                    "cast(obj, type) - Cast the given Java object to the given Java type (type name or type object). "
                    "Returns None if the cast is not possible."},

+    {"as_jobj",     JPy_as_jobj, METH_VARARGS,


I am not too crazy about the name as_jobj, would conv be better?
From what I can gather, I do like that I can now create a org.jpy.PyObject in Python directly.

devinrsmith

LGTM.

jmao-denver

I am happy with the name change.

rbasralian added 12 commits January 8, 2024 20:49

Add method to convert Python value to a Java object of the specified …

9bdbaeb

…type

make some return values more clear/meaningful

06df869

update some tests to match current behavior

d599bf0

Fix bug when casting using a type name (not a JPy_JType). Add method …

1676e46

…for extracting a JPy_JObject's declared type.

Make as_jobj actually work. Clarify variable names in cast_internal a…

d43e8df

…nd as_jobj_internal.

Update .gitignore

20444d6

Remove a stray print()

5a3fdf0

dev version bump

ea60b1f

update types in test

61bde34

comments on failing test

877a99b

more logging for tests

36ed7a0

Merge remote-tracking branch 'origin/master' into raffi_obj_conv

b49090b

# Conflicts: # jpyutil.py

rcaudy requested a review from devinrsmith January 10, 2024 16:55

rcaudy assigned rbasralian Jan 10, 2024

devinrsmith reviewed Jan 10, 2024

View reviewed changes

rbasralian added 10 commits January 10, 2024 23:18

Merge remote-tracking branch 'origin/master' into raffi_obj_conv

3dea91c

Fix potential memory leak

0996fac

Replace get_type_name() with .jclassname attribute

2794f6b

Clean up redundant conditions/extra parentheses

e20757b

more tests

525362d

Replace deprecated assertion

2d252c3

formatting

c25ab7c

Tests for converting to primitive and boxed arrays

eaff9d6

Update test for explicitly converting to PyObject. Add more details o…

dc161cf

…n why it doesn't work.

Update deprecated assertion

d204645

rbasralian requested a review from devinrsmith January 16, 2024 19:20

devinrsmith reviewed Jan 16, 2024

View reviewed changes

rbasralian added 3 commits January 17, 2024 11:11

fix broken test

1d884a8

add a macro for building Python Nones (separate fromFROM_JVOID and …

c61869d

…`FROM_JNULL`)

comment out nonfunctional test

010feaf

rbasralian added 4 commits January 17, 2024 11:37

update test to support newer JVMs

b060d17

Move convert-to-PyObject test to another file (and run it with a clas…

bdd1f72

…spath that works).

consistent capitalization in print()s

efab329

Fix incorrect returns of Py_NONE

4182956

rbasralian requested a review from devinrsmith January 17, 2024 17:46

devinrsmith reviewed Jan 18, 2024

View reviewed changes

devinrsmith requested a review from jmao-denver January 18, 2024 15:43

jmao-denver reviewed Jan 18, 2024

View reviewed changes

Rename as_jobj to convert

f364d46

devinrsmith approved these changes Jan 22, 2024

View reviewed changes

rbasralian requested a review from jmao-denver January 23, 2024 17:08

jmao-denver approved these changes Jan 24, 2024

View reviewed changes

devinrsmith merged commit 1331a91 into jpy-consortium:master Jan 24, 2024
27 checks passed

rbasralian deleted the raffi_obj_conv branch January 24, 2024 21:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Function for converting Python values to an explicit Java type #128

Function for converting Python values to an explicit Java type #128

rbasralian commented Jan 10, 2024 •

edited

devinrsmith Jan 10, 2024

devinrsmith Jan 10, 2024

rbasralian Jan 11, 2024

devinrsmith Jan 10, 2024

rbasralian Jan 11, 2024

devinrsmith Jan 10, 2024

rbasralian Jan 11, 2024

jmao-denver Jan 18, 2024

rbasralian Jan 19, 2024

devinrsmith Jan 19, 2024

devinrsmith Jan 10, 2024

rbasralian Jan 12, 2024

devinrsmith left a comment

devinrsmith Jan 16, 2024

rbasralian Jan 17, 2024

devinrsmith Jan 16, 2024

rbasralian Jan 17, 2024

devinrsmith Jan 16, 2024

devinrsmith Jan 16, 2024

devinrsmith Jan 16, 2024

rbasralian Jan 17, 2024

devinrsmith Jan 16, 2024

rbasralian Jan 17, 2024

devinrsmith left a comment

jmao-denver left a comment

jmao-denver Jan 18, 2024

devinrsmith left a comment

jmao-denver left a comment

		self.assertEqual(fixture.stringifyObjectArg(my_jcharseq2), 'String(testStr)')


		def test_ToObjectConversionTyped(self):

Function for converting Python values to an explicit Java type #128

Function for converting Python values to an explicit Java type #128

Conversation

rbasralian commented Jan 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devinrsmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devinrsmith left a comment

Choose a reason for hiding this comment

jmao-denver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devinrsmith left a comment

Choose a reason for hiding this comment

jmao-denver left a comment

Choose a reason for hiding this comment

rbasralian commented Jan 10, 2024 •

edited