Python elemental types not converted automatically when C# type is "object".

[sherryyshi]

Environment

• Pythonnet version: 3.0.1
• Python version: 3.9
• Operating System: Windows
• .NET Runtime: netcoreapp3.1

Details

According to https://pythonnet.github.io/pythonnet/python.html#type-conversion, most elemental Python types (string, int, long, etc.) convert automatically to compatible managed equivalents (String, Int32, etc.) and vice-versa. However, when the type on the C# side is object this does not seem to happen.

Here is a sample to demonstrate what I mean:

using Newtonsoft.Json;

namespace MyClass
{
    public record MyClass
    {
        public MyClass(int x)
        {
            X = x;
        }
        public object X { get; set; }

        public string ToJson() => JsonConvert.SerializeObject(this);

        public string GetType() => X.GetType().Name;
    }
}

from MyClass import MyClass

x = MyClass(1)
print(x.GetType())
print(x.ToJson())

This works; the output is:

Int32
{"X":1}

However, if I change public MyClass(int x) to public MyClass(object x) in the C# code:

using Newtonsoft.Json;

namespace MyClass
{
    public record MyClass
    {
        public MyClass(object x)
        {
            X = x;
        }
        public object X { get; set; }

        public string ToJson() => JsonConvert.SerializeObject(this);

        public string GetType() => X.GetType().Name;
    }
}

This no longer works as expected - type is PyInt and json serialization throws an error:

PyInt
Traceback (most recent call last):
  File "C:\repos\test\LongRunningTask\python\test.py", line 15, in <module>
    print(x.ToJson())
Newtonsoft.Json.JsonSerializationException: Self referencing loop detected with type 'Python.Runtime.PyObject'. Path 'X'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDynamic(JsonWriter writer, IDynamicMetaObjectProvider value, JsonDynamicContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value)
   at MyClass.MyClass.ToJson() in C:\repos\test\LongRunningTask\MyClass\MyClass.cs:line 13
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Is this behavior expected? If so, that would be unfortunate for me since I have code in C# that does runtime type checking (if (x is Int32) etc.) and JSON serialization.

[lostmsu]

This is by design to avoid ambiguous cases and surprising silent behavior.

Explicitly cast Python values to corresponding .NET types for such scenarios. e.g. System.Int32(42)

[sherryyshi]

Thanks @lostmsu for the quick response.

I see this block in the source code:

                if (Runtime.PyInt_Check(value))
                {
                    result = new PyInt(value);
                    return true;
                }

However, other primitive types like floats, bools etc. are handled with ToPrimitive:

                if (Runtime.PyFloat_Check(value))
                {
                    return ToPrimitive(value, doubleType, out result, setError);
                }

Indeed, if I try my sample with float it works as expected.

Why are ints special?

[lostmsu]

@sherryyshi because there is one type of int in Python, and many types of ints in .NET, but none of them match exactly. But Python's float is exactly .NET's double.

[sherryyshi]

"Explicitly cast Python values to corresponding .NET types for such scenarios. e.g. System.Int32(42)"

@lostmsu sorry to drag this up again this suggestion is incurring a significant performance cost for my application. I'm wondering if there is a better solution. And it is holding onto the GIL while this explicit cast is happening, which is not ideal for my case.

What if the type conversion code above checks the size of the int and creates an appropriate C# int based on the size? It might reduce perf but at least the behavior is "more correct", arguably.