Suggestion: Converters for Unity types

[applejag]

Description

Serializing Unity objects could be tricky. Can result in very unexpected results when doing a JsonConvert.Serialize on a Vector3.

Why we need this

Convert common Unity types to begin with instead of having to make custom wrapper DTO's for simple things as Vectors and Quaternion.. Then perhaps more complex objects such as GameObjects and Scenes as I've long ago seen others been able to do. But that's hoping. Basic first!

Suggested solution

New package containing converters, preferably in a separate repository. Perhaps a jilleJr/Newtonsoft.Json-for-Unity.Converters

Then perhaps registering the converters in a initialization script, such as using
RuntimeInitializeLoadType.AfterAssembliesLoaded:

RuntimeInitializeLoadType.AfterAssembliesLoaded
"Callback when all assemblies are loaded and preloaded assets are initialized."

https://docs.unity3d.com/ScriptReference/RuntimeInitializeLoadType.AfterAssembliesLoaded.html

Applied on the RuntimeInitializeOnLoadMethod attribute.

// Demonstration of RuntimeInitializeOnLoadMethod and the argument it can take.
using UnityEngine;

class MyClass
{
   [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
   static void OnBeforeSceneLoadRuntimeMethod()
   {
       Debug.Log("Before first Scene loaded");
   }

   [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterSceneLoad)]
   static void OnAfterSceneLoadRuntimeMethod()
   {
       Debug.Log("After first Scene loaded");
   }

   [RuntimeInitializeOnLoadMethod]
   static void OnRuntimeMethodLoad()
   {
       Debug.Log("RuntimeMethodLoad: After first Scene loaded");
   }
}

https://docs.unity3d.com/ScriptReference/RuntimeInitializeOnLoadMethodAttribute-ctor.html

Inspiration sources

Wanzyee's "Json.NET Converters - Simple compatible solution" is a great solution. The accessibility of having it as an UPM package next to Newtonsoft.Json-for-Unity would be grand though.

Forum post: https://forum.unity.com/threads/free-json-net-converters-simple-compatible-solution.459404/
Assets store: https://assetstore.unity.com/packages/tools/input-management/json-net-converters-simple-compatible-solution-58621

His score on search engines is superb. Collab?

@parentelement enlightened with his converters. They are a great starting ground! Let's go for that

[parentelement]

As the publisher of the original Json .Net for Unity port I'd be more than happy to submit PRs and assist with some bug fixes and converters. I can no longer update my package for compatibility reasons and because they're used by a third-party but I can contribute to yours. Feel free to lift any of my converters:
https://github.com/ianmacgillivray/Json-NET-for-Unity/tree/master/Source/Newtonsoft.Json/Converters

Note that there is currently a bug with the Matrix4x4 converter that I have not yet addressed but the others are solid. I can also help you work around a few of the platform specific issues I encountered over the years.

[applejag]

@parentelement Ah what a great honor! Somehow missed that the commits came from you. Have overlooked that repo multiple times for different reasons.

Those converters are a great starting ground! I've decided to place them in a different repo, but when that's set up I'll notify you :)

[sindrijo]

The internal JsonUtility can be leveraged to serialize/deserialize Unity's types. I just wrote this so I haven't tested it thoroughly but has worked with everything I've thrown at it so far. I haven't tested it for GC pressure but doing it like this might be worth it for the ease of maintainability?

The implementation of IsUnityEngineType(Type objectType) is maybe a bit naive...

using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using UnityEngine;

internal static class UnityTypeSupport
{
    
#if UNITY_EDITOR
    [UnityEditor.InitializeOnLoadMethod]
#else
    [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
#endif
    private static void Init()
    {
        JsonConvert.DefaultSettings += GetJsonSerializerSettings;
    }

    private static JsonSerializerSettings GetJsonSerializerSettings()
    {
        var settings = new JsonSerializerSettings();
        settings.Converters.Add(new UnityTypeConverter());
        return settings;
    }

    private class UnityTypeConverter : JsonConverter
    {
        private static readonly HashSet<Type> UnityEngineTypes;
        
        static UnityTypeConverter()
        {
            UnityEngineTypes = new HashSet<Type>(typeof(UnityEngine.Object).Assembly.GetTypes());
        }
        
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            writer.WriteRawValue(JsonUtility.ToJson(value));
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            return JsonUtility.FromJson(JObject.Load(reader).ToString(), objectType);
        }

        public override bool CanConvert(Type objectType)
        {
            return IsUnityEngineType(objectType);
        }

        private static bool IsUnityEngineType(Type objectType)
        {
            return UnityEngineTypes.Contains(objectType);
        }
    }
}

[applejag]

@sindrijo Oh hey that's smart! Actually using Unity's own serializer. So then it can actually serialize MonoBehaviours and such. Will probably anyway put this in a separate package but thanks so much for the tip!

[sindrijo]

@jilleJr No, it cannot serialize MonoBehaviours, but it should cover all other types like Color, Vector3, Quaternion, Matrix4x4 and so on.

JsonUtility.FromJson

Only plain classes and structures are supported; classes derived from UnityEngine.Object (such as MonoBehaviour or ScriptableObject) are not.

Actually that is a bit of a lie, you can serialize the 'serializable' data from a MonoBehaviour, that is what would normally be serialized, but of course there is no mechanism to serialize a whole GameObject or serialize a MonoBehaviour 'onto' a GameObject.

There is JsonUtility.FromJsonOverwrite but it requires an already existing object and has other restrictions.

Internally, this method uses the Unity serializer; therefore the object you pass in must be supported by the serializer: it must be a MonoBehaviour, ScriptableObject, or plain class/struct with the Serializable attribute applied. The types of fields that you want to be overwritten must be supported by the serializer; unsupported fields will be ignored, as will private fields, static fields, and fields with the NonSerialized attribute applied.

Any plain class or structure is supported, along with classes derived from MonoBehaviour or ScriptableObject. Other engine types are not supported. In the Editor only, you can use EditorJsonUtility.FromJsonOverwrite to overwrite other engine objects.

We could extend this code to support ScriptableObjects because there is an API for instantiating them: ScriptableObject.CreateInstance(Type)

[sindrijo]

Here is an update version which supports ScriptableObjects.

using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using Newtonsoft.Json.Linq;
using UnityEngine;

internal static class UnityTypeSupport
{
    
#if UNITY_EDITOR
    [UnityEditor.InitializeOnLoadMethod]
#else
    [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterAssembliesLoaded)]
#endif
    private static void Init()
    {
        JsonConvert.DefaultSettings += GetJsonSerializerSettings;
    }

    private static JsonSerializerSettings GetJsonSerializerSettings()
    {
        var settings = new JsonSerializerSettings();
        settings.Converters.Add(new UnityTypeConverter());
        settings.Converters.Add(new ScriptableObjectCreator());
        return settings;
    }

    private class UnityTypeConverter : JsonConverter
    {
        private static readonly HashSet<Type> UnityEngineTypes;
        
        static UnityTypeConverter()
        {
            UnityEngineTypes = new HashSet<Type>(typeof(UnityEngine.Object).Assembly.GetTypes());
        }
        
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            writer.WriteRawValue(JsonUtility.ToJson(value));
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            if (typeof(ScriptableObject).IsAssignableFrom(objectType))
            {
                JsonUtility.FromJsonOverwrite(JObject.Load(reader).ToString(), existingValue);
                return existingValue;
            }
            
            return JsonUtility.FromJson(JObject.Load(reader).ToString(), objectType);
        }

        public override bool CanConvert(Type objectType)
        {
            return IsUnityEngineType(objectType);
        }

        private static bool IsUnityEngineType(Type objectType)
        {
            return UnityEngineTypes.Contains(objectType);
        }
    }
    
    private class ScriptableObjectCreator : CustomCreationConverter<ScriptableObject>
    {
        public override ScriptableObject Create(Type objectType)
        {
            return ScriptableObject.CreateInstance(objectType);
        }
    }
}

[sindrijo]

One more thing: JsonWriter.WriteRawValue(jsonString) might not take into account formatting settings.

[applejag]

Cool! I will add some tests for this and confirm.

[applejag]

@sindrijo This solution is very good starting ground, much love. However during some tests I found something I didn't even know about Unity's JsonUtility. It can't handle a lot of different types, such as Vector2Int, Vector3Int, Rect, RectInt, Bounds, BoundsInt, NativeArray<>.

A combination solution is needed. Not too biggie. Perhaps better to use custom converters for all Unity types and not use Unity's JsonUtility as converting back and forth between Json can't be optimal at all. /shrug

Edit: Example of failing test:

[Test]
public void SerializesAsExpected()
{
    // Arrange
    JsonSerializerSettings settings = GetSettings();
    Bounds input = new Bounds(new Vector3(1, 2, 3), new Vector3(4, 5, 6));
    string expected = @"{""xMin"":-1.0,""yMin"":-0.5,""zMin"":0.0,""sizeX"":4.0,""sizeY"":5.0,""sizeZ"":6.0}";

    // Act
    string result = JsonConvert.SerializeObject(representation.input, settings);

    // Assert
    Assert.AreEqual(representation.expected, result);
}

Actual result from within Unity:

SerializesAsExpected() (0,001s)
---
Expected string length 72 but was 2. Strings differ at index 1.
  Expected: "{"xMin":-1.0,"yMin":-0.5,"zMin":0.0,"sizeX":4.0,"sizeY":5.0,"sizeZ":6.0}"
  But was:  "{}"
  ------------^
---

[applejag]

I've now released version 1.0.0 of the Newtonsoft.Json-for-Unity.Converters package over at https://github.com/jilleJr/Newtonsoft.Json-for-Unity.Converters

If you still want to help @parentelement then there's still stuff to do there for future releases ;)

Thanks both of you @parentelement and @sindrijo, you gave great inspiration and tips. In the end I skipped usage of the UnityEngine.JsonUtility for performances sake.

Closing this as resolved.