Add support to serializeNulls in the DefaultSerializer/GSONFactory

[ddellsperger]

We're currently in the progress of migrating from the 2.x graph-sdk-java to the 4.x branch of the graph-sdk-java, but the changes that we're seeing mostly are impacted by the serializer in the graph-sdk-java-core here. We previously used the raw JSON object data that was stored in the IJsonBackedObject implementation classes to get the exact response from the rest call. This provided us with a one-to-one representation of the API calls that the graph API returns. With the 3.x line on, this behavior changes as we migrate to the DefaultSerializer as the DefaultSerializer changes the representation of nulls (that are attributes of the Java object, but not that are in the additionalDataManager) to no longer serialize to a string. Since the change in 2.x to 3.x+ of the msgraph-sdk-java we're going to lose some of our support downstream in our other services that may not be java due to this change in structure. There have also been some changes to not really enable the leveraging of things like the GSONFactory which makes extending the ISerializable interface difficult. I'm wondering if it's possible to add an option to the DefaultSerializer to enable the gson option to serlialize nulls. I'm happy to submit a PR, but want to make sure I'm not potentially causing a disruption in a class that may not be accepted due to the desired behavior within the core sdk.

Expected behavior

Serialization of api responses via the SDK to at least provide the fields that the api provides in response.

Actual behavior

Serialization of the JSON objects throws away most of the null values

Steps to reproduce the behavior

Compare a response for something like the sendMessage Teams API serialized via the DefaultSerializer to the examples in the API documentation.
AB#10484

[baywet]

Hey @ddellsperger
Thanks for reaching out.
So if I'm understanding things right, you'd like for the serializer to result with

{
 "propA": null,
} 

when you do serialize a model with myModel.additionalDataManager().put("propA", null);
Or did I misunderstand something?

[ddellsperger]

So, there's a few, if I look at the First Example for the Send chatMessage API, the response provided by the API is as follows:

{
    "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#teams('fbe2bf47-16c8-47cf-b4a5-4b9b187c508b')/channels('19%3A4a95f7d8db4c4e7fae857bcebe0623e6%40thread.tacv2')/messages/$entity",
    "id": "1616990032035",
    "replyToId": null,
    "etag": "1616990032035",
    "messageType": "message",
    "createdDateTime": "2021-03-29T03:53:52.035Z",
    "lastModifiedDateTime": "2021-03-29T03:53:52.035Z",
    "lastEditedDateTime": null,
    "deletedDateTime": null,
    "subject": null,
    "summary": null,
    "chatId": null,
    "importance": "normal",
    "locale": "en-us",
    "webUrl": "https://teams.microsoft.com/l/message/19%3A4a95f7d8db4c4e7fae857bcebe0623e6%40thread.tacv2/1616990032035?groupId=fbe2bf47-16c8-47cf-b4a5-4b9b187c508b&tenantId=2432b57b-0abd-43db-aa7b-16eadd115d34&createdTime=1616990032035&parentMessageId=1616990032035",
    "policyViolation": null,
    "from": {
        "application": null,
        "device": null,
        "conversation": null,
        "user": {
            "id": "8ea0e38b-efb3-4757-924a-5f94061cf8c2",
            "displayName": "Robin Kline",
            "userIdentityType": "aadUser"
        }
    },
    "body": {
        "contentType": "text",
        "content": "Hello World"
    },
    "channelIdentity": {
        "teamId": "fbe2bf47-16c8-47cf-b4a5-4b9b187c508b",
        "channelId": "19:4a95f7d8db4c4e7fae857bcebe0623e6@thread.tacv2"
    },
    "attachments": [],
    "mentions": [],
    "reactions": []
}

When this runs through the DefaultSerializer we get the following output:

{
    "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#teams('fbe2bf47-16c8-47cf-b4a5-4b9b187c508b')/channels('19%3A4a95f7d8db4c4e7fae857bcebe0623e6%40thread.tacv2')/messages/$entity",
    "id": "1616990032035",
    "etag": "1616990032035",
    "messageType": "message",
    "createdDateTime": "2021-03-29T03:53:52.035Z",
    "lastModifiedDateTime": "2021-03-29T03:53:52.035Z",
    "importance": "normal",
    "locale": "en-us",
    "webUrl": "https://teams.microsoft.com/l/message/19%3A4a95f7d8db4c4e7fae857bcebe0623e6%40thread.tacv2/1616990032035?groupId=fbe2bf47-16c8-47cf-b4a5-4b9b187c508b&tenantId=2432b57b-0abd-43db-aa7b-16eadd115d34&createdTime=1616990032035&parentMessageId=1616990032035",
    "from": {
        "conversation": null,
        "user": {
            "id": "8ea0e38b-efb3-4757-924a-5f94061cf8c2",
            "displayName": "Robin Kline",
            "userIdentityType": "aadUser"
        }
    },
    "body": {
        "contentType": "text",
        "content": "Hello World"
    },
    "channelIdentity": {
        "teamId": "fbe2bf47-16c8-47cf-b4a5-4b9b187c508b",
        "channelId": "19:4a95f7d8db4c4e7fae857bcebe0623e6@thread.tacv2"
    },
    "attachments": [],
    "mentions": [],
    "reactions": []
}

You'll notice that the subject, summary, chatId, lastEditedDate, deletedDateTime, and a few other fields are missing, but the conversation within the from is still present as a null value.

The from.conversation is coming through as the conversation is stored in the additionalDataManager and not part of the core IdentitySet entity within the graph models (I know this isn't part of the core, but it's an example). The GSON serializer apparently serializes items which are JSONNull but not attributes of Java classes that have a value of null, by default. With our previous usage, we were able to represent this JSON fully as was we received in the response from the API as the JSON objects were accessible from the previous SDK API via the getRawObject() methods, this is no longer a viable options for us as that was removed in the 3.x release of msgraph-sdk-java (which makes sense), but we can no longer get the existing null fields as the DefaultSerializer does not provide a way for us to expose those null fields and maintain some passivity with our previous behavior.

My request here (and I can provide a PR) would be that we add a parameter to the DefaultSerializer to support serializing nulls, or at least expose a method within the GSONFactory.getGsonInstance and expose that a bit more so that others can leverage that GSONFactory for the other serializers that it contains (for DateTime, enum, etc) as another way to potentially implement the ISerializer interface.

[baywet]

Thank you for all this additional information, this is really valuable. Before we dive into putting a PR together I want to understand the problem better so we can come up with a clear solution and avoid wasting your time.

Is behavior this limited to things related to identityset? we have a global ongoing effort to fix those on the service side. Basically a lot of services used the generic identityset type when they should have come up with their own and added any missing property (so those properties exist in the SDKs).

This specific "from" property used to be of type IdentitySet in V3 and is now of type ChatMessageFromIdentitySet in V4 (and this is why we had to major bump the version).

Now, if we look at the service metadata we can see the "conversation" property is still missing from that type. So what's effectively happening is that the SDK sees a null value in the payload, and sticks it in the additional data manager during deserialization. During serialization, it takes the values in the additional data manager as is, and adds it to the payload.
While the serialization behavior is expected, I don't think we should store the null value in the additional data manager during deserialization (this is inconsistent behavior compared to properties that exist in the type).

There are still a few things I don't understand at this point:

• Why does the service reply with all those null values? Instead of simply not attaching the property to the payload.
• Why is the conversation property missing from the metadata?
• Why do you need to access null values? (the difference between, no information was provided in the payload and, an information was provided, it was null)

Thanks for all this information!

[ddellsperger]

There are a few others that do this (at least return the nulls in the API and then due to the serializer behavior remove them in the JSON serialization), We've started with migrating our teams-specific API and are still working with some of the e-mail apis and onedrive apis, but from Teams, I have seen the following:

• Example 2 and 3 in the Create Channel Response
• I think it was more the underlying api changing for the List Joined Teams response that got us a little tied because the underlying Team object is shared between that response and the Get Team response.

The reason for our desire to keep the fields with null values is mostly to better support down-stream auto-schema from responses. When we lose those fields that may be present, it means our schema is potentially disconnected from the downstream consumers, which leads to some disconnect and more logic to check for fields prior to checking for values. It's the core reason the serializeNulls option exists for the GSONBuilder in order to support the varying needs for some of the different ways that behaviors operate with serialized JSON, if our downstream consumer is converting to something like avro for further migration that schema has to then be updated to support these changes with respect to nulls being present or not. This isn't an issue for those who previously used the DefaultSerializer within the system, but only those who previously went through the getRawObject api that previously existed.

[baywet]

Can you expand on the current logic with v2 service library?
Right now property X is null (java object), so your app uses the get raw object, looks for property X, and check whether there's a null entry for it. What happens if there's an entry? what happens if there isn't?

[ddellsperger]

We actually currently call getRawObjects().toString() which gives us the String json representation of the response (it's a one-to-one mapping response from the internal API call), we then use Jackson to convert that to a Map<String, Object> which gives us internal jackson transmissibility to our downstream consumers. That, in essence relied far more heavily on the specific response coming from the API, but provided an easy way for our downstream consumers to know what data to expect from the response based on the Microsoft documentation.

[baywet]

oh wow, that's actually a niche scenario, your app is actually acting as a proxy. In which case, why does your app need to get high level deserialized objects instead of the JSON payload directly? This deserialization/serialization sequence surely adds a lot of overhead.
And still, why do the consumers need the null values at all?

[ddellsperger]

And still, why do the consumers need the null values at all?

This is mostly due to the need for consistency, if one example has a value for a field, our consumers assume that field will always be available (this has been the case for us with the v2 api for us. With the v3+ api the structure of the response may change with each reply depending on which properties are non-null or included in the Java Object implementation, specifically (see conversation in our previous examples). Our desire would be to simply serialize all nulls so attribute access doesn't error out, but value access can be null-checked. Though that also means that internal Java object changes are more impactful in the future.

[baywet]

So you'd be interested in getting all the null values from the object tree, not just the ones provided be the API originally?

[ddellsperger]

I would prefer only those that the API reports back, but all nulls would be better than a non-deterministic (at least without knowledge of Java implementation) subset of them.