Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble deserializing an array #1

Closed
omatrot opened this issue Sep 5, 2019 · 19 comments

Comments

@omatrot
Copy link

commented Sep 5, 2019

Hi @PMDiademe.
I'm trying to deserialize a complex object array with circular references.

The code is like the following:

          RefCycleDetectionEnable();
          const datis: Vehicule[] = DeserializeArray(
            message.listDatiOfCompany,
            () => Vehicule
          );
          RefClean();
          RefCycleDetectionDisable();

It fails with the following error:

  Message=Error: Expected input to be an array but received: object
  Source=
  StackTrace:
Error: Expected input to be an array but received: object
    at DeserializeArrayInternal (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:441:15)
    at DeserializeInternal (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:627:35)
    at DeserializeInternal (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:636:35)
    at DeserializeInternal (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:636:35)
    at DeserializeArrayInternal (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:461:30)
    at Object.deserializationContinuation (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:1620:20)
    at DeserializeArray (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:769:20)
    at SafeSubscriber._next (https://localhost:44338/src-app-components-dispositifsDATI-dispositifsDATI-module.js:8016:98)
    at SafeSubscriber.__tryOrUnsub (https://localhost:44338/vendor.js:235125:16)
    at SafeSubscriber.next (https://localhost:44338/vendor.js:235064:22)

In the debugger, I can see that the last call to DeserializeArrayInternal receives null for the first parameter data, and undefined for the target parmeter, The code throws because data is not an array.

I'm looking for guidance on how to troubleshoort the problem.

What can be the cause of the problem? A discrepencies between the typescript class and the c# class regarding serialization?

Thansks for your help.

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

Ok, I have overcomed this problem configuring the JSonSerializer to ignore null reference.

Now I have another problem where references are not found on the current objet, which is common when you deserialize an array where the object is not in the current object tree.

"Reference found before its definition"

 deserializationGetObject(json) {
        if (json.hasOwnProperty("$ref")) {
            const ref = parseInt(json.$ref, 10);
            if (!this.cycle.ref2obj.has(ref)) {
                throw new Error("Reference found before its definition");
            }
            return this.cycle.ref2obj.get(ref);
        }
        return undefined;
    }
@PMDiademe

This comment has been minimized.

Copy link
Collaborator

commented Sep 5, 2019

Hi,
I tried the following code, and it works:

class Vehicule {
    @autoserializeAs(() => Vehicule)
    public other: Vehicule;
}
const v1 = new Vehicule();
const v2 = new Vehicule();
v1.other = v2;
v2.other = v1;
// serialized
RefCycleDetectionEnable();
const list = SerializeArray([v1, v2], () => Vehicule);
const stringList = JSON.stringify(list); // "[{"$id":"1","other":{"$id":"2","other":{"$ref":"1"}}},{"$ref":"2"}]"

// deserialize
const jsonList = JSON.parse(stringList);
const deserializedList: Vehicule[] = DeserializeArray(
    jsonList,
    () => Vehicule
);
RefClean();
RefCycleDetectionDisable();

// assert
const itIsTrue = deserializedList[0].other === deserializedList[1]; // true

--- edit ---
I didn't see your last post.
Could you provide an example with the class definition (with its dcerialize decorators), and the string you need to de-serialize (or step to reproduce your error) ?

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

I'll try to provide you with elements tomorrow morning.
Thanks.

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 6, 2019

As requested, attached is a zip file containing all the entities. Only those of interest right now are annotated with dcerialize decorators. I'm waiting to understand exactly how this will work with this first sample to generalize the annotations on all entities.

The code uses another third party module to track local changes, before sending them back to the server. You'll have to npm/yarn install 'trackable-entities' as well as dcerialize.

The json file is the payload (an array with 6 elements) received from the server to be deserialized with the following code:

RefCycleDetectionEnable();
const datis: Vehicule[] = DeserializeArray(
message.listDatiOfCompany,
() => Vehicule
);
RefClean();
RefCycleDetectionDisable();

The server Json serializer is configured tis way when returning the data:

return JsonConvert.SerializeObject(wholeResult, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore, ContractResolver = new CamelCasePropertyNamesContractResolver(), ReferenceLoopHandling = ReferenceLoopHandling.Serialize, PreserveReferencesHandling = PreserveReferencesHandling.Objects });

I have confirmed that if the array contains only one root object, there is no reference problem because everything is there in this single graph. With several objects, references may be spread anywhere in the objetc graphs.

Hope that helps.

Kind regards.

Models.zip
ArrayOfVehicule.jsonstring.txt

@PMDiademe

This comment has been minimized.

Copy link
Collaborator

commented Sep 6, 2019

I looked into your files, and found a reason for the error I was able to reproduce.
In the file VehiculeGpsBoxInfo.ts, you have to uncomment the following lines:

  //@autoserializeAs(() => Vehicule)
  //vehicule: Vehicule;

If you don't, the serializer will drop the object (I will call it O) define at the path [0].vehiculeGpsBoxInfo.vehiculeConfigurationNavigation.vehiculeGpsBoxInfoVehiculeConfigurationNavigation[1] of your file ArrayOfVehicule.jsonstring.txt. The definition of the object referenced by "13" is inside O. It appears at this path [0].vehiculeGpsBoxInfo.vehiculeConfigurationNavigation.vehiculeGpsBoxInfoVehiculeConfigurationNavigation[1].vehicule.fleetDetail[0].fleet.fleetDetailFleet[0] of your file ArrayOfVehicule.jsonstring.txt. So the serializer will not save this object in its dictionary of references.
Then, it will continue to parse the input and find a "$ref": "13" (in [0].fleetDetail[3] of your file ArrayOfVehicule.jsonstring.txt), it will look in its dictionary of references, and throw an exception since it doesn't have a definition associated to "13".

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 7, 2019

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 11, 2019

Yep that was it !
Works perfectly now.
Thanks for that.

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

Hi @PMDiademe,

I'm facing a new issue of type "Reference found before its definition" regarding deserializing an array.
This time $id11 is not seen before being used...

If I urderstood you correctly the last time, it was because the property and serialization attribute were missing in the object.

This time, it seems that this is OK though.

In the JON file, I have the following reference:

                     "gpsBoxTypeNavigation":{
                        "$id":"11",

The containing object is of type Vehicule and has the property attributed correctly:

  @autoserializeAs(() => GpsBoxType)
  gpsBoxTypeNavigation: GpsBoxType;

I do not understand why it is not seen by the deserializer.

Attached is the JSON file and the typescript classes.

Thanks for your help.

dcerialize_Reference found before its definition_$ref11.txt
models.zip

Hint: gpsBoxTypeNavigation on object Vehicule is now sent by the server, it was not before this morning and there was no problem.

@PierreMacherel

This comment has been minimized.

Copy link

commented Sep 28, 2019

(I am @PMDiademe)
I can't reproduce your bug:

DeserializeArray(blop, () => Vehicule);

runs fine (no exception raised).
For further help, it would spare me a lot of time if you could create a repository so that I could clone it, with detailed the steps to reproduce your problem. I only need the client side (you can hard code the server answer in a variable).

@omatrot

This comment has been minimized.

Copy link
Author

commented Sep 29, 2019

@PMDiademe I'll create this repo next Wednesday.
Thanks in advance.

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 2, 2019

Here is the Repo invitation link.
You should just be able to start debugging in VS Code
Thanks in advance.

@PierreMacherel

This comment has been minimized.

Copy link

commented Oct 2, 2019

OK, I found the issue. The files dcerialize_Reference found before its definition_$ref11.txt and the json variable in the file dcerialize-issue#1.ts in your repo are not the equal.

The Json object being deserialized may not be read from top to bottom. It is read in the order in which the decorators are read (Decorator Evaluation).

for instance:

class Bar {
    @autoserializeAs(() => Number)
    val: number;
    constructor(val: number) {
        this.val = val;
    }
}

class Foo {
    @autoserializeAs(() => Bar)
    a;
    @autoserializeAs(() => Bar)
    b;
}
const fooInstance = new Foo();
fooInstance.a = new Bar(1);
fooInstance.b = fooInstance.a;
RefCycleDetectionEnable();

// correctJson = { "$id": "1", "a": { "$id": "2", "val": 1 }, "b": { "$ref": "2" } }
const correctJson = Serialize(fooInstance, () => Foo);

// json send by the server
const incorectJSON =  { "$id": "1", "a": { "$ref": "2" }, "b": { "$id": "2", "val": 1 }};
// or const incorectJSON =  { "$id": "1", "b": { "$id": "2", "val": 1 }, "a": { "$ref": "2" }};
Deserialize(incorectJSON, () => Foo); // Error: Reference found before its definition

The order of property in the JSON sent by the server doesn't matter:
Deserialize expect a Foo. It loads the annotation gathered by the decorators in the order of the decorators i.e. "a", then "b" (top to bottom for members inside a class).
incorectJSON["a"] = { "$ref": "2" } is deserilized before { "$id": "2", "val": 1 } being read.

For order of serialization with inheritance, see the documentation

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 2, 2019

The order of property in the JSON sent by the server doesn't matter

I'm pretty sure the Serializer server side is not always consistent with the order of what is sent. It depends on the data. I had the same problem today just by having additional data in the database result later serialized.

It seems to me that the order is important.

Am I wrong?

How can I ensure that it works all the time whatever the data coming in?

@PierreMacherel

This comment has been minimized.

Copy link

commented Oct 2, 2019

You are correct, the order does matter. Dcerialize expects to read $id: x before reading $ref: x. The json sent by the server is a tree. Dcerialize will traverse this tree in its own way, but the property id before ref must be true. My point was that the order of the traversal of dcerialize is deterministic.

In your case, if you use newtonesoft, you have to ensure that it produces the good tree. Newtonsoft creates $id the first time it encounters the object, so you should have the same member ordering for typescript classes and c# classes.

A better solution would be to do 2 phases during the deserialization : collect the ids, then do the proper deserialization. It should be done carefully to avoid deserializing twice the same object (with potential side effect that could be caused). I could do it, but I have ongoing modifications to complete before.

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

For now I'll put attributes on properties server side to garantee the order of serialization.
Because the generator is the same for server and client, this should be consistent.
I'll get back to you with the outcome.

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

Still having the same error after ordering properties on server during serialization.
I don't know if this is the same issue but in this case, this is a circular reference

"vehiculeConfigurationNavigation":{
                                             "$id":"22",
                                             ...
                                             "vehiculeGpsBoxInfoVehiculeConfigurationNavigation":[
                                                {
                                                    "$id":"23",
                                                   ...
                                                   "vehiculeConfigurationNavigation":{
                                                      "$ref":"22"
                                                   }

the parser seems to be actually parsing object with $id 22 that is referenced later in a nested object.

What do you think?

The issue repo is up to date with this case.

Anyways, I you think your 2 phases deserialization will be ready in a couple of month, I think I can wait.

@PierreMacherel

This comment has been minimized.

Copy link

commented Oct 3, 2019

In your json object, the "$id":"22" is at the given path (I use the vscode plugin jsonPath to extract it):

[0].device.registration.company.fleet[0].fleetDetailFleet[1].vehicule.gpsBoxTypeNavigation.vehicule[2].vehiculeGpsBoxInfo.vehiculeConfigurationNavigation.$id

And dcerialize follow this path when the exception is raised:

[0].device.registration.company.fleet[0].fleetDetailFleet[1].vehicule.gpsBoxTypeNavigation.gpsBoxConfiguration[0].$ref22

Those two paths differ after

[0].device.registration.company.fleet[0].fleetDetailFleet[1].vehicule.gpsBoxTypeNavigation

gpsBoxTypeNavigation is of type GpsBoxType. In this class, gpsBoxConfiguration is before vehicule. That is why dcerialize raises an exception: it did not deserialized vehicule yet. I guess that in the server class GpsBoxType, vehicule is before gpsBoxConfiguration.

To get the second path, break when the exception is raised. Then look at the call stack, one set after 'deserializationContinuation'.
Step 1 DeserializeArrayInternal, look at the value of (offset + i) -> [0] (first part of the second path)
Step 2 DeserializeInternal, look at the value of keyName -> device (second part of the second path)
Step 3 DeserializeInternal, look at the value of keyName -> registration (third part of the second path)
...

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 4, 2019

Ok shoot! The attribute [JsonProperty(Order = 9)] was missing server side on the Vehicule property in class GPSBoxType which explains why is it not in the correct order when sent to the client.
Thanks for your help, I think I'm close to make it work.

@omatrot

This comment has been minimized.

Copy link
Author

commented Oct 10, 2019

This works perfectly now.
Thank you so much for the awesome work.

@omatrot omatrot closed this Oct 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.