MessagePack serialization time dominated by expression compilation.

[chrish42]

I am using MessagePack for a library that sends messages from a client to a server via a message queue. The objects I am sending are a Dictionnary<string, List>. The speed was really lacking, though.

Doing some profiling, I found out that the execution profile on the client is dominated (> 80%) by the PackerUnpackerExtensions' packObjectCore method call. Actually, what's dominating the profile is the call to compile() (which is done each time).

Is it possible to speed things up there? Any reason why the right methods are not called directly (instead of compiling code dynamically that does the call)?

[yfakariya]

It helps dynamic object serialization, but your case looks like unintended one. Auto generated serializer should not use PackObject in usual cases, so it seems to be a bug.
Would you show me what type(s) actually packed?

[chrish42]

The exact object being serialized is a Dictionary<string, List<object>>. The object being deserialized is an instance of:

class MsgToDeserialize {
    double[] Results;
    string error;
}

I am working on putting together some code that exhibits the problem for you.

[yfakariya]

The reason of Packer.PackObject is that you use Object type instead of MsgToDeserialized type for item type of the list. The serializer generator uses on-the-fly evaluation to handle actual type of Object items in a list.

I wonder why do you use List<object> instead of MsgToDeserialize , but I will wait for your additional code.

Thank you!

[chrish42]

The messages in each direction are not the same. This is a request/reply RPC system.

The request is a Dict mapping from strings to a list of either int, float, double or string. (So the variable type kinda has to be List<object>, since one key in the dictionary can point to a list of strings and another to a list of ints.)

The reply is either an error message, or a list of double. Does it make a bit more sense now?

[chrish42]

Here is a .cs file with a short program that exercises all the code paths I am using in the MessagePack C# library: https://gist.github.com/chrish42/e975c355f4118ebe2e70

Thanks for looking into this!

[yfakariya]

Thank you.

If you use only primitive type or its collection only, you can use List<MessagePackObject> instead of List<Object>. It improves performance about 1000 times faster in my local test because the serializer does not have to attempt complex object tree serialization.

You can find API document here:
http://cli.msgpack.org/doc/html/T_MsgPack_MessagePackObject.htm

If you need more sample code snippets for MessagePackObject handling, it is better to post to issue 13 rather than this post :-)

I will improve the wiki to notice this performance pit anyway. Thanks!

[chrish42]

The change to MessagePackObject fixes the performance problems, thanks. Happy to hear you will also be improving the documentation with respect to this. Also, for people who do need complex object tree serialization for List<object>, etc. a LRU cache of the results for compile() in the packToCore() function might help a lot in cases when the same small number of complex objects a serialized a lot. Just a suggestion...

Closing this issue, as the switch to MessagePackObject fixes my performance problems with MessagePack. Thanks again for your time. Much appreciated.