New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiled Lambda Expressions Hurt Performance on UWP #968

Open
Yortw opened this Issue Jul 22, 2016 · 15 comments

Comments

Projects
None yet
5 participants
@Yortw

Yortw commented Jul 22, 2016

Hi,

I've been struggling with (what I consider to be) unsatisfactory performance with Json.Net in UWP projects. After much research and testing, and some discussions with a developer on the .Net Native team, it would appear the problem is the use of compiled lambda expressions within Json.Net. This is most notable on slower ARM based devices, such as a Windows Phone, but can be seen on any .Net Native build.

On the desktop frameworks (and possibly others), the compiled expressions for setting properties/fields and creating type instances etc. do provide a significant performance improvement barring the cost of the overhead of creating/compiling the expressions in the first place.

However, in UWP/.Net Native projects it appears there are two issues that mean compiled lambdas actually run slower than the equivalent reflection code (i.e propertyInfo.SetValue vs. using the compiled setter expression).

  • A lot of internal caching of reflection types has been removed within the .Net Native/UWP runtimes, so reflection is more expensive there than on desktop. This can largely be avoiding by implementing your own caching of propertyinfo/constructorinfo and similar objects, which Json.Net already does.
  • Apparently, most of the stuff System.Linq.Expressions is not well supported, and in fact expression.Compile is basically a no-op (so I'm told by the Microsoft dev). It was suggested to me that the .Net Native team actually considered making these throw 'platformnotsupportedexceptions' but eventually decided slow was better than no support at all. The types in the Expressions namespace don't implement their own caching either, so using the compiled expressions actually results in worse performance than the normal reflection calls when you already have your own caching in place.

I realise it would be a non-trivial amount of work, and it probably needs testing to prove the performance benefits under UWP, but is there any chance at all there could be a UWP specific binary that avoided the compiled expressions for improved perf?

I know that is phrased as a question, but it isn't really suitable for Stackoverflow, and while I'm asking about a potential fix I'm really reporting the problem that an existing optimisation hurts on a different platform. My apologies if this is still the wrong place to post this, but it seemed like it met the criteria for 'bug report' to me.

Thanks.

@JamesNK

This comment has been minimized.

Show comment
Hide comment
@JamesNK

JamesNK Jul 22, 2016

Owner

Json.NET already supports switching between methods of doing reflection. The simplest solution would be a flag somewhere on the .NET Core runtime that could be tested to see whether dynamic compilation (IL emit) is available. If it is true then use IL emit or System.Linq.Expressions for reflection, if it isn't then use traditional MemberInfo reflection.

Question is, is there a flag (or some other technique) that could be tested to indicate that traditional MemberInfo reflection is best.

Owner

JamesNK commented Jul 22, 2016

Json.NET already supports switching between methods of doing reflection. The simplest solution would be a flag somewhere on the .NET Core runtime that could be tested to see whether dynamic compilation (IL emit) is available. If it is true then use IL emit or System.Linq.Expressions for reflection, if it isn't then use traditional MemberInfo reflection.

Question is, is there a flag (or some other technique) that could be tested to indicate that traditional MemberInfo reflection is best.

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Jul 23, 2016

Thanks for the excellent and fast response. I do not know the answer to your question myself, but I will ask my Microsoft contacts and see if they can suggest something.

Yortw commented Jul 23, 2016

Thanks for the excellent and fast response. I do not know the answer to your question myself, but I will ask my Microsoft contacts and see if they can suggest something.

@benaadams

This comment has been minimized.

Show comment
Hide comment
@benaadams

benaadams Aug 2, 2016

Contributor

Could Json.NET have a "build" step option where the compiled expressions are emitted to an assembly? Then that assembly would work with UWP's AOT. Just an idea...

Contributor

benaadams commented Aug 2, 2016

Could Json.NET have a "build" step option where the compiled expressions are emitted to an assembly? Then that assembly would work with UWP's AOT. Just an idea...

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Aug 2, 2016

Not sure how that would work. There is no "emit" method in the UWP framework, and the expressions aren't known until runtime when the code executes against types (either type objects or generic type arguments passed to methods). Maybe I'm missing something or don't understand the suggestion? Seems quite a complicated solution too.

Yortw commented Aug 2, 2016

Not sure how that would work. There is no "emit" method in the UWP framework, and the expressions aren't known until runtime when the code executes against types (either type objects or generic type arguments passed to methods). Maybe I'm missing something or don't understand the suggestion? Seems quite a complicated solution too.

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Aug 2, 2016

Re the referenced corefx issue... I am unsure if Xamarin.iOS has the same problem. Yes it uses AoT so the expressions probably aren't compiled because they just can't, but part of the UWP issue is that internal caching of some reflection objects was removed to reduce memory footprint. The Xamarin team may not have made the same decision, and I have not been able to test on that platform to compare relative performance. It's possible the Xamarin.iOS implementation is comparable to just using reflection anyway, depending on how it was implemented.

Yortw commented Aug 2, 2016

Re the referenced corefx issue... I am unsure if Xamarin.iOS has the same problem. Yes it uses AoT so the expressions probably aren't compiled because they just can't, but part of the UWP issue is that internal caching of some reflection objects was removed to reduce memory footprint. The Xamarin team may not have made the same decision, and I have not been able to test on that platform to compare relative performance. It's possible the Xamarin.iOS implementation is comparable to just using reflection anyway, depending on how it was implemented.

@benaadams

This comment has been minimized.

Show comment
Hide comment
@benaadams

benaadams Aug 2, 2016

Contributor

You'd need to know the types upfront at build time (though I'd assume you do as you'd be serialising and deserilising to them). Though would be more of a extension to Json.NET e.g. output strongly typed serialiser and deserialiser for types X, W, Z to assembly as part of a pre-build step. (Like bundling and minification is).

Contributor

benaadams commented Aug 2, 2016

You'd need to know the types upfront at build time (though I'd assume you do as you'd be serialising and deserilising to them). Though would be more of a extension to Json.NET e.g. output strongly typed serialiser and deserialiser for types X, W, Z to assembly as part of a pre-build step. (Like bundling and minification is).

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Aug 2, 2016

Could work, but as you say is probably an extension rather than an update to the library.

Re the existing possible solutions, since we are yet to hear back from MS about how to determine if expressions are actually compiled, could we consider one of the following;

  • A property on the SerializerSettings class that allows the consumer of the library to explicitly specify whether to use reflection/expressions/emit? Could default to an 'auto' value for current behaviour. The benefit here is that as other frameworks develop in the future, or the frameworks potentially change perf characteristics between versions, the consumer of the library could make the best choice for themselves even if Json.Net temporarily doesn't know the best option itself?
  • If we can't detect if expressions are compiled easily, would we settle for detecting the platform we're running on and making an assumption about the behaviour? i.e if (and I do not yet know how) we could detect we're running in a .Net Native/Xamarin.iOS/Unity environment/CLR we avoid expressions, and for all others we use the existing behaviour? Not as clean/accurate as knowing whether the expressions are compiled but if the logic is wrapped up in a function it would be easy to change to a better flag later if one becomes available.

Yortw commented Aug 2, 2016

Could work, but as you say is probably an extension rather than an update to the library.

Re the existing possible solutions, since we are yet to hear back from MS about how to determine if expressions are actually compiled, could we consider one of the following;

  • A property on the SerializerSettings class that allows the consumer of the library to explicitly specify whether to use reflection/expressions/emit? Could default to an 'auto' value for current behaviour. The benefit here is that as other frameworks develop in the future, or the frameworks potentially change perf characteristics between versions, the consumer of the library could make the best choice for themselves even if Json.Net temporarily doesn't know the best option itself?
  • If we can't detect if expressions are compiled easily, would we settle for detecting the platform we're running on and making an assumption about the behaviour? i.e if (and I do not yet know how) we could detect we're running in a .Net Native/Xamarin.iOS/Unity environment/CLR we avoid expressions, and for all others we use the existing behaviour? Not as clean/accurate as knowing whether the expressions are compiled but if the logic is wrapped up in a function it would be easy to change to a better flag later if one becomes available.
@JamesNK

This comment has been minimized.

Show comment
Hide comment
@JamesNK

JamesNK Aug 3, 2016

Owner

I'm not doing a build step. I simply want a way to determine whether a platform is AOT or not so the optimal way of doing reflection is used.

Owner

JamesNK commented Aug 3, 2016

I'm not doing a build step. I simply want a way to determine whether a platform is AOT or not so the optimal way of doing reflection is used.

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Aug 3, 2016

I have been cced on an email between .Net Native team members suggesting they are going to use your corefx issue to track adding a suitable flag for this, not sure if/when that might happen.

Are either of the two suggestions I made viable as alternate solutions?

Yortw commented Aug 3, 2016

I have been cced on an email between .Net Native team members suggesting they are going to use your corefx issue to track adding a suitable flag for this, not sure if/when that might happen.

Are either of the two suggestions I made viable as alternate solutions?

@JamesNK

This comment has been minimized.

Show comment
Hide comment
@JamesNK

JamesNK Aug 3, 2016

Owner

The problem with a UWP only binary is it only improves UWP. There would need to be one for Xamarin and Unity as well. And eventually compiling to native in .NET Core will be a thing and a platform specific binary won't help at all in that case. I want a solution that works everywhere.

Owner

JamesNK commented Aug 3, 2016

The problem with a UWP only binary is it only improves UWP. There would need to be one for Xamarin and Unity as well. And eventually compiling to native in .NET Core will be a thing and a platform specific binary won't help at all in that case. I want a solution that works everywhere.

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Aug 3, 2016

Hi James,

I think you misunderstood what I tried to say. I agree a UWP specific binary is uncool.

One suggestion is just to allow the client to explicitly state which mechanism to use. Not ideal, but does give the consumer ultimate control (with the default being the current 'automatic' behaviour).

The other suggestion is not a specific binary, but to attempt to detect the runtime environment (on the basis that there probably isn't a current flag that determines the behaviour of expression.compile). For example, I was hoping we could use Environment.Version or something similar to detect that we were running on a .Net Native build. Unfortunately that doesn't seem possible and I haven't yet found something that does work (am still looking).

Yortw commented Aug 3, 2016

Hi James,

I think you misunderstood what I tried to say. I agree a UWP specific binary is uncool.

One suggestion is just to allow the client to explicitly state which mechanism to use. Not ideal, but does give the consumer ultimate control (with the default being the current 'automatic' behaviour).

The other suggestion is not a specific binary, but to attempt to detect the runtime environment (on the basis that there probably isn't a current flag that determines the behaviour of expression.compile). For example, I was hoping we could use Environment.Version or something similar to detect that we were running on a .Net Native build. Unfortunately that doesn't seem possible and I haven't yet found something that does work (am still looking).

@JamesNK

This comment has been minimized.

Show comment
Hide comment
@JamesNK

JamesNK Aug 4, 2016

Owner

The second option is what I want. I feel there must be some way, somewhere to find out if the app is native.

Owner

JamesNK commented Aug 4, 2016

The second option is what I want. I feel there must be some way, somewhere to find out if the app is native.

@pieceofsummer

This comment has been minimized.

Show comment
Hide comment
@pieceofsummer

pieceofsummer Sep 11, 2016

Contributor

Haven't tested it, but try something like:

internal static class NativeUtils
{
    private class NativeTest<T> { }

    public static bool IsRunningOnNative()
    {
        return null == Type.GetType("Newtonsoft.Json.Utilities.NativeUtils+NativeTest`1", false);
    }
}

This is based on assumption that metadata for open generic types is not included in the .NET Native binaries unless explicitly specified.

Contributor

pieceofsummer commented Sep 11, 2016

Haven't tested it, but try something like:

internal static class NativeUtils
{
    private class NativeTest<T> { }

    public static bool IsRunningOnNative()
    {
        return null == Type.GetType("Newtonsoft.Json.Utilities.NativeUtils+NativeTest`1", false);
    }
}

This is based on assumption that metadata for open generic types is not included in the .NET Native binaries unless explicitly specified.

@hig-dev

This comment has been minimized.

Show comment
Hide comment
@hig-dev

hig-dev Jan 3, 2017

I develop a music player app for the Windows Store and used Newtonsoft.Json for the serialization of playlists. The difference in performance between my Release build which has .Net Native enabled and my Debug build is huge. So here are my results from testing with a large playlist with 16000 songs:

Serializing with Newtonsoft.Json:
DEBUG: 0,760678 seconds
RELEASE: 7,9427199 seconds

Deserializing with Newtonsoft.Json:
DEBUG: 1,7782815 seconds
RELEASE: 10,6551608 seconds

As a comparison I tested the same playlist with ServiceStack.Text Json Serializer:

Serializing with ServiceStack.Text Json:
DEBUG: 0,9823664 seconds
RELEASE: 9.0322984 seconds

Deserializing with ServiceStack.Text Json:
DEBUG: 0,9636399 seconds
RELEASE: 1.1934371 seconds

Conclusion: Please do something about this issue because as of today Newtonsoft.Json is not usable in my app.

hig-dev commented Jan 3, 2017

I develop a music player app for the Windows Store and used Newtonsoft.Json for the serialization of playlists. The difference in performance between my Release build which has .Net Native enabled and my Debug build is huge. So here are my results from testing with a large playlist with 16000 songs:

Serializing with Newtonsoft.Json:
DEBUG: 0,760678 seconds
RELEASE: 7,9427199 seconds

Deserializing with Newtonsoft.Json:
DEBUG: 1,7782815 seconds
RELEASE: 10,6551608 seconds

As a comparison I tested the same playlist with ServiceStack.Text Json Serializer:

Serializing with ServiceStack.Text Json:
DEBUG: 0,9823664 seconds
RELEASE: 9.0322984 seconds

Deserializing with ServiceStack.Text Json:
DEBUG: 0,9636399 seconds
RELEASE: 1.1934371 seconds

Conclusion: Please do something about this issue because as of today Newtonsoft.Json is not usable in my app.

@Yortw

This comment has been minimized.

Show comment
Hide comment
@Yortw

Yortw Jan 14, 2017

For those struggling with these perf issues, writing your own code that deserialises using a jsonreader instance is generally the way to get best perf anyway and eliminates this issue (http://www.newtonsoft.com/json/help/html/Performance.htm). If you have a lot of types and can't be bothered with that, a custom converter that uses cached reflection objects can be 25-50% faster (in my completely unscientific testing). A sample (not production code) here;

https://gist.github.com/Yortw/e568b363405577d55d1b68cad8279851

Note, this will be slower in debug builds, but is faster in release builds on everything I've tested. YMMV.

Yortw commented Jan 14, 2017

For those struggling with these perf issues, writing your own code that deserialises using a jsonreader instance is generally the way to get best perf anyway and eliminates this issue (http://www.newtonsoft.com/json/help/html/Performance.htm). If you have a lot of types and can't be bothered with that, a custom converter that uses cached reflection objects can be 25-50% faster (in my completely unscientific testing). A sample (not production code) here;

https://gist.github.com/Yortw/e568b363405577d55d1b68cad8279851

Note, this will be slower in debug builds, but is faster in release builds on everything I've tested. YMMV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment