Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C# Design Notes for Apr 14, 2015 #2134

Closed
MadsTorgersen opened this issue Apr 20, 2015 · 10 comments

Comments

Projects
None yet
9 participants
@MadsTorgersen
Copy link
Contributor

commented Apr 20, 2015

C# Design Meeting Notes for Apr 14

Bart De Smet visited from the Bing team to discuss their use of Expression Trees, and in particular the consequences of their current shortcomings.

The Expression Tree API today is not able to represent all language features, and the language supports lambda conversions even for a smaller subset than that.

Background

Certain Bing services, such as parts of Cortana, rely on shipping queries between different machines, both servers in the cloud and client devices.

In a typical scenario a lambda expression tree is produced in one place, ideally via the conversion from lambda expressions to expression trees that exists in C#. The expression tree is then serialized using a custom serialization format, and sent over the wire. At the receiving end it will often be stitched into a larger expression tree, which is then compiled with the Compile method and executed.

Along the way, several transformations are often made on the trees, for efficiency reasons etc. For instance, rather than invoke a lambda its body can often be inlined in the enclosing tree that the lambda gets stitched into.

The serialization format is able to carry very specific type information along with the code, but can also represent the code loosely. A looser coupling makes for code that is more resilient to "schema" differences between the nodes, and also allows for use of types and functions that aren't present where the lambda is concocted. However, it also makes it harder to stich things back up right on the other side.

Shortcomings in the Expression Tree API

The expression tree API was introduced with Linq and C# 3.0, and was extended to support the implementation infrastructure of dynamic along with C# 4.0. However, it has not been kept up to date with newer language features.

This is a list of ones that are confounding to the Bing team.

Dynamic

Even though the API was extended for the benefit of the dynamic feature, the feature itself ironically is not well represented in the API. This is not a trivial undertaking: in Expression Trees today, all invoked members are represented using reflection structures such as MemberInfos etc. Dynamic invocations would need a completely different representation, since they are by definition not bound by the time the expression tree is produced.

In the Bing scenario, dynamic would probably help a lot with representing the loosely coupled invocations that the serialization format is able to represent.

Alternatively, if the C# language added something along the "lightweight dynamic" lines that were discussed (but ultimately abandoned) for C# 6, the representation of that in Expression Trees would probably yield similar benefit with a fraction of the infrastructure.

Await

Representing await would probably be easy in the Expression Tree API. However, the implementation of it in Expression.Compile would be just as complex as it is in the C# and VB compilers today. So again, this is a significant investment, though one with much less public surface area complexity than dynamic.

Needless to say, distributed scenarios such as that of Bing services have a lot of use for await, and it can be pretty painful to get along without it.

Null conditional operators and string interpolation

These should also be added as Expression Tree nodes, but could probably be represented as reducible nodes. Reducible nodes are a mechanism for describing the semantics of a node in terms of reduction to another expression tree, so that the Compile method or other consumers don't have to know about it directly.

Higher level statements

While expression trees have support for statements, those tend to be pretty low level. Though there is a loop node there are no for or foreach nodes. If added, those again could be reducible nodes.

Shortcomings in the languages

C# and VB are even more limited in which lambdas can be converted to expression trees. While statements are part of the Expression Tree API, the languages will not convert them. Also, assignment operators will not be converted.

This is a remnant of the first wave of Linq, which focused on allowing lambdas for simple, declarative queries, that could be translated to SQL.

There has traditionally been an argument against adding more support to the languages based on the pressure this would put on existing Linq providers to support the new nodes that would start coming from Linq queries in C# and VB.

This may or may not ever have been a very good argument. However, with Roslyn analyzers now in the world, it pretty much evaporates. Any Linq provider that wants to limit at design time the kinds of lambdas allowed, can write a diagnostic analyzer to check this, and ship it with their library.

None of this support would require new syntax in the language, obviously. It is merely a matter of giving fewer errors when lambdas are converted to expression trees, and of mapping the additional features to the corresponding nodes in what is probably a straightforward fashion.

Conclusion

There is no remaining design reason we can think of why we shouldn't a) bring the Expression Tree API up to date with the current state of the languages, and b) extend the language support accordingly.

The main issue here is simply that it is likely to be a huge undertaking. We are not sure that the sum of the scenarios will warrant it, when you think about the opportunity cost for other language evolution. Bing is a very important user, but also quite probably an atypical one.

So in summary, as a language design team, we certainly support completing the picture here. But with respect to priorities, we'd need to see a broader call for these improvements before putting them ahead of other things.

@svick

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

Isn't dynamic already supported in the API through Expression.Dynamic()? There, that "completely different representation" is done using CallSiteBinder.

@AdamSpeight2008

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

Are we talking of Expression<T>?

@AdamSpeight2008

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

Are SyntaxNode language specific?

@RichiCoder1

This comment has been minimized.

Copy link

commented Apr 21, 2015

Having done something similiar to the Bing team, albiet on a much smaller scale, I would love to see this too.

@anpete

This comment has been minimized.

Copy link

commented Apr 21, 2015

"we'd need to see a broader call for these improvements"

Consider the EF team as also wanting these improvements! :-)

In particular, being able to compile expression trees containing await would help in some of our async query composition scenarios.

@billchi-ms

This comment has been minimized.

Copy link

commented Apr 24, 2015

Mads asked me to post my mail to him here ...

Not digging super deep into this, the intro to the ETs v2 spec/docs says that it is not a goal of ETs to reflect C#. There are multiple examples where they do not 1:1 model C# or VB; there just happens to be a lot of close modeling and semantics to C# because unless we had to be different, it was a good place to be :-).

I'm not dismissing Bart's requests, just saying that a motivator and expectation should NOT be that ETs must keep track of C# features. ETs v2 were conceived to model linguistic constructs driven by implementing three dynamic languages on .NET using the DLR and to model the bound expressions in the polymorphic inline caches for dynamic CallSites.

I'm mostly mentioning this so that there is awareness of the history that didn't seem reflected in the notes, and that there is explicit design choices to veer from the original v2 goals. Veering may be very reasonable, we clearly veered from v1 to v2 :-).

Cheers,
Bill

Spec is at (I'll edit with a github link when I find out who moved this where :-)): http://dlr.codeplex.com/wikipage?title=Docs%20and%20specs&referringTitle=Documentation

@MadsTorgersen MadsTorgersen changed the title C# Design Meeting Notes for Apr 14, 2015 C# Design Notes for Apr 14, 2015 Jul 12, 2015

@bartdesmet

This comment has been minimized.

Copy link

commented Nov 12, 2015

In the meantime, I've taken an early stab at providing a runtime library with C#-specific expression nodes that are implemented as reducible expressions. More information at https://github.com/bartdesmet/ExpressionFutures/tree/master/CSharpExpressions. I'll discuss this with the team in the weeks to come.

@anpete

This comment has been minimized.

Copy link

commented Nov 12, 2015

@bartdesmet Super cool!

@gafter gafter modified the milestone: C# 7 and VB 15 Nov 21, 2015

@gafter

This comment has been minimized.

Copy link
Member

commented Apr 25, 2016

Design notes have been archived at https://github.com/dotnet/roslyn/blob/future/docs/designNotes/2015-04-14%20C%23%20Design%20Meeting.md but discussion can continue here.

@gafter gafter closed this Apr 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.