[Proposal]: Expression tree evolution #4727
-
Expression tree evolution
SummaryThis proposal provides a way to introduce changes to the expression trees generated by the compiler, while providing a reasonable backwards compatibility story for legacy LINQ providers and other consumers. MotivationThe expression trees haven't had significant updates since introduction and lack many C# features that have now become commonplace. This creates a perception of staleness for customers and makes them doubt the future of expression trees. This is a non-exhaustive list of C# features that are currently either unsupported or are more restrictive in compiler-generated expression trees:
See ExpressionFutures for more details and implementation prototype. Related discussions: #158, #2029, #2545 In addition to the above there are proposed LINQ features that would also benefit from the proposed pattern: Detailed designExpressionTreeLangVersion
If a language feature that isn’t supported by the configured version is used in the expression the compiler outputs an error message that presents the user with information on how to change the language version and advises them to check their LINQ provider’s or expression-processing library documentation for the latest supported expression tree version. As long as all the used features are supported changing The default implementation of Expression tree versioning pattern is independent of ExpressionBuilderAttributenamespace System.Runtime.CompilerServices
{
[AttributeUsage(AttributeTargets.Assembly | AttributeTargets.Class | AttributeTargets.Method)]
public class ExpressionBuilderAttribute : Attribute
{
public ExpressionBuilderAttribute(Type type)
{
BuilderType = type;
}
public Type BuilderType { get; }
}
}When For example, the following builder limits the expression trees to just a subset of three expression node types: public static class LimitingExpressionBuilder
{
public static ConstantExpression Constant(object value, Type type) => Expression.Constant(value, type);
public static ParameterExpression Parameter(Type type, string name) => Expression.Parameter(type, name);
public static BinaryExpression Add(Expression left, Expression right) => Expression.Add(left, right);
}
Non-annotated methods are treated as annotated with
When an
All LINQ operator methods are annotated with When an expression tree is assigned to a variable before being passed to the method annotated with public class VersionedExpression<TBuilder, TExpression>
where TExpression : Expression
{
public VersionedExpression(TExpression expression)
{
Expression = expression;
}
public TExpression Expression { get; }
public static implicit operator TExpression(VersionedExpression<TBuilder, TExpression> expression)
=> expression.Expression;
}For example: VersionedExpression<LimitingExpressionBuilder, Expression<Func<int>>> expr = () => 1 + 2;
var _ = Evaluate(expr);
[ExpressionBuilder(typeof(LimitingExpressionBuilder))]
public abstract int Evaluate(Expression<Func<int>> expr);If When using an See Expression Types proposal for a description of how
DrawbacksDynamic expression handlingThe above doesn’t attempt to protect against expression trees which are constructed dynamically (e.g. via
Similarly, layers such as Automapper or OData - which produce or transform expression trees - are also responsible for ensuring correctness and LINQ provider compatibility. In practice, since different providers support different tree shapes, layers such as OData already need to be aware of the LINQ provider being targeted, and tailor their output trees accordingly. The user would likely need to opt-in to a particular version by configuring a setting or installing a NuGet package. Multiple LINQ providersWhen multiple LINQ providers are used in the same application, the lowest supported language version must be selected to take advantage of compile-time checking. This can be resolved by casting to Alternative expression tree representationsSince the expression trees produced need to remain backward compatible some expressions will need be represented by a new expression type, even if semantically they could be grouped with an existing expression type. This would make the code that handles the expression trees more verbose. And this prevents from adding better representations of existing features, e.g. interpolated strings. AlternativesLet LINQ providers expose the supported expression versionThe initial experience/discovery is not ideal – the user uses a modern construct and gets a compilation error; they need to figure out how to set up the expression version and which version is supported by their LINQ provider. And if a version is selected that isn’t supported by a LINQ provider the behavior is undefined (and could include data corruption). The providers could offer a way to use the latest supported expression version in a way that could be consumed by the user project. The provider NuGet package can include a .targets file that adds a value to However in the rare case when multiple LINQ providers are referenced and one of them hasn't been updated with Default to the latest expression version for LINQ methods.Another approach would be to default to the latest expression version supported by the compiler and introduce Any custom queryable operators defined outside BCL would not have the Pros:
Cons:
Runtime checks by the queryable operatorsTo reduce the performance penalty on legacy providers of the previous approach we could introduce a Pros:
Cons:
Mirror C# versioning
However adding "1" and "2" would mean unnecessarily restricting the available features and starting at "3" for the current feature-set would be confusing as the expression trees in C# 9.0 don't support all C# 3.0 features. Using an independent pattern provides more flexibility to the order the expression tree features would be introduced. Introduce a new expression node hierarchySwitching to a new representation that's aligned with the internal Roslyn tree will make implementing new C# features less costly and having a consistent representation will make the implementation of non-LINQ consumers easier. However LINQ providers would still need to have backward-compatibility, resulting in considerable maintenance penalty. And having two or more ways of representing an operation in the expression tree could lead to confusion on both the user and consumer sides. Use pre-processor directives
However this doesn't allow to use an expression builder that produces a different type of expression nodes. Unresolved questionsDesign meetings |
Beta Was this translation helpful? Give feedback.
Replies: 15 comments 67 replies
-
|
If you look at the current implementation of expression trees you can see that the syntax-tree is not just transformed into an equilant expression tree. Expression<Func<int, short, int>> f = (a, b) => a + b;the generated code is: ParameterExpression parameterExpression = Expression.Parameter(typeof(int), "a");
ParameterExpression parameterExpression2 = Expression.Parameter(typeof(short), "b");
BinaryExpression body = Expression.Add(parameterExpression, Expression.Convert(parameterExpression2, typeof(int)));
ParameterExpression[] array = new ParameterExpression[2];
array[0] = parameterExpression;
array[1] = parameterExpression2;
Expression.Lambda<Func<int, short, int>>(body, array)instead of just: ParameterExpression parameterExpression = Expression.Parameter(typeof(int), "a");
ParameterExpression parameterExpression2 = Expression.Parameter(typeof(short), "b");
BinaryExpression body = Expression.Add(parameterExpression, parameterExpression2);
ParameterExpression[] array = new ParameterExpression[2];
array[0] = parameterExpression;
array[1] = parameterExpression2;
Expression.Lambda<Func<int, int, int>>(body, array);Applying all these tiny little things makes it hard to maintain expression tree generation in the compiler. If we want to extend the expression-tree support I would like to ignore these things and create just a raw-transformation of the syntax-tree into an expression tree. Another approach would be something like "roslynQuoter" but for expression trees. |
Beta Was this translation helpful? Give feedback.
-
|
@Bosch-Eli-Black the problem generally isn't to just add new types to the Expression hierarchy; it's how to handle a new expression tree feature being used within a LINQ query (produced from regular C# syntax), which is possibly passed to an old LINQ provider that doesn't know about it. In any case, the problem isn't really how to implement support - AFAIK the compiler team has already discussed and arrived at some satisfactory approaches. It's more about prioritizing this vs. other work. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @roji 🙂 I was envisioning that this would solve the issue of passing LINQ queries with new features into old LINQ providers: If the programmer writes an expression that contains new features, it would be of type |
Beta Was this translation helpful? Give feedback.
-
|
That would mean that in order for a LINQ provider to support new C# features, it needs to be rewritten to dump Expression altogether and accept/work with CSharp10Expression instead (and that needs to be happen again for C# 11); that doesn't sound like a great solution. Once again, there have already been discussions around implementation - that's not where the problem is; it's about prioritizing the work. So while implementation discussions are interesting to have, they won't really advance this issue. |
Beta Was this translation helpful? Give feedback.
-
|
@roji Okay, understood, thanks 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
Just to add a counterexample ‒ I was making a reflection-based Expression serializer, and there were only a handful of special expression types like I don't think it should be a big deal if a library using |
Beta Was this translation helpful? Give feedback.
-
|
off-topic: icmyi https://github.com/reaqtive/reaqtor was open-sourced by @bartdesmet moments ago. If you like expression trees then there are 15 years of implemented knowledge right there. |
Beta Was this translation helpful? Give feedback.
-
|
Question: instead of being a compiler option (and therefore effectively a dialect option) couldn't this be specified purely by the SDK, in the same manner as AssemblyVersionAttribute is specified now? That way, Roslyn and the language spec would only specify what the translation is (i.e. |
Beta Was this translation helpful? Give feedback.
-
|
Compiler support allows to generate better error messages: Instead of: |
Beta Was this translation helpful? Give feedback.
-
|
The compiler could also generate a message such as |
Beta Was this translation helpful? Give feedback.
-
|
I still think this is the right way to go if the C# language does allow this. It would still require compiler changes, but I believe that a single translation "spec" would be far easier to maintain from the language side, rather than having to maintain multiple dialects for expression trees. To clarify, I'm suggesting:
|
Beta Was this translation helpful? Give feedback.
-
|
One thing, not directly related to the language itself, I would like to see is caching the expression trees when possible, like is done already for lambdas: Func<object, string> f = o => o.ToString(); // the Func<object, string> is reused for every subsequent call
Expression<Func<object, string>> e = o => o.ToString(); // the Expression instance is created every timeReusing the Expression instance (when possible) would make it better for things like conversion to SQL as the converted query could be cached based on the identity of the Expression instance. I am not sure how this might be a breaking change (expression trees are already immutable) but, if there would be an issue with that, I had the idea for the following syntax: Expression<Func<object, string>> e = static o => o.ToString();There's a possibility to reuse this |
Beta Was this translation helpful? Give feedback.
-
|
I am very interested to stay updated as much as possible to any smallest implementation/change/evolution. |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can see this proposal isn't currently Championed which means that it's not being worked on by the team at this time. I'd love to see something along these lines too, though. One specific detail about the proposal that I'd like to address: it seems to still be definitively tied to What if the [ExpressionBuilder(typeof(MyExpressionBuilder))]
public class MyExpression {
// whatever implementation here
}
public static class MyExpressionBuilder {
public static MyExpression Parameter(Type type, string name) => new MyExpression(something);
public static MyExpression Add(MyExpression left, MyExpression right) => new MyExpression(something);
}
MyExpression expr = (a, b) => a + b;By itself this isn't a fully formed proposal - for example, it doesn't explain how the types of In general, the idea would be that the BCL could annotate This also, to some extent, resolves the language evolution problem, in that a whole new type could be defined for the newer version, and if a newer language construct were used with the old |
Beta Was this translation helpful? Give feedback.
-
|
This branch of Roslyn does implement exactly what you're describing above: This also enables creating subsets of expression nodes by virtue of specifying custom builders, e.g. one for supported language constructs in Also, FWIW, this attribute approach could potentially be taken forward to expression-typed parameters (i.e. definition site), and potentially on lambda expressions as well (i.e. usage site). I haven't explored these routes with a hands-on implementation though. Finally, note that this branch is orthogonal to https://github.com/bartdesmet/roslyn/tree/ExpressionTrees, which is the branch where all C# language features up to C# 10.0 have been implemented for binding to new factory methods (without an intermediate builder type involved). The merge of both branches would pretty much align with the thinking outlined in this thread. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, the combination of these two approaches seems to cover all the usage scenarios that come to mind, including the backwards compatbility. The latter would assume that all the goodness from https://github.com/bartdesmet/roslyn/tree/ExpressionTrees should be moved to a different builder class, e.g. ExpressionEx : Expression. The BCL than would annotate the Queryable extension methods with the What can we do to help moving this forward? |
Beta Was this translation helpful? Give feedback.
-
|
@JeremyLikness are you still championing this from an EF perspective? |
Beta Was this translation helpful? Give feedback.
-
|
@AndriySvyryd is the key driver for this on our team. I'm available to support Andriy, @ajcvickers and the team however I can. |
Beta Was this translation helpful? Give feedback.
-
|
We are working with the language team. Most likely a new proposal will be submitted sometime this year. |
Beta Was this translation helpful? Give feedback.
-
|
@AndriySvyryd Are you aware of #2545 ? Seems possibly relevant 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps a better way to enable quoting of code (which is the problem expression trees solve) would be an approach like Scala macros (https://docs.scala-lang.org/overviews/macros/overview.html). Pretty much, you'd do where MyImplementation would depend on the compiler AST, but we already have source generators that work like that. |
Beta Was this translation helpful? Give feedback.
-
|
Expression trees cannot be interpreted at compile time without completely breaking the model. The mapping data doesn't exist at compile time. |
Beta Was this translation helpful? Give feedback.
-
|
+1 - the magic of expression trees is being able to translate between data < > code interchangeably at runtime; it's very different to standard "code gen". |
Beta Was this translation helpful? Give feedback.
-
|
@HaloFour Yes, I know. My suggested inspiration Scala macros would be a completely different way to achieve the same goal. |
Beta Was this translation helpful? Give feedback.
-
I disagree that moving the evaluation of expression trees or basing them on some language/compiler-specific macro model would help to achieve that goal. I don't think that would be possible without breaking the entire queryable provider ecosystem. |
Beta Was this translation helpful? Give feedback.
-
The SQL use-case is just one of the many examples, I am not referring to it. |
Beta Was this translation helpful? Give feedback.
-
|
No updates on this? |
Beta Was this translation helpful? Give feedback.
-
|
When there are updates, the team generally posts them. No need to poll :) |
Beta Was this translation helpful? Give feedback.
-
|
@CyrusNajmabadi Sure, my comment was just meant to encourage some traction here. This is a highly requested feature but there doesn't seem to be any activity on it from the maintainers. |
Beta Was this translation helpful? Give feedback.
-
|
+1 For this proposal. For instance, I'm tired of: What I'm trying to generate: |
Beta Was this translation helpful? Give feedback.
-
That's not the kind of issue this proposal is trying to solve. As far as I can tell, there's no plan to make breaking changes to the existing API. |
Beta Was this translation helpful? Give feedback.
-
|
Well, perhaps an issue titled "evolution" should actually attempt to evolve the API into something usable. I've been fighting with expression trees for four days, and hours today trying to get Expression.Call to work. It's a steaming pile of technically correct that needs to be evolved. shrug |
Beta Was this translation helpful? Give feedback.
-
|
This is the csharplang repository, so this issue attempts to evolve the language, not the API. If you seek to add a method that behaves the same as an overload of an existing method, the runtime repository is the proper one. |
Beta Was this translation helpful? Give feedback.
-
|
My understanding is that this work has so far not met the bar for investment. That could change, but I don't expect it to any time soon. I think we are back to the status quo from the last few years that System.Expressions is what it is. |
Beta Was this translation helpful? Give feedback.
-
|
Keeping the status quo would be a shame. Expression trees are one of the most amazing features of C#. The status quo would mean this feature would become less and less relevant over time, even though it has tons of useful applications. |
Beta Was this translation helpful? Give feedback.
-
|
Horrible news honestly. |
Beta Was this translation helpful? Give feedback.
-
|
If expression trees were being considered for inclusion to C# as a new feature, I'd say yours is a fair conclusion and that it needs more consideration. However, it seems to me that ship has sailed - they were introduced and it strikes me as a poor pitch for C# as a whole if new features, once introduced, may be indefinitely abandoned in any future iteration. Would you be able to speak to why this isn't approachable via an opt-in flag and a language version like nullable reference types are/were? I don't expect a .NET Framework 2.0 library to work flawlessly from .NET 7 because things have changed in both the language and compiler. It's why older versions of .NET are EOL'ed is so the framework and language can be iterated on, I thought. Why doesn't that hold here as well? Alternatively, why can't the slight changes to Roslyn be made to support Bart's implementation such that a far more expressive tree is available as an opt-in NuGet package iterated on independently from .NET itself? Same idea, but as the package evolves, it can maintain its own versioning dependency requirements. |
Beta Was this translation helpful? Give feedback.
-
|
Could we not start with the easy ones to get the ball moving. Most of the main pain points could be lowered to the current expression tree nodes: Conditional access expressionsTranslate Named and optional parametersGiven Translate Tuple literalsTranslate Tuple equality and inequalityTranslate This 4 things, but specially the first one, will cover most of the paint points with minimal braking changes, if any. Some other expressions like Dictionary initializes, Array initializers, Discard parameters, etc could also be implemented as a lowering-only solution. Things like Pattern Matching / switch expressions, while useful, will have limited use in practice because the Where can not give parameters to the Select. Also are very hard to implement from the Roslyn side and from the LINQ provider side. As for the statement/assignments nodes, they where discarded in C# 3 because the focus was on SQL-translatable expressions, and this hasn't changed much. Async/await inside queries looks like a corner case for me and dynamic could be useful in theory but then... just use SQL instead of LINQ. As a LINQ provider implementation, I would like to give expressive power to the consumer user, but also have a reduced set of nodes to translate. Trying to solve the general solution is stopping all progress here. |
Beta Was this translation helpful? Give feedback.
-
|
There's already a fork of Roslyn that supports expression tree updates through C# 11 without having to completely design that functionality from scratch.
No, the general concern is there's no path forward to updating what's already in place as it would have backwards compatibility implications and break things and that's an unacceptable cost. That said, per the separate discussion on this on #158 I've asked whether this is something that could simply be dropped into a "modern" namespace in a separate package so as to avoid any conflict with the original expression trees and while that seemed like a viable route, there hasn't been any further traction on the discussion. |
Beta Was this translation helpful? Give feedback.
-
|
@olmobrutall take a look at the comments above, this has all been amply discussed already. |
Beta Was this translation helpful? Give feedback.
-
|
@roji can you point to any comment in particular? I am aware of the huge work of @bartdesmet and I would love that it will get merged. As of how to version the expression tree factory... I don't really think is a big problem:
The solutions proposed here are way to ambitious, supporting in theory all the new C# constructs is not worth in most of the queries: async await, pattern matching.... won't be implemented in most LINQ providers, and if so... won't be used. And a compile-time way of determining library support. This is challenging because of the runtime nature of how LINQ operators are combined ( So my approach was to try a "Mimimum Agreement". What new expression nodes we need to support Tuples? or optional arguments? Nothing really... The Expression tree API is not a full fidelity one like roslyn nodes, and as @bernd5 pointer out, there are already mismatches between the code that you write and the expression Just get 80% of the value with 20% of the effort. |
Beta Was this translation helpful? Give feedback.
-
|
So far in discussions I think there are two camps of thought here: a) b) C# has grown a lot since the old times but a lot of the features are just convenience, for example I believe these approaches should be reconciled in some fashion, that can be, hopefully, implemented in individual steps that do not have to solve everything at once:
Even these steps would permit to use some types of expressions, but many would still be disallowed. This is what the following steps remedy:
I can put this into a concrete proposal if requested. |
Beta Was this translation helpful? Give feedback.
-
|
@olmobrutall The point is that the blockers here aren't technical. The blocker is that the .NET Directors have decided that we're not investing in this area. Nothing is going to happen unless that changes. |
Beta Was this translation helpful? Give feedback.
-
|
My current (un-fleshed-out) thoughts are around whether it's possible to externalise Expression Trees from the core framework, to allow them to evolve (via community effort / corporate sponsorship) as a stand alone set of extensions / plugins. A second point is that I'm sure I've seen that EF (one of the core internal users of ETs) are looking to move away from ETs to better support AOT scenarios. If this is true it has two ramifications 1) it pushes ETs further into the domain of obsolete technologies 2) IMHO mitigates the risk of breaking changes. If there is a lack of will from the .NET Execs to invest in this modernisation, then the question really is, are they willing to invest in letting the community take ownership and evolve it instead? |
Beta Was this translation helpful? Give feedback.
-
|
The idea sounds nice from a political point of view but... how can you make Roslyn plugable? They will need to know what method to call with which parameters for each new expression node. At this point how much flexibility the library has? |
Beta Was this translation helpful? Give feedback.
-
|
@HowardvanRooijen we have no current plans at the EF team to move away from LINQ ETs; the current plans are to translate Roslyn trees into expression trees and vice versa, not to do away with ETs (both because we still need to do handling of queries at runtime which requires ETs, and because we can't rewrite all of EF). I'm very skeptical about directions like externalizing this out of Roslyn. At the very least, the effort required in defining that extensibility point properly would likely be more work than just evolving ETs; but it also really feels like something that belongs in the compiler in any case, for many reasons. |
Beta Was this translation helpful? Give feedback.
-
|
That's good to know... I must say that was my assumption, of trying to work out how you'd achieve the objective of supporting AOT, based on reading this article (where you're quoted) https://devclass.com/2022/12/22/microsoft-plans-pre-compiled-queries-for-entity-framework-and-may-replace-old-and-crufty-net-sql-server-provider/ I'm sceptical too... but that's because we're now at the "clutching at straws" phase because we have no other viable options left. |
Beta Was this translation helpful? Give feedback.
-
|
For me, the most intriguing aspect of modernization of expression trees is inspired by the Reaqtor project, but rather blocked by lack of modern support (async being the critical piece). The idea of being able to generate and serialize expressions in order to deploy aspects of work across a distributed system (e.g. send the code to the data) opens up some really neat applications of C# that isn't trivially done today without using some intermediate language and all the complexity that comes with producing and maintaining that. Outside of just adding to this thread, is there a more suitable forum to which the case for this modernization work could be made to the .NET directors? |
Beta Was this translation helpful? Give feedback.
-
|
Currently I use do not use Entity Framework or expression trees with Linq. If we could allow data access without a fixed model aka dynamic database access the importance of expression trees might change... |
Beta Was this translation helpful? Give feedback.
-
|
If the database schema changes, wouldn't you have to refactor and application anyway, because all of your (dynamic) LINQ queries are now using an incorrect column name? In fact, that refactoring is much harder with dynamic queries: with a strongly-typed code model you can use regular C# refactoring to e.g. rename the property, and all your queries immediately get updated. I don't know how you'd do that in a dynamic setting. Note that EF (and other data access layers) also allow you to generate that statically-typed code model from the database. You can do that after your schema changes, and get full confidence that your queries are still correct etc. |
Beta Was this translation helpful? Give feedback.
-
This doesn't make sense. Couldn't you just write SQL queries by hand as strings if that's what you prefer? It doesn't get more "dynamic" than that, what's stopping you from doing that? The most essential benefit of using something like EF is precisely the statically-typed-ness that it provides and the guarantees that come with that. If you don't want that for some reason, EF is not even the right tool for you in the first place. |
Beta Was this translation helpful? Give feedback.
-
|
Writing queries by hand is quite a bad idea because there is no common SQL understood by every db vendor... The problem is that I have no fixed db model - my customers have very different data requirements. Currently I build my own immutable query-expression tree. The API usage looks like: var query = new SelectQuery()
.Select(defTabAlias.Dot("*"))
.From(defTab.Alias(defTabAlias))
.Join(defAssignTab.Alias(defTabAssignAlias), defTabAlias.Dot(defTabPk).Equal(defTabAssignAlias.Dot(defTabPk))
.And(defTabAlias.Dot("IS_VALID").Equal(true)))
.Where(defTabAssignAlias.Dot(classTabPk).In(classIds));In addition I have written a Parser which creates such a query model (some queries - and especially filters are stored in the customer db). |
Beta Was this translation helpful? Give feedback.
-
|
it would be great if I could write e.g.: from t in DynTable("SomeTable")
where t.DynField("Name") == "Foo"
select new { Foo = t.DynField("SomeField") }instead of var query = new SelectQuery()
.Select("SomeField".Alias("Foo"))
.From("SomeTable".Alias(t))
.Where("t".Dot("Name").Equal("Foo"));or even better: from t in "SomeTable"
where t.Name == "Foo"
select new { Foo = t.SomeField }Such code actually compiles - but the translation is not simple... |
Beta Was this translation helpful? Give feedback.
-
|
I'm still confused on your original point of not having to refactor your application if your query code is dynamic. For example,in your own immutable query-expression tree above, if the column name
LINQ queries also don't run the same way on every DB vendor, and some queries that run on one DB won't run on another. It's not possible/recommended to fully abstract over databases regardless of what data access tool you're using.
That's a very common scenario - typically you'd write (or better, generate) a static model per customer. |
Beta Was this translation helpful? Give feedback.
-
|
Static abstract interface members should probably be added to the list.
|
Beta Was this translation helpful? Give feedback.
and others
My understanding is that this work has so far not met the bar for investment. That could change, but I don't expect it to any time soon. I think we are back to the status quo from the last few years that System.Expressions is what it is.