Champion "Replace/original and code generation extensions" #107

Open
gafter opened this Issue Feb 14, 2017 · 276 comments

Comments

Projects
None yet
@gafter
Member

gafter commented Feb 14, 2017

  • Proposal added
  • Discussed in LDM
  • Decision in LDM
  • Finalized (done, rejected, inactive)
  • Spec'ed

See also

@gafter gafter changed the title from Replace/original and code generation extensions to Champion Replace/original and code generation extensions Feb 15, 2017

@gafter gafter changed the title from Champion Replace/original and code generation extensions to Champion "Replace/original and code generation extensions" Feb 21, 2017

@gafter gafter added this to the X.0 candidate milestone Feb 22, 2017

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Feb 28, 2017

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

MgSam commented Feb 28, 2017

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

@asdfgasdfsafgsdfa

This comment has been minimized.

Show comment
Hide comment
@asdfgasdfsafgsdfa

asdfgasdfsafgsdfa Mar 5, 2017

No news on this?
Would be really helpful for a ton of things...

No news on this?
Would be really helpful for a ton of things...

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Mar 24, 2017

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

MgSam commented Mar 24, 2017

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

@asdfgasdfsafgsdfa

This comment has been minimized.

Show comment
Hide comment
@asdfgasdfsafgsdfa

asdfgasdfsafgsdfa Mar 24, 2017

Sorry if this is a dumb question but what exactly does tooling mean here?
The "debugger UI" in Visual Studio?
What parts would need to get changed?
Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

Sorry if this is a dumb question but what exactly does tooling mean here?
The "debugger UI" in Visual Studio?
What parts would need to get changed?
Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

  1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
  2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
  3. Debugging. How do you debug through the code that is generated.
  4. Project-System. How do you present generated code in the project system?
  5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

  1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
  2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
  3. Debugging. How do you debug through the code that is generated.
  4. Project-System. How do you present generated code in the project system?
  5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

@eyalsk

This comment has been minimized.

Show comment
Hide comment
@eyalsk

eyalsk Mar 24, 2017

Contributor

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

Contributor

eyalsk commented Mar 24, 2017

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

I'm not sure i understand your question @eyalsk . Can you clarify?

I'm not sure i understand your question @eyalsk . Can you clarify?

@eyalsk

This comment has been minimized.

Show comment
Hide comment
@eyalsk

eyalsk Mar 24, 2017

Contributor

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

Contributor

eyalsk commented Mar 24, 2017

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

The owners of those respective products would have to make a decision on that. :)

The owners of those respective products would have to make a decision on that. :)

@eyalsk

This comment has been minimized.

Show comment
Hide comment
@eyalsk

eyalsk Mar 24, 2017

Contributor

@CyrusNajmabadi Got'cha, thanks. 😉

Contributor

eyalsk commented Mar 24, 2017

@CyrusNajmabadi Got'cha, thanks. 😉

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

MgSam commented Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Mar 24, 2017

What doesn't make sense is the arbitrarily different bar for different MS products:

  • T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
  • As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

MgSam commented Mar 24, 2017

What doesn't make sense is the arbitrarily different bar for different MS products:

  • T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
  • As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics

It's not internal politics. We are the C# team. We decide what we feel is best for the language. And that includes insuring that it has a great development experience out of the box.

And there's a good reason for that. We don't want to ship the language changes too early, only to find out later that we royally screwed things up once we try to then get proper IDE support for it. SourceGenerators are a massively complex undertaking. We could try to get something out without considering and attempting to prove it would work in the IDE. But then it's highly likely we would get it very wrong.

Even our preliminary efforts here showed us immediately how our early source-generator designs simply would not be suitable to providing a good IDE experience.

CyrusNajmabadi commented Mar 24, 2017

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics

It's not internal politics. We are the C# team. We decide what we feel is best for the language. And that includes insuring that it has a great development experience out of the box.

And there's a good reason for that. We don't want to ship the language changes too early, only to find out later that we royally screwed things up once we try to then get proper IDE support for it. SourceGenerators are a massively complex undertaking. We could try to get something out without considering and attempting to prove it would work in the IDE. But then it's highly likely we would get it very wrong.

Even our preliminary efforts here showed us immediately how our early source-generator designs simply would not be suitable to providing a good IDE experience.

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Mar 24, 2017

And that includes insuring that it has a great development experience out of the box.

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

MgSam commented Mar 24, 2017

And that includes insuring that it has a great development experience out of the box.

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

@eyalsk

This comment has been minimized.

Show comment
Hide comment
@eyalsk

eyalsk Mar 24, 2017

Contributor

@MgSam I agree with you on some of your points but I think that they probably don't want to make the same mistake twice and the experience I had with T4 from what I can remember was pretty awful to the point I removed the extension I wrote from the VS gallery. :)

Contributor

eyalsk commented Mar 24, 2017

@MgSam I agree with you on some of your points but I think that they probably don't want to make the same mistake twice and the experience I had with T4 from what I can remember was pretty awful to the point I removed the extension I wrote from the VS gallery. :)

@eyalsk

This comment has been minimized.

Show comment
Hide comment
@eyalsk

eyalsk Mar 24, 2017

Contributor

@MgSam

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Didn't they fix this for RTM?

Contributor

eyalsk commented Mar 24, 2017

@MgSam

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Didn't they fix this for RTM?

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects. We do not ship prototypes just because they're cool and they demo'ed well. A lot of real effort went into exploring this feature. Even just the language feature alone had a whole host of issues that we did not know how to resolve. And that was before the very first problem of: how will this even hook up into the IDE compilation flow model?

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects. We do not ship prototypes just because they're cool and they demo'ed well. A lot of real effort went into exploring this feature. Even just the language feature alone had a whole host of issues that we did not know how to resolve. And that was before the very first problem of: how will this even hook up into the IDE compilation flow model?

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Didn't they fix this for RTM?

No. But we're working on it. The core problem is that there have been extremely hard perf goals to meet, such that even loading a single extra dll is not allowed. That's problematic in the context of any sort of extensibility story. We made a hard decision to ship in that state, even though we would have preferred it be better.**

And that goes to my point. That was something we did not want to do, and it was only because the scope of the problem was so limited. i.e. you would get an error about tuples, but at least you could solve it.

That doesn't even begin to compare to the issues you'd get with source generators. Again, imagine it was like tuples, and you got lots of errors in your IDE while trying to work. Except now there wouldn't be anything you could do about that. Also consider that just typing things like "replace" and whatnot would simply throw of the IDE parser completely, leading to code that was nearly impossible to edit.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails. Just basic editing will fall to the floor because we won't understand teh code you're writing on a syntactic and semantic level. And, currently, our IDE is based around the idea that we will understand at least that much.

--

** We've won back enough performance across the board though that we feel lke we can get an exception here and that this feature can be enabled soon. If/when that happens, it will mean there was a short window of time when we had a not-great experience. That's not at all comparable to the types of timeframes we're talking about for something like SourceGenerators.

And again, it's not like we can "just ship source generators now and just make the IDE work later". We have to understand the needs of hte IDE so that we can properly design source generators in the first place. We know this first hand from other work we've done in the past, including work in analyzers. The original analyzer work did not think closely enough about the compiler/IDE interaction, and as such had absolutely unacceptable performance characteristics. By designing things together, we were able to come up with something that met our feature goals while also having acceptable performance.

CyrusNajmabadi commented Mar 24, 2017

Didn't they fix this for RTM?

No. But we're working on it. The core problem is that there have been extremely hard perf goals to meet, such that even loading a single extra dll is not allowed. That's problematic in the context of any sort of extensibility story. We made a hard decision to ship in that state, even though we would have preferred it be better.**

And that goes to my point. That was something we did not want to do, and it was only because the scope of the problem was so limited. i.e. you would get an error about tuples, but at least you could solve it.

That doesn't even begin to compare to the issues you'd get with source generators. Again, imagine it was like tuples, and you got lots of errors in your IDE while trying to work. Except now there wouldn't be anything you could do about that. Also consider that just typing things like "replace" and whatnot would simply throw of the IDE parser completely, leading to code that was nearly impossible to edit.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails. Just basic editing will fall to the floor because we won't understand teh code you're writing on a syntactic and semantic level. And, currently, our IDE is based around the idea that we will understand at least that much.

--

** We've won back enough performance across the board though that we feel lke we can get an exception here and that this feature can be enabled soon. If/when that happens, it will mean there was a short window of time when we had a not-great experience. That's not at all comparable to the types of timeframes we're talking about for something like SourceGenerators.

And again, it's not like we can "just ship source generators now and just make the IDE work later". We have to understand the needs of hte IDE so that we can properly design source generators in the first place. We know this first hand from other work we've done in the past, including work in analyzers. The original analyzer work did not think closely enough about the compiler/IDE interaction, and as such had absolutely unacceptable performance characteristics. By designing things together, we were able to come up with something that met our feature goals while also having acceptable performance.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

There's no irony here. Nor is this an accurate representation of what actually happens in TypeScript. In TypeScript features are shipped and the TS team themselves does update the LS side of things at the same time as they're doing the compiler work**. It is the case that they can release a compiler that people can use before an official VS build is made available, but that doesn't mean that they're shipping language features that they then might not have any clue how to the IDE work for.

That's what's different here. IT's not like we know what source-generators are at a language level, could ship that, and could then just "do the IDE work". We do not know what source-generators are precisely because the design of the language feature itself is so informed by the tooling that will consume it.

--

** I know. I was involved with it for several years, and i still work closely with them as they're an adopter of Roslyn and an early adopter of lots of the improvements we make. I wrote a ton of both the LS features for TS as well as the majority of the LS implementation in VS. I also wrote the extensibility points in Roslyn to allow TS to become a Roslyn language.

CyrusNajmabadi commented Mar 24, 2017

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

There's no irony here. Nor is this an accurate representation of what actually happens in TypeScript. In TypeScript features are shipped and the TS team themselves does update the LS side of things at the same time as they're doing the compiler work**. It is the case that they can release a compiler that people can use before an official VS build is made available, but that doesn't mean that they're shipping language features that they then might not have any clue how to the IDE work for.

That's what's different here. IT's not like we know what source-generators are at a language level, could ship that, and could then just "do the IDE work". We do not know what source-generators are precisely because the design of the language feature itself is so informed by the tooling that will consume it.

--

** I know. I was involved with it for several years, and i still work closely with them as they're an adopter of Roslyn and an early adopter of lots of the improvements we make. I wrote a ton of both the LS features for TS as well as the majority of the LS implementation in VS. I also wrote the extensibility points in Roslyn to allow TS to become a Roslyn language.

@MgSam

This comment has been minimized.

Show comment
Hide comment
@MgSam

MgSam Mar 24, 2017

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects.

Totally fair. So there's a lot more work to be done. I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board. Good software/API design doesn't require explicit buy in and support from every user before you proceed. If it did, no software would ever get shipped.

I use PostSharp- which accomplishes many of the same things as this feature. In the first versions had nearly zero IDE support and the support still sucks pretty bad. That doesn't mean it's still not super useful.

Like I said, do your best effort, put it behind a feature flag and let your partners implement IDE support themselves. I still haven't heard a reason why making an optional feature that's turned off by default is problematic. Again, you guys do this constantly in TypeScript.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails.

Except if I'm using R#, in which case it'll probably work perfectly. Again, my point that language design shouldn't be hampered by MS politics.

TS team themselves does update the LS side of things at the same time as they're doing the compiler work

This is completely off on a tangent, but non-nullable types were released in TypeScript 2.0 in September. The --strictNullChecks flag still isn't supported in VS 2017, over 6 months later.

When TypeScript first came out, you had Mads Kristensen adding capabilities on his own into VS (like compile on save) because no one else was doing it. The TypeScript-VS team coordination has always been dysfunctional. Yet TypeScript has been a huge success nonetheless. And you guys should be applauded for pushing the language forward without waiting for VS.


I don't mean to get into an extended back-and-forth over this, but it's super-frustrating to see useful features like this continually delayed indefinitely so that the team can instead turn to syntax sugar features that have little real-world impact on code quality or productivity (like the entire C# 7.0 release). And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

So, in my opinion, the C# team needs to stop gating progress on other MS teams and move ahead doing your best to design features and APIs. If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

MgSam commented Mar 24, 2017

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects.

Totally fair. So there's a lot more work to be done. I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board. Good software/API design doesn't require explicit buy in and support from every user before you proceed. If it did, no software would ever get shipped.

I use PostSharp- which accomplishes many of the same things as this feature. In the first versions had nearly zero IDE support and the support still sucks pretty bad. That doesn't mean it's still not super useful.

Like I said, do your best effort, put it behind a feature flag and let your partners implement IDE support themselves. I still haven't heard a reason why making an optional feature that's turned off by default is problematic. Again, you guys do this constantly in TypeScript.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails.

Except if I'm using R#, in which case it'll probably work perfectly. Again, my point that language design shouldn't be hampered by MS politics.

TS team themselves does update the LS side of things at the same time as they're doing the compiler work

This is completely off on a tangent, but non-nullable types were released in TypeScript 2.0 in September. The --strictNullChecks flag still isn't supported in VS 2017, over 6 months later.

When TypeScript first came out, you had Mads Kristensen adding capabilities on his own into VS (like compile on save) because no one else was doing it. The TypeScript-VS team coordination has always been dysfunctional. Yet TypeScript has been a huge success nonetheless. And you guys should be applauded for pushing the language forward without waiting for VS.


I don't mean to get into an extended back-and-forth over this, but it's super-frustrating to see useful features like this continually delayed indefinitely so that the team can instead turn to syntax sugar features that have little real-world impact on code quality or productivity (like the entire C# 7.0 release). And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

So, in my opinion, the C# team needs to stop gating progress on other MS teams and move ahead doing your best to design features and APIs. If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

I still haven't heard a reason why making an optional feature that's turned off by default is problematic.

I have given the reason:

Because we literally don't know how the feature should work. We don't know what the right design is for it. How do we even create it if we can't figure out these questions :)

I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board.

Good thing i didn't make that argument ;-)

The argument is that the C# team itself (i.e. us) are not willing to ship the feature when we can't deliver even a basic experience that is satisfactory. Nor are we going to just ship the language/compiler feature if we don't even have an idea for a design that we think could be sufficient.

and let your partners implement IDE support themselves.

And when the partners (i.e. the C# team) says to the C# team: "uh... there's no way we can get any sort of acceptable experience here with the design you shipped to customers..." then we're in a big heap of trouble. :)

And you guys should be applauded for pushing the language forward without waiting for VS.

Again, TS does not create language/compiler features that they don't know how they'll integrate into their LS. I'm not sure how much more plainly i can state that.

And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Please stop saying "*** team" :). The reason here is that the C# team itself does not know what the right design is for thsi feature. We don't know what it is and even as we experimented and prototyped many options here we ran into enormous issues (even at the language/compiler level) that we simply had no solution for. On top of that, the C# team itself does not want to ever ship language/compiler features that it doesn't know if it will be able to deliver a suitable developer experience around. That is a core goal of our language, and we believe it is very important. Right now we could come up with literally zero designs that we could feel confident that we'd be able to provide even a basic experience around, let alone a good experience. It would be enormously irresponsible of us to ship something like that at the language level, especially with the expectations people have had from us for the last decade+.

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

It's not other teams. We are the C# team. The C# team has not been able to come up with a satisfactory design for this feature, and also feels that there are enormous complexities and problems that would be extremely difficult to deal with for the developer experience.

--

I feel like you're thinking the situation is one of: "Oh man... we have this amazing feature. And we totally know how to do it. But those pesky devs over there are preventing it from going out! :( ".

That's not what is happening here. We simply do not have a design that we feel actually delivers the right feature to users (let alone an implementation to go along with that design). And we are never going to ship something that we can't even come up with a design for.

--

If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

Sorry, the rules of C# and the promises we've made over the years still apply. Backcompat is still a thing. We're not going to just throw this feature out there, then realize it is full of problems, and go "sorry, gotta break everyone, that's agile software development". We are the language and platform. We get that this would be hugely useful. But that's all the more reason to try to get something good that can last before shipping things out to people.

I still haven't heard a reason why making an optional feature that's turned off by default is problematic.

I have given the reason:

Because we literally don't know how the feature should work. We don't know what the right design is for it. How do we even create it if we can't figure out these questions :)

I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board.

Good thing i didn't make that argument ;-)

The argument is that the C# team itself (i.e. us) are not willing to ship the feature when we can't deliver even a basic experience that is satisfactory. Nor are we going to just ship the language/compiler feature if we don't even have an idea for a design that we think could be sufficient.

and let your partners implement IDE support themselves.

And when the partners (i.e. the C# team) says to the C# team: "uh... there's no way we can get any sort of acceptable experience here with the design you shipped to customers..." then we're in a big heap of trouble. :)

And you guys should be applauded for pushing the language forward without waiting for VS.

Again, TS does not create language/compiler features that they don't know how they'll integrate into their LS. I'm not sure how much more plainly i can state that.

And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Please stop saying "*** team" :). The reason here is that the C# team itself does not know what the right design is for thsi feature. We don't know what it is and even as we experimented and prototyped many options here we ran into enormous issues (even at the language/compiler level) that we simply had no solution for. On top of that, the C# team itself does not want to ever ship language/compiler features that it doesn't know if it will be able to deliver a suitable developer experience around. That is a core goal of our language, and we believe it is very important. Right now we could come up with literally zero designs that we could feel confident that we'd be able to provide even a basic experience around, let alone a good experience. It would be enormously irresponsible of us to ship something like that at the language level, especially with the expectations people have had from us for the last decade+.

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

It's not other teams. We are the C# team. The C# team has not been able to come up with a satisfactory design for this feature, and also feels that there are enormous complexities and problems that would be extremely difficult to deal with for the developer experience.

--

I feel like you're thinking the situation is one of: "Oh man... we have this amazing feature. And we totally know how to do it. But those pesky devs over there are preventing it from going out! :( ".

That's not what is happening here. We simply do not have a design that we feel actually delivers the right feature to users (let alone an implementation to go along with that design). And we are never going to ship something that we can't even come up with a design for.

--

If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

Sorry, the rules of C# and the promises we've made over the years still apply. Backcompat is still a thing. We're not going to just throw this feature out there, then realize it is full of problems, and go "sorry, gotta break everyone, that's agile software development". We are the language and platform. We get that this would be hugely useful. But that's all the more reason to try to get something good that can last before shipping things out to people.

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Mar 24, 2017

Except if I'm using R#, in which case it'll probably work perfectly.

No, it won't. I've tried future language features in older resharper products, and it doesn't work well.

Again, my point that language design shouldn't be hampered by MS politics.

The C# team sets its direction here. We have decided as a team that we did not think we had a good enough design for this feature. And we believe that we should not be shipping features that don't have good designs. That is not a political decision. That is a decision borne of our team principles.

--

Note: TypeScript has similar principles, and this is not something they are exploring either precisely because they do not have any sort of design that they think is good enough for this sort of scenario.

Except if I'm using R#, in which case it'll probably work perfectly.

No, it won't. I've tried future language features in older resharper products, and it doesn't work well.

Again, my point that language design shouldn't be hampered by MS politics.

The C# team sets its direction here. We have decided as a team that we did not think we had a good enough design for this feature. And we believe that we should not be shipping features that don't have good designs. That is not a political decision. That is a decision borne of our team principles.

--

Note: TypeScript has similar principles, and this is not something they are exploring either precisely because they do not have any sort of design that they think is good enough for this sort of scenario.

@mattwar

This comment has been minimized.

Show comment
Hide comment
@mattwar

mattwar Mar 24, 2017

@CyrusNajmabadi is correct but leaving you guys with the wrong impression. There are serious problems to solve before source generators can be a feature. However, these are not intractable problems. The
C# and related teams just needs to make it a priority for it to happen. There are many other important things that are consuming their time right now.

mattwar commented Mar 24, 2017

@CyrusNajmabadi is correct but leaving you guys with the wrong impression. There are serious problems to solve before source generators can be a feature. However, these are not intractable problems. The
C# and related teams just needs to make it a priority for it to happen. There are many other important things that are consuming their time right now.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

Let's say we do this, and then we receive a ton of feedback and requests from people saying: uh... i don't get how i can possibly use this. Intellisense just doesn't work. I can't refactor thigns properly because the refactorings don't know about this generated code, and thus semantic analysis doesn't properly understand my code relations. I can't debug my crashing program properly.

But we are in this situation today. We deliver a product that doesn't have this debuggable feature. Do you think that it is stopping our customers to use our product or they are so unhappy about this that they are looking for something else? No, they want to use our product because we are adding a lot more value than just generating some IL code.

If we respond to all of that with "it's not a problem. we provided a fast system to do compilation transforms," then we stand the chance of having a lot of upset users who are underserved by this system. And now we'll have the problem of trying to figure out how to potentially solve those issues after having shipped something that is overly flexible, and hard to constrain.

If an ORM product provides codegen at IL time and it is not debuggable, they are in this situation already. If this ORM product starts to use this Roslyn compilation plugin, code is not going to be debuggable and it is fine. Why these customers would go back to Roslyn and tell them that this is unacceptable?

Member

xoofx commented Jun 20, 2018

Let's say we do this, and then we receive a ton of feedback and requests from people saying: uh... i don't get how i can possibly use this. Intellisense just doesn't work. I can't refactor thigns properly because the refactorings don't know about this generated code, and thus semantic analysis doesn't properly understand my code relations. I can't debug my crashing program properly.

But we are in this situation today. We deliver a product that doesn't have this debuggable feature. Do you think that it is stopping our customers to use our product or they are so unhappy about this that they are looking for something else? No, they want to use our product because we are adding a lot more value than just generating some IL code.

If we respond to all of that with "it's not a problem. we provided a fast system to do compilation transforms," then we stand the chance of having a lot of upset users who are underserved by this system. And now we'll have the problem of trying to figure out how to potentially solve those issues after having shipped something that is overly flexible, and hard to constrain.

If an ORM product provides codegen at IL time and it is not debuggable, they are in this situation already. If this ORM product starts to use this Roslyn compilation plugin, code is not going to be debuggable and it is fine. Why these customers would go back to Roslyn and tell them that this is unacceptable?

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

This is because a SyntaxTree 100% roundtrips with source. So a token's positoin is literally defined as (at the same time) "where it showed up in the original source" or "where it would show up in source if you generated the source from the tree". There is no way for it to be otherwise (and even attempting anything like that would lead to massive breaking of roslyn invariants).

So you can't insert a SyntaxTree that would be built with something similar to using pragma #line directive that is supported by C#?

Member

xoofx commented Jun 20, 2018

This is because a SyntaxTree 100% roundtrips with source. So a token's positoin is literally defined as (at the same time) "where it showed up in the original source" or "where it would show up in source if you generated the source from the tree". There is no way for it to be otherwise (and even attempting anything like that would lead to massive breaking of roslyn invariants).

So you can't insert a SyntaxTree that would be built with something similar to using pragma #line directive that is supported by C#?

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Jun 20, 2018

But we are in this situation today

You are in this situation today. Roslyn is not :)

If an ORM product provides codegen at IL time and it is not debuggable, they are in this situation already. If this ORM product starts to use this Roslyn compilation plugin, code is not going to be debuggable and it is fine. Why these customers would go back to Roslyn and tell them that this is unacceptable?

This presumes that people will not expect and want this feature in roslyn to do more. There are a lot more scenarios than just the ones you've outlined. People do want to be able to generate more than just private/invisible impl details.

--

So, as has been said a bunch of times: Perhaps, if we scoped things down to a very limited set of things that could be allowed. It would enable some scenarios, while making it clear what wouldn't work.

This might be something tenable. However, it would likely be something very different than the current explorations into source-generation that are encompassed here.

But we are in this situation today

You are in this situation today. Roslyn is not :)

If an ORM product provides codegen at IL time and it is not debuggable, they are in this situation already. If this ORM product starts to use this Roslyn compilation plugin, code is not going to be debuggable and it is fine. Why these customers would go back to Roslyn and tell them that this is unacceptable?

This presumes that people will not expect and want this feature in roslyn to do more. There are a lot more scenarios than just the ones you've outlined. People do want to be able to generate more than just private/invisible impl details.

--

So, as has been said a bunch of times: Perhaps, if we scoped things down to a very limited set of things that could be allowed. It would enable some scenarios, while making it clear what wouldn't work.

This might be something tenable. However, it would likely be something very different than the current explorations into source-generation that are encompassed here.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

This presumes that people will not expect and want this feature in roslyn to do more. There are a lot more scenarios than just the ones you've outlined. People do want to be able to generate more than just private/invisible impl details.

I know (and I'm also looking for these scenarios, I have quite a few private projects doing this actually, with some ugly build tasks in the middle and a double compilation step 😉 ) but as I said, if we can at least get the bare minimum standardized of an "IL like undebuggable post-patching approach" (that is covering most of what popular libraries are doing today, used by thousands of projects and products), that's better than staying in a statu-quo. I would love to see the full feature source codegen described here to come to reality, but it sounds that it is unlikely to happen, because there will be always something less costly/complex/more prioritized to implement in Roslyn (I really don't blame you, I see the constraints and that's perfectly understandable)

Member

xoofx commented Jun 20, 2018

This presumes that people will not expect and want this feature in roslyn to do more. There are a lot more scenarios than just the ones you've outlined. People do want to be able to generate more than just private/invisible impl details.

I know (and I'm also looking for these scenarios, I have quite a few private projects doing this actually, with some ugly build tasks in the middle and a double compilation step 😉 ) but as I said, if we can at least get the bare minimum standardized of an "IL like undebuggable post-patching approach" (that is covering most of what popular libraries are doing today, used by thousands of projects and products), that's better than staying in a statu-quo. I would love to see the full feature source codegen described here to come to reality, but it sounds that it is unlikely to happen, because there will be always something less costly/complex/more prioritized to implement in Roslyn (I really don't blame you, I see the constraints and that's perfectly understandable)

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Jun 20, 2018

but as I said, if we can at least get the bare minimum standardized of an "IL like undebuggable post-patching approach" (that is covering most of what popular libraries are doing today, used by thousands of projects and products), that's better than staying in a statu-quo.

Yes. i can def see how that would help things out a lot. Honestly... i could even accept basically:

  1. hidden from intellisense.
  2. nothing to navigate to.
  3. no project system exposure.

The part that i think really does need a solution is debugging. I get that your customers have been ok with the limited debugging ability your exisitng solutions have. However, i go back to this being a min-bar:

  1. people be able to debug their code without any issues.
  2. ideally, people could actually debug the generated code. After all, if the generated code barfs, you need something to be able to work with to get through it. Otherwise, we'll just be inundated with messages from people going "the program is null-ref'ing... but i have no idea where/why".

Note: i'm allowing everything else to potentially be out of scope. but i see any potential solution needing to at least solve those points. note that this should already greatly lower the cost, and also address your concerns. But it would still need a thoughtful design to make sure the above was thought through and a suitable solution was found :)

Note: these are just my thoughts on the matter. Others may absolutely not agree. :)

but as I said, if we can at least get the bare minimum standardized of an "IL like undebuggable post-patching approach" (that is covering most of what popular libraries are doing today, used by thousands of projects and products), that's better than staying in a statu-quo.

Yes. i can def see how that would help things out a lot. Honestly... i could even accept basically:

  1. hidden from intellisense.
  2. nothing to navigate to.
  3. no project system exposure.

The part that i think really does need a solution is debugging. I get that your customers have been ok with the limited debugging ability your exisitng solutions have. However, i go back to this being a min-bar:

  1. people be able to debug their code without any issues.
  2. ideally, people could actually debug the generated code. After all, if the generated code barfs, you need something to be able to work with to get through it. Otherwise, we'll just be inundated with messages from people going "the program is null-ref'ing... but i have no idea where/why".

Note: i'm allowing everything else to potentially be out of scope. but i see any potential solution needing to at least solve those points. note that this should already greatly lower the cost, and also address your concerns. But it would still need a thoughtful design to make sure the above was thought through and a suitable solution was found :)

Note: these are just my thoughts on the matter. Others may absolutely not agree. :)

@m0sa

This comment has been minimized.

Show comment
Hide comment
@m0sa

m0sa Jun 20, 2018

@CyrusNajmabadi basically, what razor does, but more integrated, allow for 3rd parties to do the same thing more easily, as part of Roslyn itself?

m0sa commented Jun 20, 2018

@CyrusNajmabadi basically, what razor does, but more integrated, allow for 3rd parties to do the same thing more easily, as part of Roslyn itself?

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

Note: these are just my thoughts on the matter. Others may absolutely not agree. :)

Ok, so let's do this meeting then 😉

Member

xoofx commented Jun 20, 2018

Note: these are just my thoughts on the matter. Others may absolutely not agree. :)

Ok, so let's do this meeting then 😉

@m0sa

This comment has been minimized.

Show comment
Hide comment
@m0sa

m0sa Jun 20, 2018

We already have a tool that replaces CscTool with a custom exe, that uses the public Roslyn API to parse the command-line and then lets us modify it before emmitting it. It's been getting increasingly difficult to maintain it, though, especially perf-wise (I've found the shared compilation impossible to replace without totally reimplementing everything, most of all of that roslyn code is internal, with hard-coded things in the msbuild tasks), as well as getting it to run under dotnetcore (the commandline workspace is only available on desktop).

m0sa commented Jun 20, 2018

We already have a tool that replaces CscTool with a custom exe, that uses the public Roslyn API to parse the command-line and then lets us modify it before emmitting it. It's been getting increasingly difficult to maintain it, though, especially perf-wise (I've found the shared compilation impossible to replace without totally reimplementing everything, most of all of that roslyn code is internal, with hard-coded things in the msbuild tasks), as well as getting it to run under dotnetcore (the commandline workspace is only available on desktop).

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Jun 20, 2018

@m0sa

@CyrusNajmabadi basically, what razor does, but more integrated, allow for 3rd parties to do the same thing more easily, as part of the compilation itself?

Funny that you should mention Razor. :) That's a good example of:

  1. a massively complex system that has been hugely costly to support.
  2. something with very difficult issues that caused an enormously long tail of bugs.
  3. something that has morphed over time esp due to customer requests due to limitations.
  4. something that has occupied many man-decades (probably centuries at this point) of effort, with numerous devs having to maintain/fix/update it :)

Effectively, i would like to avoid the mistakes we've made in the past :) It's rarely the case that it will be as simple as people want. And, as time goes on, the problems continue to plague you and lead to customer unhappiness. Years later, you're still paying, and you are now trying to figure out how to do basic stuff customers are asking for like "make refactorings work gosh-darnit!" :)

--

This is not to say that we should not be working toward better code-gen answers. Just that these sorts o examples demonstrate why it's so important to not hand-wave away the parts of the problem space one does not currently care about. These things come back to bite you, and you commonly can end up in the "ugh... why did we make such bad decisions... i wish i could burn it all down and start over again" place :)

CyrusNajmabadi commented Jun 20, 2018

@m0sa

@CyrusNajmabadi basically, what razor does, but more integrated, allow for 3rd parties to do the same thing more easily, as part of the compilation itself?

Funny that you should mention Razor. :) That's a good example of:

  1. a massively complex system that has been hugely costly to support.
  2. something with very difficult issues that caused an enormously long tail of bugs.
  3. something that has morphed over time esp due to customer requests due to limitations.
  4. something that has occupied many man-decades (probably centuries at this point) of effort, with numerous devs having to maintain/fix/update it :)

Effectively, i would like to avoid the mistakes we've made in the past :) It's rarely the case that it will be as simple as people want. And, as time goes on, the problems continue to plague you and lead to customer unhappiness. Years later, you're still paying, and you are now trying to figure out how to do basic stuff customers are asking for like "make refactorings work gosh-darnit!" :)

--

This is not to say that we should not be working toward better code-gen answers. Just that these sorts o examples demonstrate why it's so important to not hand-wave away the parts of the problem space one does not currently care about. These things come back to bite you, and you commonly can end up in the "ugh... why did we make such bad decisions... i wish i could burn it all down and start over again" place :)

@CyrusNajmabadi

This comment has been minimized.

Show comment
Hide comment
@CyrusNajmabadi

CyrusNajmabadi Jun 20, 2018

It's been getting increasingly difficult to maintain it, though, especially perf-wise (I've found the shared compilation impossible to replace without totally reimplementing everything, most of all of that roslyn code is internal)

Sorry, would need more details to understand better. Based on what @xoofx has been saying, it sounds like primarily he just wants a system where he could do syntactic transformations (and, of course, analyze it using the public Roslyn API). What things do you feel like you need to reimplement? Thanks!

It's been getting increasingly difficult to maintain it, though, especially perf-wise (I've found the shared compilation impossible to replace without totally reimplementing everything, most of all of that roslyn code is internal)

Sorry, would need more details to understand better. Based on what @xoofx has been saying, it sounds like primarily he just wants a system where he could do syntactic transformations (and, of course, analyze it using the public Roslyn API). What things do you feel like you need to reimplement? Thanks!

@m0sa

This comment has been minimized.

Show comment
Hide comment
@m0sa

m0sa Jun 20, 2018

If one wants to do syntactic transformations, as a custom msbuild step outside of roslyn, you loose all advantages, and support for what dotnet build-servers offer, for starters:

  • each run has to load all roslyn dlls, except if you implement some custom build-server-like thing around it
  • roslyn has to re-parse all the source files, so you can get access to the Compilation and modify, analyze it

m0sa commented Jun 20, 2018

If one wants to do syntactic transformations, as a custom msbuild step outside of roslyn, you loose all advantages, and support for what dotnet build-servers offer, for starters:

  • each run has to load all roslyn dlls, except if you implement some custom build-server-like thing around it
  • roslyn has to re-parse all the source files, so you can get access to the Compilation and modify, analyze it
@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

If one wants to do syntactic transformations, as a custom msbuild step outside of roslyn, you loose all advantages, and support for what dotnet build-servers offer, for starters:

I concur and I'm not looking for that, as it would be falling exactly into the same problem we have with IL patching today

Member

xoofx commented Jun 20, 2018

If one wants to do syntactic transformations, as a custom msbuild step outside of roslyn, you loose all advantages, and support for what dotnet build-servers offer, for starters:

I concur and I'm not looking for that, as it would be falling exactly into the same problem we have with IL patching today

@PathogenDavid

This comment has been minimized.

Show comment
Hide comment
@PathogenDavid

PathogenDavid Jun 20, 2018

My apologies to mix up problems from dotnet/roslyn#19505 with here.

@xoofx You've said this a couple times, but I'd say they're very closely related. dotnet/roslyn#19505 even mentions this issue in the main issue text.

I feel like the discussion Roslyn issue is focusing more on improving the tools we have today (leaning towards IL mutation without double compilation) while the discussion here focuses more on replacing those tools with something better entirely (namely, source generation.) Both solutions can accomplish many of the same goals, it's just two different ways of looking at the issue.

PathogenDavid commented Jun 20, 2018

My apologies to mix up problems from dotnet/roslyn#19505 with here.

@xoofx You've said this a couple times, but I'd say they're very closely related. dotnet/roslyn#19505 even mentions this issue in the main issue text.

I feel like the discussion Roslyn issue is focusing more on improving the tools we have today (leaning towards IL mutation without double compilation) while the discussion here focuses more on replacing those tools with something better entirely (namely, source generation.) Both solutions can accomplish many of the same goals, it's just two different ways of looking at the issue.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

@xoofx You've said this a couple times, but I'd say they're very closely related. dotnet/roslyn#19505 even mentions this issue in the main issue text.

Indeed. What I'm trying to understand are the hidden/underlying requirements that people have in mind on this entire issue and why it is blocking also the less ambitious option (IL like patching as compilation rewriter, no debugger/Intellisense/IDE/project-system involved). From this discussion, I learned for example from @agocke that "If you want this to ship, the Compilation seen by the IDE needs to be the Compilation emitted. Period." which is something very concerning for me (actually, I would not like to generate hidden serializers or any IL like patching code we have today at IDE time) and from @CyrusNajmabadi that they expect the debug experience to be almost mandatory even for IL like patching scenarios. So we are making progress in understanding the "tensions" here and I would like to proceed further with everybody involved. 😉

Member

xoofx commented Jun 20, 2018

@xoofx You've said this a couple times, but I'd say they're very closely related. dotnet/roslyn#19505 even mentions this issue in the main issue text.

Indeed. What I'm trying to understand are the hidden/underlying requirements that people have in mind on this entire issue and why it is blocking also the less ambitious option (IL like patching as compilation rewriter, no debugger/Intellisense/IDE/project-system involved). From this discussion, I learned for example from @agocke that "If you want this to ship, the Compilation seen by the IDE needs to be the Compilation emitted. Period." which is something very concerning for me (actually, I would not like to generate hidden serializers or any IL like patching code we have today at IDE time) and from @CyrusNajmabadi that they expect the debug experience to be almost mandatory even for IL like patching scenarios. So we are making progress in understanding the "tensions" here and I would like to proceed further with everybody involved. 😉

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jun 20, 2018

Wow, what a thread. An issue dear to me, with my thinking evolving over two decades.

I'm sorry that I don't have time today to dissect this thread to have this comment better acknowledge previous input. Skimming I didn't see the approaches for the generative step laid out. In my experience, people have a strong vision on (generally) one approach and more easily see that vision and problems in the others.

There are several places to generate code. I'll put them in four buckets, almost all of which are used in everyday programming today:

  • Independent generation with source code as artifacts (T4)
  • Early generation with source code as artifacts (Razor)
  • During compilation ("source generation")
  • Late compilation/IL generation (PostSharp and [Design Safari])(dotnet/vblang#282)
  • Runtime generation

I think this thread has touched on all but the last, which I'll ignore.

Independent generation

Generate artifacts independent and before compile/build. Real artifacts exist on disk, generally in spaces not checked in to source control.

Example: T4

  • IDE issues vanish because there is no difference between generated and hand written code
  • It requires a gesture or a watcher - so programmers don't have to think about generation
  • Some issues would be more graceful if generated code was identified: accidental editing (now solvable) and avoiding refactoring generated code
  • People associate this approach with some negative history:
    • T4 had a horrible execution model (historic)
    • Previous versions (that I've seen, except for a spike I did) have been text base - text base is fundamentally broken for a mixed language world (solvable)
    • Insufficient delineation of code that runs for the generator and outputted code (solvable)
    • Most folks didn't discover pre-compiled templates
    • Outdated architectures - in today's world any generation is the capstone (small, critical piece) of an architecture using the many tools we didn't have in 2001
  • Programmer can learn from artifacts

Early generation

Generate artifacts just before compile publish. Real artifacts exist on disk. Unless something has changed, this is Razor. I think this has aspects of the adjacent approaches and is mostly valuable in tooling, and I don't understand how it works with IDE compilation, so I'm skipping it.

During compilation ("source generation")

Alter the compilation in flight. If new artifacts exist, they are ethereal and if they exist on disk it's an optimization.

Examples: None, we can't do this today

  • IDE issues
  • Generation is a natural part of the inner loop for the programmer, they don't have to think about generation
  • There is no real artifact for the programmer to accidentally change
  • Either requires writing against the compilation, or uses a generation language we have not yet created (creating templates requires skill)
  • Programmer cannot see or learn from artifacts and if IDE problems are not handled well, the programmer may have difficulty understanding the execution of their program

Late compilation/IL generation

Do not generate artifacts, but change what IL is emitted.

Examples: PostSharp usage, optimization, async/await, @AnthonyDGreen's vblang #282 Design Safari

  • Some IDE issues are finessed because the programmer "doesn't need to know about" the generated code
    • Debugging arguably presents problems with the "doesn't need to know about" philosophy
  • Generation is a natural part of the programmer's process, they don't have to think about generation
  • There is no real artifact for the programmer to accidentally change
  • Requires writing to the compilation or IL (creating templates requires skill)
  • Programmer cannot see or learn from artifacts and the programmer may have difficulty understanding the execution of their program

Personal opinion:

  • We have many approaches to reducing boilerplate code today. Generic inheritance for implementations and functional programming greatly reduce the need for generated code
  • After all the above refactorings, there remains some boiler plate code that follows relatively narrow well defined patterns that can be solved with IL generation/AOP. async/await and maybe INotifyPropertyChanged are poster children (*)
  • After all above refactoring and above hidden pattern application (AOP), there remains some code that is boilerplate. Much of this code tends to have a heavy reliance on metadata with relatively simple patterns (EF).

When I look at the problem this way, I simply can't see justification for the high level of effort in source generation in Roslyn.

I've tried. I asked for source generation as the core feature of Roslyn when I first heard of the project (thankfully they ignored me). I absolutely love the vision. But, the execution would have been tricky in the old world where the compiler wasn't a core feature of the IDE. In the new world, even if the problems with the Visual Studio IDE were solved, what about OSS? How could a generative approach be considered real if it didn't work in every editor on every platform?

I would like instead to see good support for independent generation:

  • "replace"/similar language constructs to allow replacing generated code
  • better identification of generated status within code
  • IDE showing which code is generated (watermark/etc & Solution/File explorer)
  • IDE understanding generated code in key actions (refactoring, replace, optionally search)
  • watcher becoming an integral part of inner loop or a trigger for out of proc action on compilation
  • default generator
  • leadership/guidelines on metadata patterns
  • spike on a new cross-language generation approach/language

I also feel a deep passion about generative code being a route for programmer learning at many levels.

There are a number of places in .NET that would benefit, including EF and other generation along with scaffolding and project templates replaced with reentrant generation.

I believe the amazing outcome of this work would be community driven architectures. Where tools have allowed this, cool stuff happened (FoxPro, CodeSmith, etc). I want us to do that again.

(*) For AOP, I like a variation of @AnthonyDGreen's vblang #282 Design Safari that I spiked. The variation of Anthony's approach is to demand that all code exist to allow stepping. The attribute methods overload on interfaces, and some other tweaks. It allows programmers to use AOP with almost no new concepts and makes the debugging experience much simpler to create.

Apologies for the length of this comment

KathleenDollard commented Jun 20, 2018

Wow, what a thread. An issue dear to me, with my thinking evolving over two decades.

I'm sorry that I don't have time today to dissect this thread to have this comment better acknowledge previous input. Skimming I didn't see the approaches for the generative step laid out. In my experience, people have a strong vision on (generally) one approach and more easily see that vision and problems in the others.

There are several places to generate code. I'll put them in four buckets, almost all of which are used in everyday programming today:

  • Independent generation with source code as artifacts (T4)
  • Early generation with source code as artifacts (Razor)
  • During compilation ("source generation")
  • Late compilation/IL generation (PostSharp and [Design Safari])(dotnet/vblang#282)
  • Runtime generation

I think this thread has touched on all but the last, which I'll ignore.

Independent generation

Generate artifacts independent and before compile/build. Real artifacts exist on disk, generally in spaces not checked in to source control.

Example: T4

  • IDE issues vanish because there is no difference between generated and hand written code
  • It requires a gesture or a watcher - so programmers don't have to think about generation
  • Some issues would be more graceful if generated code was identified: accidental editing (now solvable) and avoiding refactoring generated code
  • People associate this approach with some negative history:
    • T4 had a horrible execution model (historic)
    • Previous versions (that I've seen, except for a spike I did) have been text base - text base is fundamentally broken for a mixed language world (solvable)
    • Insufficient delineation of code that runs for the generator and outputted code (solvable)
    • Most folks didn't discover pre-compiled templates
    • Outdated architectures - in today's world any generation is the capstone (small, critical piece) of an architecture using the many tools we didn't have in 2001
  • Programmer can learn from artifacts

Early generation

Generate artifacts just before compile publish. Real artifacts exist on disk. Unless something has changed, this is Razor. I think this has aspects of the adjacent approaches and is mostly valuable in tooling, and I don't understand how it works with IDE compilation, so I'm skipping it.

During compilation ("source generation")

Alter the compilation in flight. If new artifacts exist, they are ethereal and if they exist on disk it's an optimization.

Examples: None, we can't do this today

  • IDE issues
  • Generation is a natural part of the inner loop for the programmer, they don't have to think about generation
  • There is no real artifact for the programmer to accidentally change
  • Either requires writing against the compilation, or uses a generation language we have not yet created (creating templates requires skill)
  • Programmer cannot see or learn from artifacts and if IDE problems are not handled well, the programmer may have difficulty understanding the execution of their program

Late compilation/IL generation

Do not generate artifacts, but change what IL is emitted.

Examples: PostSharp usage, optimization, async/await, @AnthonyDGreen's vblang #282 Design Safari

  • Some IDE issues are finessed because the programmer "doesn't need to know about" the generated code
    • Debugging arguably presents problems with the "doesn't need to know about" philosophy
  • Generation is a natural part of the programmer's process, they don't have to think about generation
  • There is no real artifact for the programmer to accidentally change
  • Requires writing to the compilation or IL (creating templates requires skill)
  • Programmer cannot see or learn from artifacts and the programmer may have difficulty understanding the execution of their program

Personal opinion:

  • We have many approaches to reducing boilerplate code today. Generic inheritance for implementations and functional programming greatly reduce the need for generated code
  • After all the above refactorings, there remains some boiler plate code that follows relatively narrow well defined patterns that can be solved with IL generation/AOP. async/await and maybe INotifyPropertyChanged are poster children (*)
  • After all above refactoring and above hidden pattern application (AOP), there remains some code that is boilerplate. Much of this code tends to have a heavy reliance on metadata with relatively simple patterns (EF).

When I look at the problem this way, I simply can't see justification for the high level of effort in source generation in Roslyn.

I've tried. I asked for source generation as the core feature of Roslyn when I first heard of the project (thankfully they ignored me). I absolutely love the vision. But, the execution would have been tricky in the old world where the compiler wasn't a core feature of the IDE. In the new world, even if the problems with the Visual Studio IDE were solved, what about OSS? How could a generative approach be considered real if it didn't work in every editor on every platform?

I would like instead to see good support for independent generation:

  • "replace"/similar language constructs to allow replacing generated code
  • better identification of generated status within code
  • IDE showing which code is generated (watermark/etc & Solution/File explorer)
  • IDE understanding generated code in key actions (refactoring, replace, optionally search)
  • watcher becoming an integral part of inner loop or a trigger for out of proc action on compilation
  • default generator
  • leadership/guidelines on metadata patterns
  • spike on a new cross-language generation approach/language

I also feel a deep passion about generative code being a route for programmer learning at many levels.

There are a number of places in .NET that would benefit, including EF and other generation along with scaffolding and project templates replaced with reentrant generation.

I believe the amazing outcome of this work would be community driven architectures. Where tools have allowed this, cool stuff happened (FoxPro, CodeSmith, etc). I want us to do that again.

(*) For AOP, I like a variation of @AnthonyDGreen's vblang #282 Design Safari that I spiked. The variation of Anthony's approach is to demand that all code exist to allow stepping. The attribute methods overload on interfaces, and some other tweaks. It allows programmers to use AOP with almost no new concepts and makes the debugging experience much simpler to create.

Apologies for the length of this comment

@orthoxerox

This comment has been minimized.

Show comment
Hide comment
@orthoxerox

orthoxerox Jun 20, 2018

@KathleenDollard thanks a lot for the extensive summary!

I have just one pressing question: what is "spike"? I suppose it's some Microspeak, but Raymond Chen hasn't covered it yet in his blog.

@KathleenDollard thanks a lot for the extensive summary!

I have just one pressing question: what is "spike"? I suppose it's some Microspeak, but Raymond Chen hasn't covered it yet in his blog.

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jun 20, 2018

Spike (software development), a small task done to reduce uncertainty about a larger task
(wikipedia.org, which is an attribution, not a judgement. Thanks for asking)

Spike (software development), a small task done to reduce uncertainty about a larger task
(wikipedia.org, which is an attribution, not a judgement. Thanks for asking)

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

@KathleenDollard thank you for this good summary 👍

I'm perfectly fine with T4 for what it can and cannot do. I'm using it sometimes in my projects and I'm not looking for improvement there. It is doing its job.

I would like instead to see good support for independent generation:

I would like to understand:

  • What kind of problem this is solving exactly?
  • Who has been requesting this feature? I'm not able to find a post with an actual library relying on this for its end-users (with a wide impact indirectly on end-users).
  • Which OSS projects today is actually looking for a better support for this?

Most of the concrete use cases we have been discussing in the posts are coming from Serializers/RPC (@ReubenBond, myself) , ORM (@mgravell) , AOP (a few folks around, including myself)....etc. that are done today mostly through post compilation steps via IL patching (or worse, via reflection emit at runtime) but could be done a lot more efficiently through a Roslyn integration (no debug, no ide, no intellisense, no project-system). I have also usecases related to UI (similar to WPF) which requires "During compilation", but let's say that I could live without that.

I'm really failing to see how independent generation respond to our requirements which are roughly:

  • We want to be able to generate code based on existing code (that can even modify existing code)
  • We don't want users to access the generated code from their project (they have ILSpy for that), or to store this code in the project/sourcecontrol, nor to be editable. They are mainly for post scenarios. Worse, if we are making an evolution to the underlying generated code (e.g removing a method), by just upgrading the package, we don't want users to have an un-compilable project.
  • We are not looking for improving the debugging experience (there are tools to step into post generated IL code actually)
  • We want to have a very fast compilation experience that would not require to re-parse the files, or re-read all the assemblies, multiple times (because today, we don't have any standard, so each solution has to queue an IL patcher to the build process via custom msbuild target files)
  • We want to have a standard and integrated way for libraries to provide such a compilation step through a NuGet package (ala Diagnostic analyzers)
  • We want to work at a higher level than IL level
  • We want a transparent and fluid experience for the users of our library/nuget compiler plugin

We know that this could be done easily, just with a few weeks of work, inside Roslyn, and would help many of our projects.

Member

xoofx commented Jun 20, 2018

@KathleenDollard thank you for this good summary 👍

I'm perfectly fine with T4 for what it can and cannot do. I'm using it sometimes in my projects and I'm not looking for improvement there. It is doing its job.

I would like instead to see good support for independent generation:

I would like to understand:

  • What kind of problem this is solving exactly?
  • Who has been requesting this feature? I'm not able to find a post with an actual library relying on this for its end-users (with a wide impact indirectly on end-users).
  • Which OSS projects today is actually looking for a better support for this?

Most of the concrete use cases we have been discussing in the posts are coming from Serializers/RPC (@ReubenBond, myself) , ORM (@mgravell) , AOP (a few folks around, including myself)....etc. that are done today mostly through post compilation steps via IL patching (or worse, via reflection emit at runtime) but could be done a lot more efficiently through a Roslyn integration (no debug, no ide, no intellisense, no project-system). I have also usecases related to UI (similar to WPF) which requires "During compilation", but let's say that I could live without that.

I'm really failing to see how independent generation respond to our requirements which are roughly:

  • We want to be able to generate code based on existing code (that can even modify existing code)
  • We don't want users to access the generated code from their project (they have ILSpy for that), or to store this code in the project/sourcecontrol, nor to be editable. They are mainly for post scenarios. Worse, if we are making an evolution to the underlying generated code (e.g removing a method), by just upgrading the package, we don't want users to have an un-compilable project.
  • We are not looking for improving the debugging experience (there are tools to step into post generated IL code actually)
  • We want to have a very fast compilation experience that would not require to re-parse the files, or re-read all the assemblies, multiple times (because today, we don't have any standard, so each solution has to queue an IL patcher to the build process via custom msbuild target files)
  • We want to have a standard and integrated way for libraries to provide such a compilation step through a NuGet package (ala Diagnostic analyzers)
  • We want to work at a higher level than IL level
  • We want a transparent and fluid experience for the users of our library/nuget compiler plugin

We know that this could be done easily, just with a few weeks of work, inside Roslyn, and would help many of our projects.

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jun 20, 2018

@xoofx End to start:

We know that this could be done easily, just with a few weeks of work, inside Roslyn, and would help many of our projects.

I can't comment on how much work, but I have to say that I don't think doing this halfway is a good idea. It does help a small group of folks vocal here, but I think putting a solution in Roslyn should have broader benefit.

We want to be able to generate code based on existing code (that can even modify existing code)

I think available metadata should include the compilation, so agree on that. I'm not yet sold on modifying existing code.

We don't want users to access the generated code from their project (they have ILSpy for that), or to store this code in the project/sourcecontrol, nor to be editable. They are mainly for post scenarios. Worse, if we are making an evolution to the underlying generated code (e.g removing a method), by just upgrading the package, we don't want users to have an un-compilable project.
We are not looking for improving the debugging experience (there are tools to step into post generated IL code actually)

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

We want to have a very fast compilation experience that would not require to re-parse the files, or re-read all the assemblies, multiple times (because today, we don't have any standard, so each solution has to queue an IL patcher to the build process via custom msbuild target files)

Pre-compilation independent generation out of proc runs at the speed of the user and does not affect compilation speed (except having a slightly larger project to parse). I understand this as a comment on today's IL generation scenario.

We want to have a standard and integrated way for libraries to provide such a compilation step through a NuGet package (ala Diagnostic analyzers)
We want to work at a higher level than IL level
We want a transparent and fluid experience for the users of our library/nuget compiler plugin

Did you look at @AnthonyDGreen's Design Safari approach to an AOP like experience? It would allow you to add attributes (via a NuGet package) that added code at emit time in a very controlled manner. The guard rails I have in mind for that might be too tight. But if you have a scenario you think would not work I'd be happy to noodle on it.

Most of the concrete use cases we have been discussing in the posts are coming from Serializers/RPC (@ReubenBond, myself) , ORM (@mgravell) , AOP (a few folks around, including myself)...

I don't see an issue, but I assume you fleshed out your thoughts on inconsistencies with an independent approach in the bullet points I already answered.

I'd really like you to take a look at the vblang issue. It's AOP via C# code. If it only handles INotifyPropertyChanged, it's viability is quite a bit different than if it also handles your pain points.

If IL is painful for authors, and source-generation is seen as a bigger issue by the team, can we finesse by having attributes actually tell us what they will do and inserting that code.

@xoofx End to start:

We know that this could be done easily, just with a few weeks of work, inside Roslyn, and would help many of our projects.

I can't comment on how much work, but I have to say that I don't think doing this halfway is a good idea. It does help a small group of folks vocal here, but I think putting a solution in Roslyn should have broader benefit.

We want to be able to generate code based on existing code (that can even modify existing code)

I think available metadata should include the compilation, so agree on that. I'm not yet sold on modifying existing code.

We don't want users to access the generated code from their project (they have ILSpy for that), or to store this code in the project/sourcecontrol, nor to be editable. They are mainly for post scenarios. Worse, if we are making an evolution to the underlying generated code (e.g removing a method), by just upgrading the package, we don't want users to have an un-compilable project.
We are not looking for improving the debugging experience (there are tools to step into post generated IL code actually)

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

We want to have a very fast compilation experience that would not require to re-parse the files, or re-read all the assemblies, multiple times (because today, we don't have any standard, so each solution has to queue an IL patcher to the build process via custom msbuild target files)

Pre-compilation independent generation out of proc runs at the speed of the user and does not affect compilation speed (except having a slightly larger project to parse). I understand this as a comment on today's IL generation scenario.

We want to have a standard and integrated way for libraries to provide such a compilation step through a NuGet package (ala Diagnostic analyzers)
We want to work at a higher level than IL level
We want a transparent and fluid experience for the users of our library/nuget compiler plugin

Did you look at @AnthonyDGreen's Design Safari approach to an AOP like experience? It would allow you to add attributes (via a NuGet package) that added code at emit time in a very controlled manner. The guard rails I have in mind for that might be too tight. But if you have a scenario you think would not work I'd be happy to noodle on it.

Most of the concrete use cases we have been discussing in the posts are coming from Serializers/RPC (@ReubenBond, myself) , ORM (@mgravell) , AOP (a few folks around, including myself)...

I don't see an issue, but I assume you fleshed out your thoughts on inconsistencies with an independent approach in the bullet points I already answered.

I'd really like you to take a look at the vblang issue. It's AOP via C# code. If it only handles INotifyPropertyChanged, it's viability is quite a bit different than if it also handles your pain points.

If IL is painful for authors, and source-generation is seen as a bigger issue by the team, can we finesse by having attributes actually tell us what they will do and inserting that code.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

Did you look at @AnthonyDGreen's Design Safari approach to an AOP like experience? It would allow you to add attributes (via a NuGet package) that added code at emit time in a very controlled manner. The guard rails I have in mind for that might be too tight. But if you have a scenario you think would not work I'd be happy to noodle on it.
If IL is painful for authors, and source-generation is seen as a bigger issue by the team, can we finesse by having attributes actually tell us what they will do and inserting that code.

What would be these attributes? (like FreezableAttribute?) Would they be hardcoded by Roslyn? Usually, we are working with domain specific attributes, and the compiler rewriter is able to understand them. For serializer, they are usually domain specific (various domains can have a very different way/restrictions to handle serialization). So if you are proposing hardcoded attributes, this is too restricting and not what we are using today.

One thing that I forgot (and the reason why "readonly" post-scenarios are important). We don't want to have reverse dependencies, user code starting to reference generated code... with a pre-step solution like "independent generation" this is a bit more difficult to avoid (though maybe you could provide an analyzer that would throw an error in that case, ok).

Also, pre-compilation-step/"independent generation" will just exclude any versatile AOP scenario (unless the attributes allow this, but I don't see how, as AOP is by essence a post-step operation)

But fair enough, if you are implying that "independent generation" is the only plan that is accepted by the Roslyn Team, that's of course better than having nothing, and it will certainly help a few projects out there. But it is going to be a lot more involving than a post-step compilation (you need a close IDE integration story, a way to remove files previously generated by plugins, a way for the plugin to not take into account the generated files when re-generating...etc.), and unlikely to happen in the coming months if not years (let's come back here in one year to challenge this).

But hey, thinking about it, we could have both, post-step done now, pre-step done whenever, and we would be more than happy! 😉

Anyway, on our side, at Unity, we have already started to use our own Roslyn based C# compiler, so we will extend the compilation pipeline from there instead.

Member

xoofx commented Jun 20, 2018

Did you look at @AnthonyDGreen's Design Safari approach to an AOP like experience? It would allow you to add attributes (via a NuGet package) that added code at emit time in a very controlled manner. The guard rails I have in mind for that might be too tight. But if you have a scenario you think would not work I'd be happy to noodle on it.
If IL is painful for authors, and source-generation is seen as a bigger issue by the team, can we finesse by having attributes actually tell us what they will do and inserting that code.

What would be these attributes? (like FreezableAttribute?) Would they be hardcoded by Roslyn? Usually, we are working with domain specific attributes, and the compiler rewriter is able to understand them. For serializer, they are usually domain specific (various domains can have a very different way/restrictions to handle serialization). So if you are proposing hardcoded attributes, this is too restricting and not what we are using today.

One thing that I forgot (and the reason why "readonly" post-scenarios are important). We don't want to have reverse dependencies, user code starting to reference generated code... with a pre-step solution like "independent generation" this is a bit more difficult to avoid (though maybe you could provide an analyzer that would throw an error in that case, ok).

Also, pre-compilation-step/"independent generation" will just exclude any versatile AOP scenario (unless the attributes allow this, but I don't see how, as AOP is by essence a post-step operation)

But fair enough, if you are implying that "independent generation" is the only plan that is accepted by the Roslyn Team, that's of course better than having nothing, and it will certainly help a few projects out there. But it is going to be a lot more involving than a post-step compilation (you need a close IDE integration story, a way to remove files previously generated by plugins, a way for the plugin to not take into account the generated files when re-generating...etc.), and unlikely to happen in the coming months if not years (let's come back here in one year to challenge this).

But hey, thinking about it, we could have both, post-step done now, pre-step done whenever, and we would be more than happy! 😉

Anyway, on our side, at Unity, we have already started to use our own Roslyn based C# compiler, so we will extend the compilation pipeline from there instead.

@agocke

This comment has been minimized.

Show comment
Hide comment
@agocke

agocke Jun 20, 2018

Contributor

Just to say once more for the record, I think we can do source generation, although everyone's pet feature may not be there, and it's tractable to make it work everywhere with a good experience. I think that we will eventually do it, it's simply a matter of priority. We would basically have to cancel C# 8 to get it working in the near-term

Contributor

agocke commented Jun 20, 2018

Just to say once more for the record, I think we can do source generation, although everyone's pet feature may not be there, and it's tractable to make it work everywhere with a good experience. I think that we will eventually do it, it's simply a matter of priority. We would basically have to cancel C# 8 to get it working in the near-term

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jun 20, 2018

@xoofx I am certainly not saying that pre-compilation is the only plan accepted by the Roslyn Team. We have accepted nothing, but individual members have been thinking hard on this topic, mostly for a long time.

vblang Design Safari has some rules for accessing the methods on the attributes and some conventions, but the choice of attributes and their naming is up to you. You're entirely in control, and I would not otherwise bring it up in a conversation on generative techniques.

As @agocke points out, the source generated solution is a lot of work, not impossible. Personally, I'd like to see consideration of both the vblang #282 approach and the pre-compilation approach. I think they cross up at the middle enough that if source generation is delayed a long time, most scenarios have an avenue forward.

@xoofx I am certainly not saying that pre-compilation is the only plan accepted by the Roslyn Team. We have accepted nothing, but individual members have been thinking hard on this topic, mostly for a long time.

vblang Design Safari has some rules for accessing the methods on the attributes and some conventions, but the choice of attributes and their naming is up to you. You're entirely in control, and I would not otherwise bring it up in a conversation on generative techniques.

As @agocke points out, the source generated solution is a lot of work, not impossible. Personally, I'd like to see consideration of both the vblang #282 approach and the pre-compilation approach. I think they cross up at the middle enough that if source generation is delayed a long time, most scenarios have an avenue forward.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

@KathleenDollard Forgot also a few comments to challenge the "independent generation" approach, I don't need necessarily answers now, but don't forget these cases:

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

For sure, in general, I will prefer a code debuggable rather than a non debuggable code.

But have you heard a story with PostSharp users frustrated by this that it would make the product completely unreliable because generated code is not production compatible if it is not debuggable? I'm all for a good debugging experience, but there are also scenarios where the generated debug code is not where the debugging experience is meaningful, where the generated code is rock solid (battery tested), while user code is usually... not.

Also, don't forget merging branches of people working concurrently on a code base with pre-step generators: It will generate merging conflicts on existing generated code (assuming that the code is checked-in in the repository), conflicts that can't be merged without re-generating the generated code after the merge (oh, and you could merge a code that have no conflicts, but is wrong, because the generated code would be different... )

Pre-compilation independent generation out of proc runs at the speed of the user and does not affect compilation speed (except having a slightly larger project to parse). I understand this as a comment on today's IL generation scenario.

What is going to be editing life cycle? How does the generation is going to be triggered? (on idle?) What if you generate serializers in your project and you remove a field that is used by the serializer? If a users has edited some generated code, should it be overwritten entirely or not? What is going to be the sync policies? What if you rename a field? It will change the generated code (or not) but it should re-trigger a generation? There will be some serious IDE integration story here, quite close to what "source generation" is also trying to challenge.

Member

xoofx commented Jun 20, 2018

@KathleenDollard Forgot also a few comments to challenge the "independent generation" approach, I don't need necessarily answers now, but don't forget these cases:

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

For sure, in general, I will prefer a code debuggable rather than a non debuggable code.

But have you heard a story with PostSharp users frustrated by this that it would make the product completely unreliable because generated code is not production compatible if it is not debuggable? I'm all for a good debugging experience, but there are also scenarios where the generated debug code is not where the debugging experience is meaningful, where the generated code is rock solid (battery tested), while user code is usually... not.

Also, don't forget merging branches of people working concurrently on a code base with pre-step generators: It will generate merging conflicts on existing generated code (assuming that the code is checked-in in the repository), conflicts that can't be merged without re-generating the generated code after the merge (oh, and you could merge a code that have no conflicts, but is wrong, because the generated code would be different... )

Pre-compilation independent generation out of proc runs at the speed of the user and does not affect compilation speed (except having a slightly larger project to parse). I understand this as a comment on today's IL generation scenario.

What is going to be editing life cycle? How does the generation is going to be triggered? (on idle?) What if you generate serializers in your project and you remove a field that is used by the serializer? If a users has edited some generated code, should it be overwritten entirely or not? What is going to be the sync policies? What if you rename a field? It will change the generated code (or not) but it should re-trigger a generation? There will be some serious IDE integration story here, quite close to what "source generation" is also trying to challenge.

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jun 20, 2018

@xoofx In my limited experience, PostSharp users are working with a subset of generated code that is rock solid. It seems much harder for project specific generation (with a need to debug, etc) to work in a late-generation scenario - thus making it good for a subset of problems (I hope I already said that).

The question of merge I think is the tip of a broader conversation. All of the portions of the generation need to be available for CI - thus, redoing the generation should not be hard. The merge conflict should occur on the metadata - which I hope is checked in too. It's not just the merge to have a clear story on.

It's important users don't edit generated code. Some of my points about why I think there is some work to be done on a pre-compilation/independent generation approach were due to this. Programmers and source control rules need to know what code is generated and treat it that way. If we stick to a file level, extensions work pretty well.

I don't see being able to trigger an external process on an analyzer as being as complicated to build as source generation, but we really have not explored this approach with the team. I tried to mark my earlier comments as my opinion, and I'm sorry if that wasn't clear. We haven't worked out the details of any approach, except @agocke work on source-generation.

Is it clear that I think AOP and pre-compilation are each valid in a set of scenarios?

@xoofx In my limited experience, PostSharp users are working with a subset of generated code that is rock solid. It seems much harder for project specific generation (with a need to debug, etc) to work in a late-generation scenario - thus making it good for a subset of problems (I hope I already said that).

The question of merge I think is the tip of a broader conversation. All of the portions of the generation need to be available for CI - thus, redoing the generation should not be hard. The merge conflict should occur on the metadata - which I hope is checked in too. It's not just the merge to have a clear story on.

It's important users don't edit generated code. Some of my points about why I think there is some work to be done on a pre-compilation/independent generation approach were due to this. Programmers and source control rules need to know what code is generated and treat it that way. If we stick to a file level, extensions work pretty well.

I don't see being able to trigger an external process on an analyzer as being as complicated to build as source generation, but we really have not explored this approach with the team. I tried to mark my earlier comments as my opinion, and I'm sorry if that wasn't clear. We haven't worked out the details of any approach, except @agocke work on source-generation.

Is it clear that I think AOP and pre-compilation are each valid in a set of scenarios?

@m0sa

This comment has been minimized.

Show comment
Hide comment
@m0sa

m0sa Jun 20, 2018

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

FWIW compiler optimizations in release builds are not exactly easily understandable and debuggable either...

m0sa commented Jun 20, 2018

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

FWIW compiler optimizations in release builds are not exactly easily understandable and debuggable either...

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 20, 2018

Member

Is it clear that I think AOP and pre-compilation are each valid in a set of scenarios?

Yes, and I was actually half joking when saying "we could have both, post-step done now, pre-step done whenever, and we would be more than happy"

  1. Post-compilation:
    • Pros: Allow AOP scenario and covers most of ORM/Serializers/RPC codegen scenarios that are already used today. Very easy to implement in Roslyn (one person, maybe two to plug the stuffs in msbuild targets to pass the arguments coming from the compiler-rewriters to csc.exe, as it is done currently for diagnostic analyzers)
    • Cons: Not debuggable
  2. Pre-compilation / source generator:
    • Pros: Debuggable, you can inspect codegen without going through ILSpy
    • Cons: No AOP scenarios. A lot more complex to develop (ide, intellisense, sync...etc., lots of teams to involved), not even sure it will make it one day.

I'm pretty sure that the community would be fine to make a PR for 1) (next week?), while for 2) it is barely impossible. But from the discussions, 1) seems to be considered as an entire no-go (nothing has changed since discussions happened more than one year ago). I'm very practical when it comes to deliver things and so, I'm highly skeptical that 2) will ever make it (plus the fact that it doesn't cover AOP scenarios).

Member

xoofx commented Jun 20, 2018

Is it clear that I think AOP and pre-compilation are each valid in a set of scenarios?

Yes, and I was actually half joking when saying "we could have both, post-step done now, pre-step done whenever, and we would be more than happy"

  1. Post-compilation:
    • Pros: Allow AOP scenario and covers most of ORM/Serializers/RPC codegen scenarios that are already used today. Very easy to implement in Roslyn (one person, maybe two to plug the stuffs in msbuild targets to pass the arguments coming from the compiler-rewriters to csc.exe, as it is done currently for diagnostic analyzers)
    • Cons: Not debuggable
  2. Pre-compilation / source generator:
    • Pros: Debuggable, you can inspect codegen without going through ILSpy
    • Cons: No AOP scenarios. A lot more complex to develop (ide, intellisense, sync...etc., lots of teams to involved), not even sure it will make it one day.

I'm pretty sure that the community would be fine to make a PR for 1) (next week?), while for 2) it is barely impossible. But from the discussions, 1) seems to be considered as an entire no-go (nothing has changed since discussions happened more than one year ago). I'm very practical when it comes to deliver things and so, I'm highly skeptical that 2) will ever make it (plus the fact that it doesn't cover AOP scenarios).

@m0sa

This comment has been minimized.

Show comment
Hide comment
@m0sa

m0sa Jun 20, 2018

I'd argue that anything that has a run-time implementation, which works w/o post-compilation, and wants to do post-compilation steps, is an optimization. ORM/Serializers as well as a lot of AOP in the .NET ecosystem would fall under this definition.

What most folks use locally (when developing / debugging simple stuff) is the not-optimized / debug build (see my point above, you can't event step through a simple for loop in it's entirety if you debug the release build with optimizations turned on). Unfortunately, I know a lot of people who are used to dealing with behavior mismatches between debug / release builds, and go digging into IL / JITed code when they stumble across such a case.

It feels like that the main pain point for debugging in this (optimization) scenario is the #line directive, especially when there is a chain of transformations on a single file. To address this, I'd take inspiration from source-maps. They work nicely throughout the pipeline (e.g. typescript -> javascript -> minifiers/optimizers -> bundlers). The original / intermediate files can nowadays be stored as embedded sources in PDBs (dotnet/roslyn#12625)

m0sa commented Jun 20, 2018

I'd argue that anything that has a run-time implementation, which works w/o post-compilation, and wants to do post-compilation steps, is an optimization. ORM/Serializers as well as a lot of AOP in the .NET ecosystem would fall under this definition.

What most folks use locally (when developing / debugging simple stuff) is the not-optimized / debug build (see my point above, you can't event step through a simple for loop in it's entirety if you debug the release build with optimizations turned on). Unfortunately, I know a lot of people who are used to dealing with behavior mismatches between debug / release builds, and go digging into IL / JITed code when they stumble across such a case.

It feels like that the main pain point for debugging in this (optimization) scenario is the #line directive, especially when there is a chain of transformations on a single file. To address this, I'd take inspiration from source-maps. They work nicely throughout the pipeline (e.g. typescript -> javascript -> minifiers/optimizers -> bundlers). The original / intermediate files can nowadays be stored as embedded sources in PDBs (dotnet/roslyn#12625)

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Jun 22, 2018

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

Compare to the situation today. The people using the tools that we're talking about already don't have that. Nobody has complained. Yes, I understand the issue of rogue tooling doing nasty things, but you pretty much have that today the moment you install any package. Being readable source code isn't the thing that adds safety.

We'll just have to agree to disagree on whether programmers should have easy direct access to the code that runs in production and be able to easily understand and debug it.

Compare to the situation today. The people using the tools that we're talking about already don't have that. Nobody has complained. Yes, I understand the issue of rogue tooling doing nasty things, but you pretty much have that today the moment you install any package. Being readable source code isn't the thing that adds safety.

@LokiMidgard

This comment has been minimized.

Show comment
Hide comment
@LokiMidgard

LokiMidgard Jun 22, 2018

The people using the tools that we're talking about already don't have that. Nobody has complained.

I don't like the way how IL manipulation works. But it is the only way to do some tasks (like replacing a method with another implementation e.g. INotifyProeprtyChanged).
But I would never complain that a project uses IL manipulation. After all nobody forces me to use it.


That saied I would like to see the original replace feature implemented even without source code generation support in the compiler, vs and intelisense. There are already usecases (INotifyPropertyChanged may be the prominetesd) where we dont need support to generate code on every keystroke. If no new methods/clases are generated ony ones that are there replaced. those two keywords would be enogh to have a value. The only reason I see to stop this untill the source code generating stuff is done, if we create problems with how the keywords are implemented.

In addition there are already source code generating librarys out there which could benefit from this.
Including those integrated in Visual Studio today. E,g, The Windows Form designer creates the void Dispose(bool disposing) method, you cannot change it because generated files tend to be regenerated and overiding you changes, you can not override it, for that you would need to inherited your from. What you want is replace the method. So the two keywords can not only benefit new tools but also existing ones.

The people using the tools that we're talking about already don't have that. Nobody has complained.

I don't like the way how IL manipulation works. But it is the only way to do some tasks (like replacing a method with another implementation e.g. INotifyProeprtyChanged).
But I would never complain that a project uses IL manipulation. After all nobody forces me to use it.


That saied I would like to see the original replace feature implemented even without source code generation support in the compiler, vs and intelisense. There are already usecases (INotifyPropertyChanged may be the prominetesd) where we dont need support to generate code on every keystroke. If no new methods/clases are generated ony ones that are there replaced. those two keywords would be enogh to have a value. The only reason I see to stop this untill the source code generating stuff is done, if we create problems with how the keywords are implemented.

In addition there are already source code generating librarys out there which could benefit from this.
Including those integrated in Visual Studio today. E,g, The Windows Form designer creates the void Dispose(bool disposing) method, you cannot change it because generated files tend to be regenerated and overiding you changes, you can not override it, for that you would need to inherited your from. What you want is replace the method. So the two keywords can not only benefit new tools but also existing ones.

@amis92

This comment has been minimized.

Show comment
Hide comment
@amis92

amis92 Jun 22, 2018

Undebuggable (?) compiler-generated, well-known and well-defined desugared code (often with support for sugar debugging, as is the case with e.g. iterators and async) cannot be compared to hiding externally generated source code. It was said a couple of times already, but the expectations differ hugely between compiler platform and 3rd parties.

I'm definitely expecting a full-blown, debug-supported, source code based, and tooling-friendly scenario from the compiler platform. Cutting down a couple of compilations isn't worth enabling poor-man's integration within compiler. This platform's strength is it's integrity as well, and I feel like this integrity would be a little broken if it enabled magical weaving of source trees that doesn't have reflection in source code. You can do that today (Fody etc), but it's not what I'd like to see flourish in decades to come, so to speak.


That said, I'd also like to ask what AOP scenarios can't be covered by source generation as proposed in OP?

Pre-compilation / source generator:

  • Pros: Debuggable, you can inspect codegen without going through ILSpy
  • Cons: No AOP scenarios. [...]

amis92 commented Jun 22, 2018

Undebuggable (?) compiler-generated, well-known and well-defined desugared code (often with support for sugar debugging, as is the case with e.g. iterators and async) cannot be compared to hiding externally generated source code. It was said a couple of times already, but the expectations differ hugely between compiler platform and 3rd parties.

I'm definitely expecting a full-blown, debug-supported, source code based, and tooling-friendly scenario from the compiler platform. Cutting down a couple of compilations isn't worth enabling poor-man's integration within compiler. This platform's strength is it's integrity as well, and I feel like this integrity would be a little broken if it enabled magical weaving of source trees that doesn't have reflection in source code. You can do that today (Fody etc), but it's not what I'd like to see flourish in decades to come, so to speak.


That said, I'd also like to ask what AOP scenarios can't be covered by source generation as proposed in OP?

Pre-compilation / source generator:

  • Pros: Debuggable, you can inspect codegen without going through ILSpy
  • Cons: No AOP scenarios. [...]
@popcatalin81

This comment has been minimized.

Show comment
Hide comment
@popcatalin81

popcatalin81 Jun 22, 2018

Being readable source code isn't the thing that adds safety.

It's very important, I personally, would say it's Paramount. Otherwise, you might push users into debugging hell, considering how much a generator can modify the code and alter the semantics.

(I'm not implying that fair playing library designers would do this intentionally ...)

Edit:

On the other hand, we've used reflection emit successfully over the years, so black box reflection emit code generation is not unusable for the general case, because of the restrictions it inherently has IE: cannot modify existing code unless it was an explicit override point exposed like virtual properties.

popcatalin81 commented Jun 22, 2018

Being readable source code isn't the thing that adds safety.

It's very important, I personally, would say it's Paramount. Otherwise, you might push users into debugging hell, considering how much a generator can modify the code and alter the semantics.

(I'm not implying that fair playing library designers would do this intentionally ...)

Edit:

On the other hand, we've used reflection emit successfully over the years, so black box reflection emit code generation is not unusable for the general case, because of the restrictions it inherently has IE: cannot modify existing code unless it was an explicit override point exposed like virtual properties.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 22, 2018

Member

That said, I'd also like to ask what AOP scenarios can't be covered by source generation as proposed in OP?

@amis92 sure, almost all AOP scenarios are incompatible with it. Source generation is assuming that the generated source is a strong part of the project right? The generated code is accessible by your code (which is a chicken egg problem, so you need to assume that there is some pre-compilation step involved here)

Now, what's happening if you have a file Program.cs which an AOP is going to emit a pre and post callback on it:

public static void Main()
{
     Console.WriteLine("Hello World");
}

AOP is supposed to modify your code to write this, by somehow rewriting Program.cs by a virtual Program1.cs:

public static void Main()
{
#line hidden
     CallSomeCodeProlog();
#line 3 "Program.cs"
     Console.WriteLine("Hello World");
#line hidden
     CallSomeCodeEpilog();
}

So assuming that this file is part of your project now. How can compilation work now that we have Program.cs and Program1.cs? That's the whole point of AOP: you can't have it as part of your project because it is a post-processing step that can modify existing code. Same apply if you want to produce a serializer library that will transform existing class with "partial," access private fields...etc. You would have to modify the existing code.

Post-compilation steps allow this kind of scenarios because modified file are not part of your project, but they still can be debugged (because you can dump them to obj folder and use proper #line pragma as I have shown in the example above)

Again, we are not talking into the vacuum of a theory here, we have been practicing this for years (with thousands of users reached indirectly through our products)

Member

xoofx commented Jun 22, 2018

That said, I'd also like to ask what AOP scenarios can't be covered by source generation as proposed in OP?

@amis92 sure, almost all AOP scenarios are incompatible with it. Source generation is assuming that the generated source is a strong part of the project right? The generated code is accessible by your code (which is a chicken egg problem, so you need to assume that there is some pre-compilation step involved here)

Now, what's happening if you have a file Program.cs which an AOP is going to emit a pre and post callback on it:

public static void Main()
{
     Console.WriteLine("Hello World");
}

AOP is supposed to modify your code to write this, by somehow rewriting Program.cs by a virtual Program1.cs:

public static void Main()
{
#line hidden
     CallSomeCodeProlog();
#line 3 "Program.cs"
     Console.WriteLine("Hello World");
#line hidden
     CallSomeCodeEpilog();
}

So assuming that this file is part of your project now. How can compilation work now that we have Program.cs and Program1.cs? That's the whole point of AOP: you can't have it as part of your project because it is a post-processing step that can modify existing code. Same apply if you want to produce a serializer library that will transform existing class with "partial," access private fields...etc. You would have to modify the existing code.

Post-compilation steps allow this kind of scenarios because modified file are not part of your project, but they still can be debugged (because you can dump them to obj folder and use proper #line pragma as I have shown in the example above)

Again, we are not talking into the vacuum of a theory here, we have been practicing this for years (with thousands of users reached indirectly through our products)

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 22, 2018

Member

The problem with Source Code generator described here, is that it is very very similar to what we would do with a post-compilation step (similar API to rewrite Compilation, replace SyntaxTree, output to obj files, debuggable...etc.) but the difference is that the generated code is part of your project and user can access it, so it is a pre-compilation step in the end. And that's very different, because it is bringing a lot more trouble for the IDE story (and even more problems that we are not looking for, like generating serializers on every single key stroke or on whatever agreement on the sync it would do)

Member

xoofx commented Jun 22, 2018

The problem with Source Code generator described here, is that it is very very similar to what we would do with a post-compilation step (similar API to rewrite Compilation, replace SyntaxTree, output to obj files, debuggable...etc.) but the difference is that the generated code is part of your project and user can access it, so it is a pre-compilation step in the end. And that's very different, because it is bringing a lot more trouble for the IDE story (and even more problems that we are not looking for, like generating serializers on every single key stroke or on whatever agreement on the sync it would do)

@amis92

This comment has been minimized.

Show comment
Hide comment
@amis92

amis92 Jun 22, 2018

@xoofx I'd imagine two solutions:

  1. Rewrite and replace completely

    // Program.generated.cs
    public static replace void Main()
    {
    #line hidden
        CallSomeCodeProlog();
    #line 3 "Program.cs"
        Console.WriteLine("Hello World");
    #line hidden
        CallSomeCodeEpilog();
    }
  2. Call back to the original

    // Program.generated.cs
    public static replace void Main()
    {
    #line hidden
        CallSomeCodeProlog();
    #line 3 "Program.cs"
        original();
    #line hidden
        CallSomeCodeEpilog();
    }

The developer story would be that the IDE detects complete method replacement and shows the Main in Program.cs greyed out or sth with clear visual that the method is replaced and a navigable list of replacements that call into original and single base call (not necessarily from the original source file).

I do assume that we have replace/original keyword support, and multiple generator support (multiple replacements of single member), which both are suggested in the original proposal. I don't think this feature can exist without these additions.

I'd imagine that multiple replacement would be resolved by assembly-level attribute that defines the order in which generators' replacements are called into each other.

// AssemblyInfo.cs

[assembly: SourceGeneratorsOrder(
    typeof(BoundaryLoggerGenerator), // outer-most
    typeof(ValidationGenerator), // middle
    typeof(LinqRewriterGenerator) // inner
)]
// Program.cs
public partial class Program
{
    public void Print(int[] numbers)
    {
        var query = from x in numbers
                    where x > 3
                    select x.ToString();
        query.Select(Console.WriteLine);
    }
}
// Program.LinqRewriterGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        foreach (var x in new[] {1,2,3,4})
        {
            if (x > 3)
            {
                Console.WriteLine(x.ToString());
            }
        }
    }
}
// Program.ValidationGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        if (numbers == null)
            throw new ArgumentNullException(nameof(numbers));
        original(numbers);
    }
}
// Program.BoundaryLoggerGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        Console.WriteLine("Entering Program.Print(int[])");
        original(numbers);
        Console.WriteLine("Exiting Program.Print(int[])");
    }
}

And the result of the call to Program.Print would be:

Program.Print // Program.BoundaryLoggerGenerator.generated.cs calls:
Program.Print // Program.ValidationGenerator.generated.cs calls:
Program.Print // Program.LinqRewriterGenerator.generated.cs doesn't call any other original/replacement

The original code was never called. If it was called, how would you debug the query.Select invocation?

amis92 commented Jun 22, 2018

@xoofx I'd imagine two solutions:

  1. Rewrite and replace completely

    // Program.generated.cs
    public static replace void Main()
    {
    #line hidden
        CallSomeCodeProlog();
    #line 3 "Program.cs"
        Console.WriteLine("Hello World");
    #line hidden
        CallSomeCodeEpilog();
    }
  2. Call back to the original

    // Program.generated.cs
    public static replace void Main()
    {
    #line hidden
        CallSomeCodeProlog();
    #line 3 "Program.cs"
        original();
    #line hidden
        CallSomeCodeEpilog();
    }

The developer story would be that the IDE detects complete method replacement and shows the Main in Program.cs greyed out or sth with clear visual that the method is replaced and a navigable list of replacements that call into original and single base call (not necessarily from the original source file).

I do assume that we have replace/original keyword support, and multiple generator support (multiple replacements of single member), which both are suggested in the original proposal. I don't think this feature can exist without these additions.

I'd imagine that multiple replacement would be resolved by assembly-level attribute that defines the order in which generators' replacements are called into each other.

// AssemblyInfo.cs

[assembly: SourceGeneratorsOrder(
    typeof(BoundaryLoggerGenerator), // outer-most
    typeof(ValidationGenerator), // middle
    typeof(LinqRewriterGenerator) // inner
)]
// Program.cs
public partial class Program
{
    public void Print(int[] numbers)
    {
        var query = from x in numbers
                    where x > 3
                    select x.ToString();
        query.Select(Console.WriteLine);
    }
}
// Program.LinqRewriterGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        foreach (var x in new[] {1,2,3,4})
        {
            if (x > 3)
            {
                Console.WriteLine(x.ToString());
            }
        }
    }
}
// Program.ValidationGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        if (numbers == null)
            throw new ArgumentNullException(nameof(numbers));
        original(numbers);
    }
}
// Program.BoundaryLoggerGenerator.generated.cs
partial class Program
{
    replace void Print(int[] numbers)
    {
        Console.WriteLine("Entering Program.Print(int[])");
        original(numbers);
        Console.WriteLine("Exiting Program.Print(int[])");
    }
}

And the result of the call to Program.Print would be:

Program.Print // Program.BoundaryLoggerGenerator.generated.cs calls:
Program.Print // Program.ValidationGenerator.generated.cs calls:
Program.Print // Program.LinqRewriterGenerator.generated.cs doesn't call any other original/replacement

The original code was never called. If it was called, how would you debug the query.Select invocation?

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 22, 2018

Member

I do assume that we have replace/original keyword support,

That's a requirement that AOP doesn't want nor you would like to have this for serliazers or ORM mapper or....etc. The purpose of AOP (or serializers...etc.) is to be able to work horizontally/vertically on your code without you having to modify your codebase. It would mean that you would have to make all method or type "replace" by default. That's not sustainable.Plus the case I used with Program.cs is really too simple. When you start to have other things in the same file that are not modified or partial class...etc. It is going to be just a nightmare to get all of this working.

And what you propose here, is actually serving the whole point of why post compilation is a lot more easier. Your solution is bringing such a complexity for IDE integration, intellisense, project system...etc. and is basically why the whole codegen issue is being blocked (including the post-compilation scenario)

We have already a solution (post-compilation) that is perfectly viable, proven working (by existing solutions like IL patching that are less flexible), allow debugging and has zero impact on IDE/project-system...etc.

Member

xoofx commented Jun 22, 2018

I do assume that we have replace/original keyword support,

That's a requirement that AOP doesn't want nor you would like to have this for serliazers or ORM mapper or....etc. The purpose of AOP (or serializers...etc.) is to be able to work horizontally/vertically on your code without you having to modify your codebase. It would mean that you would have to make all method or type "replace" by default. That's not sustainable.Plus the case I used with Program.cs is really too simple. When you start to have other things in the same file that are not modified or partial class...etc. It is going to be just a nightmare to get all of this working.

And what you propose here, is actually serving the whole point of why post compilation is a lot more easier. Your solution is bringing such a complexity for IDE integration, intellisense, project system...etc. and is basically why the whole codegen issue is being blocked (including the post-compilation scenario)

We have already a solution (post-compilation) that is perfectly viable, proven working (by existing solutions like IL patching that are less flexible), allow debugging and has zero impact on IDE/project-system...etc.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 22, 2018

Member

Again, not saying that pre-compilation doesn't bring interesting scenario (e.g access to generated code, useful for UI framework for example) nor that it is either a fight between pre or post. Both have their goals and should exist on their own. But in practice, post-compilation is the only one that could make it in a very short term, assuming that the Roslyn team agree with that (instead of trying to fit everything into a single solution, that everybody sees as so complex that it will probably not make it)

Member

xoofx commented Jun 22, 2018

Again, not saying that pre-compilation doesn't bring interesting scenario (e.g access to generated code, useful for UI framework for example) nor that it is either a fight between pre or post. Both have their goals and should exist on their own. But in practice, post-compilation is the only one that could make it in a very short term, assuming that the Roslyn team agree with that (instead of trying to fit everything into a single solution, that everybody sees as so complex that it will probably not make it)

@LokiMidgard

This comment has been minimized.

Show comment
Hide comment
@LokiMidgard

LokiMidgard Jun 22, 2018

If I understood it correctly ther would be no requirement to mark all your methods with replace. Only a method replacing another (which will in most cases be generated) would use this keyword and an replace keyword to call the original one. It is not like virtual

And I actually don't see the difference between post compile IL magic and pre-compile source code generation. As far as I know one issue is that there is a problem with performance of generating when it will be used by intelisense. It does not matter if a method will be created using IL manipulation or generating of source code. In both cases the the worst case is that the generator runs on every keystroke to update the methods that apear in intelisense. Post compile generating does not change this. The only thing that would change this is, we are not allowed to generate new stuff but only to replace already existing one. In this case again it does not matter if post or pre compile generation is used.

If I understood it correctly ther would be no requirement to mark all your methods with replace. Only a method replacing another (which will in most cases be generated) would use this keyword and an replace keyword to call the original one. It is not like virtual

And I actually don't see the difference between post compile IL magic and pre-compile source code generation. As far as I know one issue is that there is a problem with performance of generating when it will be used by intelisense. It does not matter if a method will be created using IL manipulation or generating of source code. In both cases the the worst case is that the generator runs on every keystroke to update the methods that apear in intelisense. Post compile generating does not change this. The only thing that would change this is, we are not allowed to generate new stuff but only to replace already existing one. In this case again it does not matter if post or pre compile generation is used.

@xoofx

This comment has been minimized.

Show comment
Hide comment
@xoofx

xoofx Jun 22, 2018

Member

And I actually don't see the difference between post compile IL magic and pre-compile source code generation.

There is a structural difference: pre-compilation allows the user code to reference generated code, while post-compilation doesn't allow this. The implication of this is following:

As far as I know one issue is that there is a problem with performance of generating when it will be used by intelisense. It does not matter if a method will be created using IL manipulation or generating of source code. In both cases the the worst case is that the generator runs on every keystroke to update the methods that apear in intelisense.
Post compile generating does not change this.

That's incorrect. Post-compile doesn't have to happen on IDE time and is only relevant at compile time. No IDE integration is needed at all (except the debugging side that doesn't need modification to the IDE as it is done already through PDB loading)

Member

xoofx commented Jun 22, 2018

And I actually don't see the difference between post compile IL magic and pre-compile source code generation.

There is a structural difference: pre-compilation allows the user code to reference generated code, while post-compilation doesn't allow this. The implication of this is following:

As far as I know one issue is that there is a problem with performance of generating when it will be used by intelisense. It does not matter if a method will be created using IL manipulation or generating of source code. In both cases the the worst case is that the generator runs on every keystroke to update the methods that apear in intelisense.
Post compile generating does not change this.

That's incorrect. Post-compile doesn't have to happen on IDE time and is only relevant at compile time. No IDE integration is needed at all (except the debugging side that doesn't need modification to the IDE as it is done already through PDB loading)

@LokiMidgard

This comment has been minimized.

Show comment
Hide comment
@LokiMidgard

LokiMidgard Jun 22, 2018

There is a structural difference: pre-compilation allows the user code to reference generated code, while post-compilation doesn't allow this.

That's not nessasary true. You could ignore all generated source code so it will only used in compilation and not interfere with the IDE expirience. I'm not sure if this works the other directions. You could defenetly predict the outcome of a generator and add intelisense for those methods, but I don't think the compiler allows to compile against methods that would exist after post il modification step.

I'm actually not a fan of IL manipulation, and try to avoid it if possible. So I must admit my perspective is a litle biased I think. Normaly I use custom tools that generate source code before compile. But as soon as I need to replace something, like propertys for INotifyPropertyChanged I currently have no good alternative. So those two keywords would help me alot even without first party support for code generator (still would like to see that).

There is a structural difference: pre-compilation allows the user code to reference generated code, while post-compilation doesn't allow this.

That's not nessasary true. You could ignore all generated source code so it will only used in compilation and not interfere with the IDE expirience. I'm not sure if this works the other directions. You could defenetly predict the outcome of a generator and add intelisense for those methods, but I don't think the compiler allows to compile against methods that would exist after post il modification step.

I'm actually not a fan of IL manipulation, and try to avoid it if possible. So I must admit my perspective is a litle biased I think. Normaly I use custom tools that generate source code before compile. But as soon as I need to replace something, like propertys for INotifyPropertyChanged I currently have no good alternative. So those two keywords would help me alot even without first party support for code generator (still would like to see that).

@BenjaBobs BenjaBobs referenced this issue in sschmid/Entitas-CSharp Jun 27, 2018

Open

Roslyn integrated generation. #734

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment