Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Champion "Replace/original and code generation extensions" #107

Open
1 of 5 tasks
gafter opened this issue Feb 14, 2017 · 317 comments
Open
1 of 5 tasks

Champion "Replace/original and code generation extensions" #107

gafter opened this issue Feb 14, 2017 · 317 comments
Assignees
Milestone

Comments

@gafter
Copy link
Contributor

gafter commented Feb 14, 2017

  • Proposal added
  • Discussed in LDM
  • Decision in LDM
  • Finalized (done, rejected, inactive)
  • Spec'ed

See also

@gafter gafter changed the title Replace/original and code generation extensions Champion Replace/original and code generation extensions Feb 15, 2017
@gafter gafter changed the title Champion Replace/original and code generation extensions Champion "Replace/original and code generation extensions" Feb 21, 2017
@gafter gafter added this to the X.0 candidate milestone Feb 22, 2017
@MgSam
Copy link

MgSam commented Feb 28, 2017

@MgSam
Copy link

MgSam commented Feb 28, 2017

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

@asdfgasdfsafgsdfa
Copy link

asdfgasdfsafgsdfa commented Mar 5, 2017

No news on this?
Would be really helpful for a ton of things...

@MgSam
Copy link

MgSam commented Mar 24, 2017

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

@asdfgasdfsafgsdfa
Copy link

asdfgasdfsafgsdfa commented Mar 24, 2017

Sorry if this is a dumb question but what exactly does tooling mean here?
The "debugger UI" in Visual Studio?
What parts would need to get changed?
Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

  1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
  2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
  3. Debugging. How do you debug through the code that is generated.
  4. Project-System. How do you present generated code in the project system?
  5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

@eyalalonn
Copy link
Contributor

eyalalonn commented Mar 24, 2017

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

I'm not sure i understand your question @eyalsk . Can you clarify?

@eyalalonn
Copy link
Contributor

eyalalonn commented Mar 24, 2017

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

The owners of those respective products would have to make a decision on that. :)

@eyalalonn
Copy link
Contributor

eyalalonn commented Mar 24, 2017

@CyrusNajmabadi Got'cha, thanks. 😉

@MgSam
Copy link

MgSam commented Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

@MgSam
Copy link

MgSam commented Mar 24, 2017

What doesn't make sense is the arbitrarily different bar for different MS products:

  • T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
  • As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics

It's not internal politics. We are the C# team. We decide what we feel is best for the language. And that includes insuring that it has a great development experience out of the box.

And there's a good reason for that. We don't want to ship the language changes too early, only to find out later that we royally screwed things up once we try to then get proper IDE support for it. SourceGenerators are a massively complex undertaking. We could try to get something out without considering and attempting to prove it would work in the IDE. But then it's highly likely we would get it very wrong.

Even our preliminary efforts here showed us immediately how our early source-generator designs simply would not be suitable to providing a good IDE experience.

@MgSam
Copy link

MgSam commented Mar 24, 2017

And that includes insuring that it has a great development experience out of the box.

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

@eyalalonn
Copy link
Contributor

eyalalonn commented Mar 24, 2017

@MgSam I agree with you on some of your points but I think that they probably don't want to make the same mistake twice and the experience I had with T4 from what I can remember was pretty awful to the point I removed the extension I wrote from the VS gallery. :)

@eyalalonn
Copy link
Contributor

eyalalonn commented Mar 24, 2017

@MgSam

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Didn't they fix this for RTM?

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects. We do not ship prototypes just because they're cool and they demo'ed well. A lot of real effort went into exploring this feature. Even just the language feature alone had a whole host of issues that we did not know how to resolve. And that was before the very first problem of: how will this even hook up into the IDE compilation flow model?

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

Didn't they fix this for RTM?

No. But we're working on it. The core problem is that there have been extremely hard perf goals to meet, such that even loading a single extra dll is not allowed. That's problematic in the context of any sort of extensibility story. We made a hard decision to ship in that state, even though we would have preferred it be better.**

And that goes to my point. That was something we did not want to do, and it was only because the scope of the problem was so limited. i.e. you would get an error about tuples, but at least you could solve it.

That doesn't even begin to compare to the issues you'd get with source generators. Again, imagine it was like tuples, and you got lots of errors in your IDE while trying to work. Except now there wouldn't be anything you could do about that. Also consider that just typing things like "replace" and whatnot would simply throw of the IDE parser completely, leading to code that was nearly impossible to edit.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails. Just basic editing will fall to the floor because we won't understand teh code you're writing on a syntactic and semantic level. And, currently, our IDE is based around the idea that we will understand at least that much.

--

** We've won back enough performance across the board though that we feel lke we can get an exception here and that this feature can be enabled soon. If/when that happens, it will mean there was a short window of time when we had a not-great experience. That's not at all comparable to the types of timeframes we're talking about for something like SourceGenerators.

And again, it's not like we can "just ship source generators now and just make the IDE work later". We have to understand the needs of hte IDE so that we can properly design source generators in the first place. We know this first hand from other work we've done in the past, including work in analyzers. The original analyzer work did not think closely enough about the compiler/IDE interaction, and as such had absolutely unacceptable performance characteristics. By designing things together, we were able to come up with something that met our feature goals while also having acceptable performance.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

There's no irony here. Nor is this an accurate representation of what actually happens in TypeScript. In TypeScript features are shipped and the TS team themselves does update the LS side of things at the same time as they're doing the compiler work**. It is the case that they can release a compiler that people can use before an official VS build is made available, but that doesn't mean that they're shipping language features that they then might not have any clue how to the IDE work for.

That's what's different here. IT's not like we know what source-generators are at a language level, could ship that, and could then just "do the IDE work". We do not know what source-generators are precisely because the design of the language feature itself is so informed by the tooling that will consume it.

--

** I know. I was involved with it for several years, and i still work closely with them as they're an adopter of Roslyn and an early adopter of lots of the improvements we make. I wrote a ton of both the LS features for TS as well as the majority of the LS implementation in VS. I also wrote the extensibility points in Roslyn to allow TS to become a Roslyn language.

@MgSam
Copy link

MgSam commented Mar 24, 2017

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects.

Totally fair. So there's a lot more work to be done. I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board. Good software/API design doesn't require explicit buy in and support from every user before you proceed. If it did, no software would ever get shipped.

I use PostSharp- which accomplishes many of the same things as this feature. In the first versions had nearly zero IDE support and the support still sucks pretty bad. That doesn't mean it's still not super useful.

Like I said, do your best effort, put it behind a feature flag and let your partners implement IDE support themselves. I still haven't heard a reason why making an optional feature that's turned off by default is problematic. Again, you guys do this constantly in TypeScript.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails.

Except if I'm using R#, in which case it'll probably work perfectly. Again, my point that language design shouldn't be hampered by MS politics.

TS team themselves does update the LS side of things at the same time as they're doing the compiler work

This is completely off on a tangent, but non-nullable types were released in TypeScript 2.0 in September. The --strictNullChecks flag still isn't supported in VS 2017, over 6 months later.

When TypeScript first came out, you had Mads Kristensen adding capabilities on his own into VS (like compile on save) because no one else was doing it. The TypeScript-VS team coordination has always been dysfunctional. Yet TypeScript has been a huge success nonetheless. And you guys should be applauded for pushing the language forward without waiting for VS.


I don't mean to get into an extended back-and-forth over this, but it's super-frustrating to see useful features like this continually delayed indefinitely so that the team can instead turn to syntax sugar features that have little real-world impact on code quality or productivity (like the entire C# 7.0 release). And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

So, in my opinion, the C# team needs to stop gating progress on other MS teams and move ahead doing your best to design features and APIs. If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 24, 2017

I still haven't heard a reason why making an optional feature that's turned off by default is problematic.

I have given the reason:

Because we literally don't know how the feature should work. We don't know what the right design is for it. How do we even create it if we can't figure out these questions :)

I just don't buy the argument that it's impossible to do the work unless 6 other MS teams first jump on board.

Good thing i didn't make that argument ;-)

The argument is that the C# team itself (i.e. us) are not willing to ship the feature when we can't deliver even a basic experience that is satisfactory. Nor are we going to just ship the language/compiler feature if we don't even have an idea for a design that we think could be sufficient.

and let your partners implement IDE support themselves.

And when the partners (i.e. the C# team) says to the C# team: "uh... there's no way we can get any sort of acceptable experience here with the design you shipped to customers..." then we're in a big heap of trouble. :)

And you guys should be applauded for pushing the language forward without waiting for VS.

Again, TS does not create language/compiler features that they don't know how they'll integrate into their LS. I'm not sure how much more plainly i can state that.

And the reason is always the same- "this feature requires *** team inside Microsoft to cooperate so let's delay it indefinitely"

Please stop saying "*** team" :). The reason here is that the C# team itself does not know what the right design is for thsi feature. We don't know what it is and even as we experimented and prototyped many options here we ran into enormous issues (even at the language/compiler level) that we simply had no solution for. On top of that, the C# team itself does not want to ever ship language/compiler features that it doesn't know if it will be able to deliver a suitable developer experience around. That is a core goal of our language, and we believe it is very important. Right now we could come up with literally zero designs that we could feel confident that we'd be able to provide even a basic experience around, let alone a good experience. It would be enormously irresponsible of us to ship something like that at the language level, especially with the expectations people have had from us for the last decade+.

Seriously, you guys do a better job cooperating with the Angular team on TypeScript then you do cooperating with other teams inside of Microsoft.

It's not other teams. We are the C# team. The C# team has not been able to come up with a satisfactory design for this feature, and also feels that there are enormous complexities and problems that would be extremely difficult to deal with for the developer experience.

--

I feel like you're thinking the situation is one of: "Oh man... we have this amazing feature. And we totally know how to do it. But those pesky devs over there are preventing it from going out! :( ".

That's not what is happening here. We simply do not have a design that we feel actually delivers the right feature to users (let alone an implementation to go along with that design). And we are never going to ship something that we can't even come up with a design for.

--

If it ends up needing changes for native VS support to work later on so be it- that's agile software development.

Sorry, the rules of C# and the promises we've made over the years still apply. Backcompat is still a thing. We're not going to just throw this feature out there, then realize it is full of problems, and go "sorry, gotta break everyone, that's agile software development". We are the language and platform. We get that this would be hugely useful. But that's all the more reason to try to get something good that can last before shipping things out to people.

@Pzixel
Copy link

Pzixel commented Oct 10, 2019

Yes, that's what I meant when saying "I understand IDE concerns". But without it it's just a toy feature "look, we can inject logging in the beggining and the end of the method. Woah". IMHO, of course.

@vbcodec
Copy link

vbcodec commented Mar 4, 2020

https://github.com/dotnet/roslyn/projects/54
What does this means ? Seems like wish list. Hard to say if this is serious commitment or another prototypical attempt.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 4, 2020

Wouldn't trying to prototype this indicate a serious commitment?

@vbcodec
Copy link

vbcodec commented Mar 5, 2020

Actually not. You may build prototype after prototype, but without serious commitment, never able to push it through RTM phase.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 5, 2020

We're not going to be able to make this happen without prototyping it. It's a challenging and complex space. I cannot be designed in absence as this is really about a tooling feature (like analyzers), and much less of a language feature.

MS is currently paying people to do this and allocating the time to make this happen, ahead of a long list of other with that could be fine instead. If you don't see that as commitment, then I don't know what to tell you.

@jaredpar
Copy link
Member

jaredpar commented Mar 5, 2020

https://github.com/dotnet/roslyn/projects/54

This is the GitHub project used to track our upcoming source generator work. The aspiration is to ship this as a part of the C# 9.0 release.

You may build prototype after prototype, but without serious commitment, never able to push it through RTM phase.

As @CyrusNajmabadi noted this is a complex area and one where we feel strongly that we need direct customer feedback to ensure we deliver the correct solution. Delivering a regular stream of prototypes through the C# 9.0 release where we can iterate with those customers is the best way to get that feedback.

If you don't see that as commitment, then I don't know what to tell you.

+100.

@dominikjeske
Copy link

dominikjeske commented Mar 5, 2020

Maybe you can join forces with https://github.com/AArnott/CodeGeneration.Roslyn

@agocke
Copy link
Member

agocke commented Mar 5, 2020

@AArnott Is certainly welcome to weigh in

@AArnott
Copy link
Contributor

AArnott commented Mar 6, 2020

I'm stoked about the C# 9 built-in feature. I'd love to try a prototype.

@HamedFathi
Copy link

HamedFathi commented Mar 22, 2020

About the source generator proposal, Can it brings Rust`s macro and procedural macro as a new game-changer feature into C#?

https://doc.rust-lang.org/rust-by-example/macros.html
https://medium.com/@phoomparin/a-beginners-guide-to-rust-macros-5c75594498f1

Example: Implementing Ruby’s HashMap syntax in Rust

a

(Procedural) Macros in Rust is so powerful, We can generate source codes with custom DSLs, syntax and much more.
Any idea?

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 22, 2020

Custom DSLs are very intentionally not in scope for this.

@Pzixel
Copy link

Pzixel commented Mar 22, 2020

@HamedFathi rust macros are known for poor IDE integration, especially procedural ones. I don't think it worth taking them as they are even if LDM was interested in it, which they aren't.

@bugproof
Copy link

bugproof commented Mar 28, 2020

I really like how macros work in Rust. So things like Serde (mapping to struct directly ) are possible without using reflection.

This would change completely the way you can create your serializer and it would improve performance a lot. You could generate your mapping/serialization methods.

Now the workarounds are - reflection caching, expression trees(but that requires compiling during runtime) or using some external tool for generation. Or using some kind of AOP like Fody , PostSharp... but it adds a dependency.

@Pzixel there is always room for improvement and plugins are constantly evolving.

@Pzixel
Copy link

Pzixel commented Mar 28, 2020

@bugproof rust macros are bad for two reasons

  1. they don't allow you to intoinspect types, they work on text level. So you can't write something like #[derive_via(MyTrait, Field(my_delegating_field)] that work for any trait, because you can't introinspect MyTrait's members
  2. they don't integrate well with IDE and they force recompile everything, because they are native. If they could be compiled into some IR then it could be much better. I discussed this matter with intellij-rust author so I think he's quite aware about macros drawbacks and ways of improvement.

I'd like to have macro system in C#, but LDM don't think so. This is why I probably will switch to scala when next major version comes. Partial evaluation, HKT, functional paradigm - all the nice stuff.

MS has different priorities for C#, sadly.

@bugproof

This comment has been minimized.

@Pzixel

This comment has been minimized.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Mar 29, 2020

@Pzixel @bugproof please move this discussion somewhere else. It is offtopic for this issue.

@charlesroddie
Copy link

charlesroddie commented Apr 15, 2020

https://github.com/dotnet/roslyn/blob/master/docs/features/source-generators.md
Generators produce one or more strings that represent C# source code to be added to the compilation.

It's great that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

  1. Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )
  2. Source generators can be used by any .Net lanuage which has made the necessary connections. If they generated IL it would be a more level playing field, and if they have to generate C# then as long as the C# is not somehow inside the referencing project then it might still work.

@CyrusNajmabadi
Copy link
Contributor

CyrusNajmabadi commented Apr 15, 2020

It's greate that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )

Isn't this just a description of IL... ?

If you want a non-C#-specific, language-independent, way to pass information into a compilation... isn't that just a dll?

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made

Isn't that IL?

Basically, why would we need a special feature for this since it's already the fully supported scenario?

@333fred
Copy link
Member

333fred commented Apr 16, 2020

but I wonder what is C#-specific about this?

@charlesroddie to be clear, this proposal is not being worked on. This proposal is replace/original, which adds new keywords to the language to support metaprogramming and AOP. The proposal being worked on is purely additive, and as you said, is not C# specific. It's so not C# specific, in fact, that it is not tied to C# 9, and we are considering it a compiler feature, not a language feature.

@Jose10go
Copy link

Jose10go commented Jul 4, 2020

I think #45648 is a good middle-way proposal between this proposal (original/replace) and the Source Generators proposal being implemented for C# 9.(https://github.com/dotnet/roslyn/issues/45648)

@sighoya
Copy link

sighoya commented Dec 16, 2020

I like source generators, but I'm not used to insert code over strings, that's ugly.

Maybe some limited form of dlang templates + String to macro expansion:

static foreach(String type in ["class","struct"])
    static foreach(String fieldName in ["a","b"])
         static foreach(String fieldType in ["Int,","Float"])
             $type {
              public $FieldType $fieldName;
             }

Generating 2^3 types. In contrast to Source generators, we only put those code fragments in strings which are changing and expand them by $ to valid C# code.
Or more safely, we could introduce new types for block kws, general identifiers and type names which can be checked against the provided strings by the user, s.t. only the "unknown" string will be red marked by the compiler/IDE.

static foreach(BlockKwString type in ["class","struct","unknown"])
    static foreach(IdentifierString fieldName in ["a","b"])
         static foreach(TypeString fieldType in ["Int,","Float"])
             $type {
              public $FieldType $fieldName;
             }

Also, mixins for code insertion would also be beneficial.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests