Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Champion "Replace/original and code generation extensions" #107

Open
gafter opened this issue Feb 14, 2017 · 315 comments
Open

Champion "Replace/original and code generation extensions" #107

gafter opened this issue Feb 14, 2017 · 315 comments

Comments

@gafter
Copy link
Member

@gafter gafter commented Feb 14, 2017

  • Proposal added
  • Discussed in LDM
  • Decision in LDM
  • Finalized (done, rejected, inactive)
  • Spec'ed

See also

@gafter gafter changed the title Replace/original and code generation extensions Champion Replace/original and code generation extensions Feb 15, 2017
@gafter gafter changed the title Champion Replace/original and code generation extensions Champion "Replace/original and code generation extensions" Feb 21, 2017
@gafter gafter added this to the X.0 candidate milestone Feb 22, 2017
@MgSam
Copy link

@MgSam MgSam commented Feb 28, 2017

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

@asdfgasdfsafgsdfa
Copy link

@asdfgasdfsafgsdfa asdfgasdfsafgsdfa commented Mar 5, 2017

No news on this?
Would be really helpful for a ton of things...

@MgSam
Copy link

@MgSam MgSam commented Mar 24, 2017

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

@asdfgasdfsafgsdfa
Copy link

@asdfgasdfsafgsdfa asdfgasdfsafgsdfa commented Mar 24, 2017

Sorry if this is a dumb question but what exactly does tooling mean here?
The "debugger UI" in Visual Studio?
What parts would need to get changed?
Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

  1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
  2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
  3. Debugging. How do you debug through the code that is generated.
  4. Project-System. How do you present generated code in the project system?
  5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

@eyalsk
Copy link
Contributor

@eyalsk eyalsk commented Mar 24, 2017

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

I'm not sure i understand your question @eyalsk . Can you clarify?

@eyalsk
Copy link
Contributor

@eyalsk eyalsk commented Mar 24, 2017

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

The owners of those respective products would have to make a decision on that. :)

@eyalsk
Copy link
Contributor

@eyalsk eyalsk commented Mar 24, 2017

@CyrusNajmabadi Got'cha, thanks. 😉

@MgSam
Copy link

@MgSam MgSam commented Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

@MgSam
Copy link

@MgSam MgSam commented Mar 24, 2017

What doesn't make sense is the arbitrarily different bar for different MS products:

  • T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
  • As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics

It's not internal politics. We are the C# team. We decide what we feel is best for the language. And that includes insuring that it has a great development experience out of the box.

And there's a good reason for that. We don't want to ship the language changes too early, only to find out later that we royally screwed things up once we try to then get proper IDE support for it. SourceGenerators are a massively complex undertaking. We could try to get something out without considering and attempting to prove it would work in the IDE. But then it's highly likely we would get it very wrong.

Even our preliminary efforts here showed us immediately how our early source-generator designs simply would not be suitable to providing a good IDE experience.

@MgSam
Copy link

@MgSam MgSam commented Mar 24, 2017

And that includes insuring that it has a great development experience out of the box.

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

It's also ironic that you're the one making this argument, Cyrus, when not long ago you were on the TypeScript team helping to ship new features every couple months.

@eyalsk
Copy link
Contributor

@eyalsk eyalsk commented Mar 24, 2017

@MgSam I agree with you on some of your points but I think that they probably don't want to make the same mistake twice and the experience I had with T4 from what I can remember was pretty awful to the point I removed the extension I wrote from the VS gallery. :)

@eyalsk
Copy link
Contributor

@eyalsk eyalsk commented Mar 24, 2017

@MgSam

As you say this you guys have just finished shipping a feature which is functionally broken out-of-the-box - tuples. It requires you to know that you need to go to NuGet and search for a specific package in order to use it. The VS feature which suggest packages to download is turned off by default.

Didn't they fix this for RTM?

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

Dustin was demoing replace/original last year, so clearly some form of it is possible to ship without the sky falling.

Yes. That was a demo. It was woefully incomplete. It worked in extraordinarily small scenarios. It had serious issue that we did not (and still do not) have solutions for real projects. We do not ship prototypes just because they're cool and they demo'ed well. A lot of real effort went into exploring this feature. Even just the language feature alone had a whole host of issues that we did not know how to resolve. And that was before the very first problem of: how will this even hook up into the IDE compilation flow model?

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 24, 2017

Didn't they fix this for RTM?

No. But we're working on it. The core problem is that there have been extremely hard perf goals to meet, such that even loading a single extra dll is not allowed. That's problematic in the context of any sort of extensibility story. We made a hard decision to ship in that state, even though we would have preferred it be better.**

And that goes to my point. That was something we did not want to do, and it was only because the scope of the problem was so limited. i.e. you would get an error about tuples, but at least you could solve it.

That doesn't even begin to compare to the issues you'd get with source generators. Again, imagine it was like tuples, and you got lots of errors in your IDE while trying to work. Except now there wouldn't be anything you could do about that. Also consider that just typing things like "replace" and whatnot would simply throw of the IDE parser completely, leading to code that was nearly impossible to edit.

Seriously, if you want to get a feel for what it would be like, go back to something like 2013 and try to type modern C# in it and see how quickly the IDE experience goes off the rails. Just basic editing will fall to the floor because we won't understand teh code you're writing on a syntactic and semantic level. And, currently, our IDE is based around the idea that we will understand at least that much.

--

** We've won back enough performance across the board though that we feel lke we can get an exception here and that this feature can be enabled soon. If/when that happens, it will mean there was a short window of time when we had a not-great experience. That's not at all comparable to the types of timeframes we're talking about for something like SourceGenerators.

And again, it's not like we can "just ship source generators now and just make the IDE work later". We have to understand the needs of hte IDE so that we can properly design source generators in the first place. We know this first hand from other work we've done in the past, including work in analyzers. The original analyzer work did not think closely enough about the compiler/IDE interaction, and as such had absolutely unacceptable performance characteristics. By designing things together, we were able to come up with something that met our feature goals while also having acceptable performance.

@LokiMidgard
Copy link
Contributor

@LokiMidgard LokiMidgard commented Oct 10, 2019

@Pzixel Why should it not allow derive interfaces or add mebers? (As long as the changed class is partial)

From the name I would assume it would only ommit fancy and relay helpfull Keywords. Unless you have found mor information then the title.

Of course it would be nice if you wouldn't need to make those classes partial. But if ommitting this will bring code generation earlier, I'm fine with it.

I already use source generation today in form of scripts or small programms that that run in preBuild or before CoreCompile. However this is kind of quirky today and offten runs in problems. Having an offical supported way to use code generation can make it only better.

@Pzixel
Copy link

@Pzixel Pzixel commented Oct 10, 2019

For some reason I thought it cannot add new members. But now I reread the docs

To add members to an existing class, the generated source will define a partial class. In C# this means the original class definition must be defined as partial.

Adding partial keyword is a minor inconvenience. If it works then it's great.


In the end, I'd like to have seeing something like

[Derive(typeof(IComparable<>), typeof(IEquitable<>)]
public partial class Point {
    public int X { get; set; }
    public int Y { get; set; }
}

...

var comparision = new Point(10, 20).CompareTo(new Point(20,3));
@HaloFour
Copy link
Contributor

@HaloFour HaloFour commented Oct 10, 2019

IMO doing it without the language changes is probably more about proving out the tooling support of the feature first. I would assume that adding new members or types would cause much more impact on the tooling support than replace/original would as the IDE needs to discover those types/members in order to offer them via autocomplete.

@Pzixel
Copy link

@Pzixel Pzixel commented Oct 10, 2019

Yes, that's what I meant when saying "I understand IDE concerns". But without it it's just a toy feature "look, we can inject logging in the beggining and the end of the method. Woah". IMHO, of course.

@vbcodec
Copy link

@vbcodec vbcodec commented Mar 4, 2020

https://github.com/dotnet/roslyn/projects/54
What does this means ? Seems like wish list. Hard to say if this is serious commitment or another prototypical attempt.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 4, 2020

Wouldn't trying to prototype this indicate a serious commitment?

@vbcodec
Copy link

@vbcodec vbcodec commented Mar 5, 2020

Actually not. You may build prototype after prototype, but without serious commitment, never able to push it through RTM phase.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 5, 2020

We're not going to be able to make this happen without prototyping it. It's a challenging and complex space. I cannot be designed in absence as this is really about a tooling feature (like analyzers), and much less of a language feature.

MS is currently paying people to do this and allocating the time to make this happen, ahead of a long list of other with that could be fine instead. If you don't see that as commitment, then I don't know what to tell you.

@jaredpar
Copy link
Member

@jaredpar jaredpar commented Mar 5, 2020

https://github.com/dotnet/roslyn/projects/54

This is the GitHub project used to track our upcoming source generator work. The aspiration is to ship this as a part of the C# 9.0 release.

You may build prototype after prototype, but without serious commitment, never able to push it through RTM phase.

As @CyrusNajmabadi noted this is a complex area and one where we feel strongly that we need direct customer feedback to ensure we deliver the correct solution. Delivering a regular stream of prototypes through the C# 9.0 release where we can iterate with those customers is the best way to get that feedback.

If you don't see that as commitment, then I don't know what to tell you.

+100.

@bigdnf
Copy link

@bigdnf bigdnf commented Mar 5, 2020

Maybe you can join forces with https://github.com/AArnott/CodeGeneration.Roslyn

@agocke
Copy link
Contributor

@agocke agocke commented Mar 5, 2020

@AArnott Is certainly welcome to weigh in

@AArnott
Copy link
Contributor

@AArnott AArnott commented Mar 6, 2020

I'm stoked about the C# 9 built-in feature. I'd love to try a prototype.

@HamedFathi
Copy link

@HamedFathi HamedFathi commented Mar 22, 2020

About the source generator proposal, Can it brings Rust`s macro and procedural macro as a new game-changer feature into C#?

https://doc.rust-lang.org/rust-by-example/macros.html
https://medium.com/@phoomparin/a-beginners-guide-to-rust-macros-5c75594498f1

Example: Implementing Ruby’s HashMap syntax in Rust

a

(Procedural) Macros in Rust is so powerful, We can generate source codes with custom DSLs, syntax and much more.
Any idea?

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 22, 2020

Custom DSLs are very intentionally not in scope for this.

@Pzixel
Copy link

@Pzixel Pzixel commented Mar 22, 2020

@HamedFathi rust macros are known for poor IDE integration, especially procedural ones. I don't think it worth taking them as they are even if LDM was interested in it, which they aren't.

@bugproof
Copy link

@bugproof bugproof commented Mar 28, 2020

I really like how macros work in Rust. So things like Serde (mapping to struct directly ) are possible without using reflection.

This would change completely the way you can create your serializer and it would improve performance a lot. You could generate your mapping/serialization methods.

Now the workarounds are - reflection caching, expression trees(but that requires compiling during runtime) or using some external tool for generation. Or using some kind of AOP like Fody , PostSharp... but it adds a dependency.

@Pzixel there is always room for improvement and plugins are constantly evolving.

@Pzixel
Copy link

@Pzixel Pzixel commented Mar 28, 2020

@bugproof rust macros are bad for two reasons

  1. they don't allow you to intoinspect types, they work on text level. So you can't write something like #[derive_via(MyTrait, Field(my_delegating_field)] that work for any trait, because you can't introinspect MyTrait's members
  2. they don't integrate well with IDE and they force recompile everything, because they are native. If they could be compiled into some IR then it could be much better. I discussed this matter with intellij-rust author so I think he's quite aware about macros drawbacks and ways of improvement.

I'd like to have macro system in C#, but LDM don't think so. This is why I probably will switch to scala when next major version comes. Partial evaluation, HKT, functional paradigm - all the nice stuff.

MS has different priorities for C#, sadly.

@bugproof

This comment was marked as off-topic.

@Pzixel

This comment was marked as off-topic.

@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Mar 29, 2020

@Pzixel @bugproof please move this discussion somewhere else. It is offtopic for this issue.

@charlesroddie
Copy link

@charlesroddie charlesroddie commented Apr 15, 2020

https://github.com/dotnet/roslyn/blob/master/docs/features/source-generators.md
Generators produce one or more strings that represent C# source code to be added to the compilation.

It's great that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

  1. Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )
  2. Source generators can be used by any .Net lanuage which has made the necessary connections. If they generated IL it would be a more level playing field, and if they have to generate C# then as long as the C# is not somehow inside the referencing project then it might still work.
@CyrusNajmabadi
Copy link
Contributor

@CyrusNajmabadi CyrusNajmabadi commented Apr 15, 2020

It's greate that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )

Isn't this just a description of IL... ?

If you want a non-C#-specific, language-independent, way to pass information into a compilation... isn't that just a dll?

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made

Isn't that IL?

Basically, why would we need a special feature for this since it's already the fully supported scenario?

@333fred
Copy link
Member

@333fred 333fred commented Apr 16, 2020

but I wonder what is C#-specific about this?

@charlesroddie to be clear, this proposal is not being worked on. This proposal is replace/original, which adds new keywords to the language to support metaprogramming and AOP. The proposal being worked on is purely additive, and as you said, is not C# specific. It's so not C# specific, in fact, that it is not tied to C# 9, and we are considering it a compiler feature, not a language feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.