Visual Studio does not reload source generator assemblies when they change on disk

[vatsan-madhavan]

I'm just started writing a source generator, and I'm finding that Visual Studio is caching source generators aggressively, and it's making it very hard to do iterative development.

This is what I'm having to do to make even small changes.

• Close Visual Studio (devenv.exe) instance in which the source generator project and its consumer (i.e., the project that ProjectReference's the source generator)
• pskill /t servicehub.roslyncodeanalysisservice
• del $env:TEMP\VS\AnalyzerAssemblyLoader -Recurse -Force <- This is where the source generator binaries seem to get cached. Both devenv and servicehub.roslyncodeanalysisservice seems to hold handles to files under this location

Am I missing something simple to get the development process working more seamlessly ?

I'm on 5.0.100-rc.2.20473.20 + Visual Studio 2019 Enterprise 16.8.0 Preview 4.0 [30517.14.main

[vatsan-madhavan]

Is there a way to view the generated sources except through VS/intellisense ?

[Youssef1313]

@vatsan-madhavan

For the purpose of testing/debugging, I was viewing them with a simple hack (which isn't good, but did the purpose for me).

Youssef1313/PrintMembersGenerator@31e31ce

Then after things got stable with me, I moved to unit testing to confirm the correctness of the generated sources.

https://github.com/Youssef1313/PrintMembersGenerator/blob/master/src/PrintMembersGeneratorTests/PrintMembersGeneratorTest.cs

[CyrusNajmabadi]

Is there a way to view the generated sources except through VS/intellisense ?

As of yesterday, you can now emit generated files to disk so you can inspect them: #47047

[vatsan-madhavan]

This is awesome! This means I can get by with commandline builds while VS catches up!

[jasonmalinowski]

So the one problem @vatsan-madhavan you're probably running into is once we've loaded your assembly...the CLR doesn't give us sane ways to unload it or load a different version if the assembly version hasn't changed. @chsienki or @cartermp any chance somebody already has some MSBuild magic to work around this in some way?

[jaredpar]

Curious: if the generators in this case are strong name signed could we potentially manipulate the version here and load the new copy?

[jasonmalinowski]

@jaredpar Potentially. As crazy as the feature request is, I almost wish the compiler had a feature where it'd generate a (determinstic) but effectively random version and stuff that into the assembly version. 😄

[jaredpar]

Honestly this could be done as a simple post-build sttep. Load the binary in memory, use the metadata writer to flip the strong name bit, change the version and then load that vs. the one on disk.

[jasonmalinowski]

@jaredpar Thanks for volunteering!

[jaredpar]

The best part about being a lead, maybe the only good part, is the ability to delegate ... @chsienki

😉

[PathogenDavid]

I almost wish the compiler had a feature where it'd generate a (determinstic) but effectively random version and stuff that into the assembly version. 😄

The module version ID would be good for this. Just kludge it into the version number. (It's been a hot minute since I've looked into it, but if I remember right with determinism enabled it's the SHA1/SHA256 of all the compiler inputs or something along those lines.)

Right now specifying a wildcard for the assembly version results in CS8357 if determinism is enabled. Maybe determinism + wildcard could mean "kludge MVID into version number".

[MisinformedDNA]

Is the Roslyn team not going through the same pains as us?

[jaredpar]

We use source generators inside our main solution hence we are dogfooding the experience every day. The source generators aren't iterated on as frequently though hence we don't hit the specific reload problem.

This is a problem we are taking a look at. It's existed since Roslyn 1.0 with analyzers (so roughly five years now) hence it's not a new problem, generators has just shined a new light on it. There are some ideas on how to work around this (see my comments above). At the moment though Roslyn does most of the evaluation in process and given we are still on .NET Desktop that limits our options a bit because of the inability to load multiple copies of a DLL into the same process space unless it's strong name signed + changes versions on every build.

[tmat]

@PathogenDavid Using hash for version is not viable since it's not monotonic and versions are expected to be.

I'd suggest that instead of running source generator in VS when developing it it's better to run it in a unit test. Write a unit test that runs the generator and produces output. In that setting you can iterate fast - even using Edit and Continue to modify the generator code as you are debugging it. There is no need to mess with versions/reloading/VS complexity etc.

[tmat]

@jasonmalinowski

So the one problem @vatsan-madhavan you're probably running into is once we've loaded your assembly...the CLR doesn't give us sane ways to unload it or load a different version if the assembly version hasn't changed. @chsienki or @cartermp any chance somebody already has some MSBuild magic to work around this in some way?

The magic is called Core CLR ;-).

[PathogenDavid]

Using hash for version is not viable since it's not monotonic and versions are expected to be.

Fair enough for making that the default behavior of *, but for the purposes of this discussion the version number being monotonic doesn't matter.

I'd suggest that instead of running source generator in VS when developing it it's better to run it in a unit test.

This is essentially what I've been doing. (Except running dotnet build outside of Visual Studio with EmitCompilerGeneratedFiles enabled.)

The main frustration for me has been that once I have finished working on my source generator, convincing Visual Studio to relinquish whatever hidden cached source generator it is seemingly impossible to do reliably. At the very least I'd expect restarting Visual Studio to fix things, but even stopping all instances of Visual Studio, killing any lingering service hubs, deleting the cache folder named in the main issue, deleting all folders starting with vs in my temp folder, making a blood sacrifice, deleting my .vs folder, and finally restarting Visual Studio: My old generator still sometimes (somehow) sticks around. (But only sometimes.)

[CyrusNajmabadi]

i honestly wonder how the good people at MS doing that can stand all the waiting all the time.

I don't find myself ever waiting. I literally just right click the appropriate test folder in Test Explorer and say "run these tests". I pay the cost of the build, but then get results back in ms after that.

I could probably also just turn in 'live unit testing'. But I haven't found the existing process an issue. It feels just as fast as any other inner-loop dev process that I'm good with it.

[palpha]

which mention how to unit test these

Our test coverage goal is 100%, and any exclusions need to be explained. Like I said, how changes are driven is not really relevant to the discussion in this issue as I understand it.

I don't find myself ever waiting.

So, you never actually use the source generators you develop? :)

[CyrusNajmabadi]

When source generators were introduced, I saw it as a potential technique for metaprogramming in C#. If it by design is not, that's fine, but it needs to be communicated more clearly.

I believe the cookbook and our other docs cover this. SGs are a system for generating additional documents that will be part of the compilation. They will act the same as if you wrote those files manually. They are very costly (we have to compile twice), and can deeeeeeply impact IDE perf, so great care should be made to only use them sparingly, and to keep all work to an absolute minimum.

[CyrusNajmabadi]

So, you never actually use the source generators you develop? :)

I use all the things I develop. If I want to use it live in the VS host I'm in, building and launching a new instance works for me. That works once I've made an my changes and I'm satisfied (with tests) that they're doing the right thing.

[AdmiralSnyder]

@CyrusNajmabadi

I don't find myself ever waiting. I literally just right click the appropriate test folder in Test Explorer and say "run these tests". I pay the cost of the build, but then get results back in ms after that.

yeah, when you have a testing-based approach, sure. but if you're building a source generator as a tool for your application (as it is argued in this thread that people want to do it), that's not feasible... we discussed that over on the discord sufficiently, though.

[CyrusNajmabadi]

yeah, when you have a testing-based approach, sure.

This would be my strong recommendation. Like strong strong. It's blazingly fast, super tight development loop, and you end up with test assets (which means further work is even faster and more efficient as your cash make changes with confidence, and you discover issues immediately). I def recommend doing this as I think you'll find it fast (and liberating) tbh :-)

we discussed that over on the discord sufficiently, though.

Not sure I recall the discussion. I'd be surprised if I felt differently there though... Unit testing just seems like the way to go. All benefits, no drawbacks afaict.

[AdmiralSnyder]

Not sure I recall the discussion.
Now i'm sad - i thought we were friends :-P

Seriously, though, yep, you didn't feel differently. and since i ended up coming up with a working workaround, i didn't feel the need to steal more of your time.

All benefits, no drawbacks afaict.
Unless your actual solution is the environment you'd need to build unit test abstractions for - which would mean having to duplicate everything :-P

[CyrusNajmabadi]

Seriously, though, yep, you didn't feel differently.

Oh, glad i'm consistent at least. That would def be confusing otherwise :D

Unless your actual solution is the environment you'd need to build unit test abstractions for - which would mean having to duplicate everything :-P

Sorry... not exactly sure what you mean by this. (i tried reading it a few times, but i think it's not clicking). Can you clarify? Thanks!

[Bosch-Eli-Black]

I remember a Roslyn dev commenting in another issue that directly depending on a source generator project only works by accident, and shouldn't be considered supported.

The recommended development path from Microsoft seems to be:

1. Develop source generators via unit test
2. Ship and consume source generators as NuGet packages.
   • Don't directly link to a source generator project, even though this is possible.

If the above recommendations are followed, then VS never needs to be restarted.

That being said, I hope that taking a direct dependency on a source generator project is supported in the future! 🙂 For internal projects, having to package the source generator as a NuGet should be unnecessary 🙂

[Bosch-Eli-Black]

BTW, although @CyrusNajmabadi is happy developing source generators via unit test, and I totally respect that, the Roslyn team at large does seem to be interesting in fixing this issue. See #48083 (comment), #48083 (comment), and #48083 (comment)

I think the question is more one of timing (and I'm glad that this topic is generating a lot of noise lately 🙂)

[MisinformedDNA]

I believe we cover this in the docs, which mention how to unit test these.

Where? I see one GitHub issue there, but no official documentation.

This is the third-party documentation that finally made be understand (and love) unit testing source generators.

[CyrusNajmabadi]

Our SG guidelines: https://github.com/dotnet/roslyn/blob/main/docs/features/source-generators.cookbook.md

Unit testing is covered here: https://github.com/dotnet/roslyn/blob/main/docs/features/source-generators.cookbook.md#unit-testing-of-generators

[MisinformedDNA]

I think a core problem is that when users are experimenting with a new technology, they tend to skip unit tests to get a feel for it (if they ever write unit tests at all). They quickly experience issues with IntelliSense or whatever and end up here. If we're lucky, they actually read this thread. Otherwise, they leave their 2 cents and wait for VS to be "fixed".

What would help?

1. A source generator project template (like the analyzer and code fix templates) with example unit tests. And within that a README that makes the importance of unit testing very clear and the potential consequences (and a link to the cookbook).
2. All other documentation should likewise stress the importance and consequences. That would include the cookbook, MS Learn, videos, etc.

[frudman]

I believe we cover this in the docs

Guess that's the problem! You don't realize how incredibly obtuse [complex, difficult, disorganized] the docs are!!! Sure, they may be "complete" (in someone's mind) but they sure don't make for easy reading.
Just to reiterate what's been said before (by others) some of us (many?) are not in the (your?) business of creating generators. We're merely using them as tools to solve a larger problem. If the tool becomes a barrier, then it's a useless tool (for us). Or if the tool is so difficult to use (because of, say, poor docs) then that's the same result.

[CyrusNajmabadi]

You don't realize how incredibly obtuse [complex, difficult, disorganized] the docs are

PRs welcome :-)

Or if the tool is so difficult to use

Generators are hard. But their very nature. They're letting you plug into the core of the compiler. But that must be done with extreme care as this impacts literally everything what in the system.

The fish of generators was to make this possible. But make no mistake, it's not simple precisely because this domain is itself enormously complex and hard to do right.

If you have ideas on how to make it simple, def feel free to open prs on that. We'd absolutely look.

[frudman]

I'm not sure I agree with your characterization of generators: they may be hard to *implement* (your side, or at least implement so they run efficiently) but *they're not that hard to use*. v1 generators were particularly easy (walking through syntax trees, for example). v2, incremental generators (which I'm now using) are a little more complex (a slightly steeper learning curve) but once you get the hang of it, they're also pretty straightforward (as evidenced by a generator I'm working on now). What I was referring to is the DOCUMENTATION. And, to answer your question, for the very (very very) many (many many) pages I had to read to understand (in this case) incremental generators, the one that FINALLY did the trick, buried somewhere inside those inter-tubes was this one: https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md This single page should be splashed ANYWHERE someone searches code generators. And the rest of the docs should be written with that level of explanation. That page is a great start (not perfect, but it got me going, and i'm pretty dense). And as for improving generators, maybe *start by making easy stuff easy*!!! A trivial task (for example) is customizing the generator's code generation (e.g. changing method names, adding/removing logging). And the most obvious, easiest way to do this is simply let users/consumers define custom props in their .csproj files (as propertygroup) that can then be read by the generator on execution. Wow, that's NOT happening. Not only are you making us go through hoops just to read this (via, e.g., context.AnalyzerConfigOptionsProvider.Select(...)) but then you won't even let us read our actual values UNLESS they're also SPECIFICALLY identified in that same file (using <ItemGroup><CompilerVisibleProperty.../>) WHY so difficult? And, more importantly, WHERE ARE THE DOCS for this??? Had to stumble on multiple posts to finally figure it out, and still had difficulties with whether to put these directives in the generator project or the user project (it's the user project, btw :-)). My point is, this has nothing to do with the implementation of generators. It's simply about the documentation of them. And making some better(?) choices about how to implement some of these features (e.g. why not simply a context.ProjectProps.Get() or something). Or another one is how to attach a 3rdparty lib to the generator without it being then packaged to the consuming project's output. Again, ridiculously hard (I actually gave up on that one, even though I got 99% there, couldn't resolve the "lib not found" errors, or lib conflicts errors, even though it listed (both?) libs as exactly the same versions). Hmmm... I think generators are an incredible addition to a dev's toolkit and I just get a little frustrated with using them. Sorry. Freddy. ps: thanks for taking the time to reply!

…

On Sat, Jan 28, 2023 at 11:00 AM CyrusNajmabadi ***@***.***> wrote: You don't realize how incredibly obtuse [complex, difficult, disorganized] the docs are PRs welcome :-) Or if the tool is so difficult to use Generators are hard. But their very nature. They're letting you plug into the core of the compiler. But that must be done with extreme care as this impacts literally everything what in the system. The fish of generators was to make this possible. But make no mistake, it's not simple precisely because this domain is itself enormously complex and hard to do right. If you have ideas on how to make it simple, def feel free to open prs on that. We'd absolutely look. — Reply to this email directly, view it on GitHub <#48083 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJLIRLOFBQNYZSKK5DVYJTWUU7BVANCNFSM4R3DC3DA> . You are receiving this because you commented.Message ID: ***@***.***>

-- Frederic Rudman, CTO Enervision Media Inc. <http://www.EnervisionMedia.com> 877.330.3077 ext 21

[CyrusNajmabadi]

but they're not that hard to use. v1 generators were particularly easy (walking through syntax trees, for example)

That's a good example of their difficulty. Doing things this way was so detrimental that we're now deprecating the v1 API since it's core design is effectively unsalvageable.

As I stated before. This stuff is fundamentally hard. And apis that attempt to mask that are a strict mistake since you get the worst if all worlds.

This single page should be splashed ANYWHERE someone searches code
generators.

Honestly, we should plaster a page first that takes every effort to convince people to buy use generators and to explain why they're a poor choice for most use cases. We should then be clear about the narrow set of cases it is good for, and implore users to not use them unless they fall into that narrow slice of things.

[CyrusNajmabadi]

And as for improving generators, maybe start by making easy stuff easy!!!

There are no easy cases here. That's my point. :-)

and I just get a little frustrated with using them. Sorry.

This is all open source. If you want your scenarios to improve, the best thing would be to invest in the improvements there. That said, as mentioned before, these are expert, extremely complex systems (think double black diamond level) intended for niche scenarios where no other mechanics are viable. They are not a catch all system to do basic code gen.

You are literally hooking the compiler at the deepest level. You get to run prior to us even getting the compilation value that powers literally everything higher up. With that great power comes great responsibility. It's difficult by it's very nature, and will continue to be that way.

[Bosch-Eli-Black]

For everyone saying that the docs need improving: I've just filed dotnet/docs#33936 , which asks that part of the Source Generators Cookbook be migrated to official documentation at https://learn.microsoft.com/

If you feel the same, please upvote that suggestion! 🙂

[AdmiralSnyder]

@palpha
I completely forgot that i had already released this extension which makes writing source generators with vs and not needing to restart VS all the time a blast:
https://marketplace.visualstudio.com/items?itemName=AlexanderGayko.AutoUpdateAssemblyName

[AdmiralSnyder]

@Bosch-Eli-Black see above :-)

[AdmiralSnyder]

Hey @CyrusNajmabadi - sorry for the delay, Uni had prio.

Unless your actual solution is the environment you'd need to build unit test abstractions for - which would mean having to duplicate everything :-P

Sorry... not exactly sure what you mean by this. (i tried reading it a few times, but i think it's not clicking). Can you clarify? Thanks!

in my case i build source generators that auto-generate lots of boiler plate stuff for our codebase - stuff for our ORM, mixin stuff for my imports, different places, different needs. and the usecases for these code generations are not giving my generators to others so they can do that, too, but helping our codebase - which changes, naturally, over time. so when my codebase changes, obviously, my generators will have to change, too.
If i'd be using the unit test based approach for writing those source generators, i'd not only have to abstract my codebase into test cases, i'd also have to modify those test cases whenever our codebase changes, too. which is just hard, unnecessary work that i want to avoid :-)

[TWhidden]

@palpha I completely forgot that i had already released this extension which makes writing source generators with vs and not needing to restart VS all the time a blast: https://marketplace.visualstudio.com/items?itemName=AlexanderGayko.AutoUpdateAssemblyName

Dude - I hate that I love this - Works / Thanks!

[sharwell]

If i'd be using the unit test based approach for writing those source generators ... i'd also have to modify those test cases whenever our codebase changes ...

This isn't necessarily the case. Here's a source generator test suite that fully automates the creation and maintenance of expected outputs (via the WRITE_EXPECTED directive in one of the source files):
https://github.com/microsoft/vs-extension-testing/tree/main/src/Microsoft.VisualStudio.Extensibility.Testing.SourceGenerator.UnitTests

[teneko]

I remember two times where I wanted to implement a source generator, that I stopped, because VS/Roslyn denied to accept code changes towards my source generator and had therefore always to restart VS. Thanks @palpha for the extension, it works, but I am just curious: why your names from your VS extension page and your GitHub page differ when you claim that it is your extension? Maybe out of scope of this issue.

EDIT: @palpha any vision to make the source code openly available? Because on the extension page it says that there some features worked on that I would like to help out with.

[palpha]

when you claim that it is your extension

I have never claimed this, @teneko. Read again and you'll find the author.

Open-sourcing seems like a reasonable suggestion.

[Bosch-Eli-Black]

@teneko, The author is @AdmiralSnyder 🙂 He was replying to @palpha, which is why his name is in that quote 🙂

[AdmiralSnyder]

hey, yeah, open-sourcing it would be possible, but i have still not decided whether i want at some point make a living by building extensions commercially, and in that case, it wouldn't be that wise to open source it - so i think providing the extensions for free for now is the best i can do :-)

@sharwell i didn't look at the state of generators for a while, so if there's better approaches, that's great, thank you.

@TWhidden thank you so much

@teneko i decided to go with my real name for my extensions (i.e. my vsix marketplace profile) because: see above :-) but, for help with features, just ping me over on the c# discord server (I'm AdmSnyder there)