SourceGenerators Proposal: PreExecution Source Addition

[chsienki]

PreExecution Source Addition

It has been shown to be a common problem that Source Generators want to both add source to a compilation, and simultaneously make semantic decisions about that source in the same pass.
The obvious use case of this is adding an attribute to the users compilation, then finding usages of said attribute.

Today, the user is forced to first append the source themselves to the compilation, creating a new compilation, and obtain a SemanticModel from the new compilation to ask questions about. We've seen this not only is customer code, but in our own samples too.

This is not only inconvenient and confusing for the source generator author, but also defeats other performance goals we would like to achieve, especially around semantic model sharing.

Proposed Solution

As part of Initialization we would provide a new callback RegisterForPreExecution that would allow a generator to inform the host it would like to provide sources prior to execution.
These would be appended to the compilation before the Execute method is called, allowing the generator to make semantic decisions about the added source.

In order to facilitate semantic model sharing in the future, the pre execution sources from each generator would be collected first, and appended to the compilation as a whole; this will mean generators will be able to see the pre execution sources of other generators during execution.

It would be up to the host to decide the granularity of the invocation of the callback to RegisterForPreExecution. In the command line compiler, it would be called exactly once, directly after Initialize. The IDE would be free to adopt the same policy, or discard the results as memory pressure requires and call it again in the future.

Because of this, the compilation itself would not be available to the callback, but it is likely we would want to make available the ParseOptions and AnalyzerConfigOptionsProvider so that the generator can ensure any added source will be compatible with the eventual compilation.

The added sources will not be visited by any SyntaxReceivers or future semantic receivers (as they are not part of the users compilation), but they would be part of the compilation that the receiver is operating on. This will be invisible to a SyntaxRecevier but would mean any semantic models made available as part of a future semantic walker would have the added sources available for semantic lookup.

Implementation

public readonly struct GeneratorInitializationContext
{
    public void RegisterForPreExecution(Action<GeneratorPreExecutionContext> callback);
}

public readonly struct GeneratorPreExecutionContext
{
    public void AddSource(string fileNameHint, SourceText sourceText) { ... }

    public ParseOptions ParseOptions { get; }

    public CancellationToken CancellationToken { get; }

    public AnalyzerConfigOptionsProvider AnalyzerConfigOptions { get; }

    // do we need this? It seems useful, but not required
    public void ReportDiagnostic(Diagnostic diagnostic) { ... }

    // also this
    public ImmutableArray<AdditionalText> AdditionalFiles { get; }
}

Example

[Generator]
public class AutoNotifyGenerator : ISourceGenerator
{
    private const string attributeText = @"
using System;
namespace AutoNotify
{
    [AttributeUsage(AttributeTargets.Field, Inherited = false, AllowMultiple = false)]
    sealed class AutoNotifyAttribute : Attribute
    {
        public AutoNotifyAttribute()
        {
        }
        public string PropertyName { get; set; }
    }
}
";

    public void Initialize(GeneratorInitializationContext context)
    {
        context.RegisterForPreExecution(PreExecute);
    }

    public void PreExecute(GeneratorPreExecutionContext context)
    {
        // add the attribute text
        context.AddSource("AutoNotifyAttribute", SourceText.From(attributeText, Encoding.UTF8));
    }

    public void Execute(GeneratorExecutionContext context)
    {
        // get the added attribute
        INamedTypeSymbol attributeSymbol = compilation.GetTypeByMetadataName("AutoNotify.AutoNotifyAttribute");

        // ...
    }
}

Alternative Solution

We could just add an AddSource method to the InitializationContext rather than requiring a callback. However, initialize has a strong guarantee that it will be called exactly once. This would therefore require the host to retain any source for the lifetime of the generator, without the option to drop it and re-acquire it later on.

Issues / Open Questions

• Allowing a generator to add source to the compilation before execution, by definition 'leaks' the existence of this generator to other generators in the compilation, as they are able to
  see the added source as part of the compilation. We could potentially create a 'shadow' Compilation that we pass to the generator that excludes the added source of other generators, but this seems likely to be error prone and probably unnecessary.

• Should we allow access to the additional files and report diagnostics?

• What should we call this phase? PreExecution is fine, but there are probably better alternatives.

[pakrym]

I hit this problem as well. The https://github.com/pakrym/jab/ tries to add the internal types used in compiled code.

Today, the user is forced to first append the source themselves to the compilation, creating a new compilation, and obtain a SemanticModel from the new compilation to ask questions about. We've seen this not only is customer code, but in our own samples too.

If the code you are trying to add is static you can use the nuget contentFiles feature instead https://github.com/pakrym/jab/blob/main/src/Jab/Jab.csproj#L24.