Compiler pipeline extension API

[Inverness]

This is a request for a simple API that allows a class to be hooked into the compilation pipeline. At minimum, after the point syntax trees are parsed but before the compilation object is created.

I recently had to make a fork to introduce a minimalistic implementation of this because I was interested in aspect-oriented programming of the type PostSharp provides but found no where to actually step into the compilation process.

An important thing to note here is that I did this all with Visual Studio 2013. I built my version of the compiler, added it to a local directory, configured it to load the extension assembly, and finally edited my project file to change CscToolPath and CscToolExe.

My own implementation:

public class CompilerExtension
{
    public virtual void OnSyntaxTreesParsed(IList<SyntaxTree> trees)
    {
    }
}

Syntax trees could be freely added or removed from the provided list. An ExportCompilerExtension attribute would exist too for identification. This is the extent of what I'm proposing.

I'm forced to use the syntax tree API for this because nothing else is public. It is not ideal. There is no type checking and declaring aspect usage is more verbose since the aspect implementation type and any options must be specified in attributes. Care must also be taken to introduce line directives to keep the debugger from becoming confused. I was successful in implementing a rewriter for a method boundary aspect, but its not something I would want to take beyond the proof-of-concept stage since I do not like that it requires a custom compiler or working at the syntax level much.

I understand the bound tree level would be a much better place for this sort of work, but unfortunately it is not public, which is quite disappointing.

[svick]

I'm forced to use the syntax tree API for this because nothing else is public.

SemanticModel is public, isn't that what you want?

[Inverness]

SemanticModel is not mutable, and changes need to be made before the syntax trees are compiled since the compilation process is full of internal code.

[ghost]

It would be extremely useful. Please consider adding of this functionality.

[mattwarren]

I'd be interested in having this functionality as well, can anyone from Microsoft comment on whether or not it is being considered?

I'm currently working on a benchmarking project where I have to use Post-Build steps to re-compile a project (using Roslyn) after VS has already compiled it. It would be much nicer if I could hook into the VS/Roslyn compilation pipeline and make my changes there. Also it seems like there are other scenarios in which this functionality could be useful.

For what it's worth, I like the [Attribute] approach (outlined above), that gives you the ability to alter the SyntaxTree for a given piece of code that has an attribute applied. But I'd also like it to go further, so that you could also add extra SyntaxTrees (i.e. new classes) to the project being compiled.

BTW Java has this functionality, it's called Annotation Processors (also see this blog). But it's more basic as the Java compiler doesn't have as many features as Roslyn, so you have to work at the source code level.

[Inverness]

It would be nice to have some sort of update on Microsoft's opinion of this.

@mattwarren I'd just like to clarify something. The approach I proposed doesn't seem to match what you've said. My proposal was for a more general compiler extension that would receive all syntax trees that have been parsed and be able to change them or optionally add new ones to the list (as you want to do). This does not target a specific piece of code with an attribute applied.

Here's an outline:

1. The compiler is invoked with a command line argument specifying the paths of extension assemblies that will be used when compiling the specified source files.
2. These extension assemblies are loaded, and an instance of each class with an ExportCompilerExtensionAttribute is created and stored. With CSC2 these are only loaded once.
3. All source files have their syntax trees parsed.
4. Each compiler extension class has their OnSyntaxTreesParsed() method invoked with the list of parsed syntax trees. Syntax trees can be added to or removed from the list. Null entries are ignored. With CSC2 you must not give the syntax trees to all loaded extensions, but only the ones specified for a particular build.
5. A compilation unit is created from the final syntax trees and the rest of the compiler pipeline occurs.

[mattwar]

This functionality was considered. The possibility of doing something in this space was one of the original motivations for doing the new compilers. It is still a possibility for future work.

[mattwarren]

@mattwar thanks for the update (from one Matt Warren to another). It's good to know that it's being considered for the future, I'll look out for any updates

@Inverness I think we're talking about the same thing. I was just wondering if being able to link it with an attribute might be more useful for scenarios like INotifyPropertyChanged. I.e. if you we're to create a INotifyPropertyChangedRewriter, I could then include it in the Compile Pipeline for my projects, but only have it applied to methods that have an [NotifyPropertyChanged] attribute (for instance).

[Inverness]

@mattwarren Having access to the syntax tree means you can easily search through it for the attribute you want to perform an action on its object.

What I did for my PoC was create a CSharpSyntaxRewriter subclass that applied the changes I want:

    [ExportCompilerExtension(LanguageNames.CSharp)]
    public class AspectsCompilerExtension : CompilerExtension
    {
        public override void OnSyntaxTreesParsed(IList<SyntaxTree> trees)
        {
            var aspectRewriter = new AspectSyntaxRewriter();

            for (int i = 0; i < trees.Count; i++)
            {
                if (trees[i] == null)
                    continue;

                SyntaxNode root = trees[i].GetRoot();

                root = aspectRewriter.Visit(root);

                trees[i] = trees[i].WithRootAndOptions(root, trees[i].Options);
            }
        }
    }

[mattwarren]

@Inverness yeah that's a good point, I didn't think about that.

[Inverness]

Well, one advantage of the new compiler design is that an extensible fork can be easily distributed as a NuGet package that sets up your project(s) to build with it.

[simonegli8]

It would be nice, if one could just extend the compiler via hooks by importing a namespace or placing an attribute on a class.
With the namespace approach, the compiler would check the dll that contains the namespace for compiler extension classes and hook them into roslyn.
Places to extend the compiler could be preprocessing sourcecode text & syntaxtree postprocessing.

[ymassad]

Are there any updates on this? Is this related to Code Generators?

[Jishun]

This is a very useful feature that I've been hoping to get , though, the upvotes are so few..

[jmarolf]

The source generators feature that is shipping in November should address some of the use cases here

[0x0737]

This is the feature I was thinking of a long time. Much more useful than source generators... It could open tons of new possibilities given that all changes to syntax tree are reflected back to the ide, like java does with annotation processing.
Though I don't know for sure how it works in java, but all changes that lombok does to source code are reflected in a form of hidden identifiers

[CyrusNajmabadi]

This would need to go through an API proposal. in general, the reason we don't just make things public is because it increases maintenance costs and locks down our ability to change things in teh future. We can open things up, but we have a process that requests need to go through to make sure the appropriate people weigh in and the right API shape is determined and shipped. Importantly, any extension points need strong evidence that they can be provided in ways that do not impact scalability, and can have good debugging/editing experiences around them.