Compiler plugin design

[heshanpadmasiri]

Objectives

• Allow libraries to mutate the AST (based on annotations) and provide custom diagnostics.

Non-objectives

• Extend the language/frontend to support metaprogramming or compile-time code execution.

---

In jBallerina, we used SPI-based dynamic loading of code provided by a module, allowing it to perform AST mutations and provide diagnostics. While Go also has the ability to load dynamic libraries, this is only supported on Unix systems (ref). Thus, we can't use this on Windows or in the browser.

With this in mind, I think there are three options to consider.

1. Statically linked Go functions for a given library.

   Each library that needs this support can declare a function that takes a compiler context and ast.Pkg and returns an ast.Pkg. Using the compiler context, it can provide diagnostics; since it returns the AST, it can perform arbitrary mutations on the AST. Essentially, this is very similar to how desugar is implemented (and most inline with how current jBallerina compiler plugins work). For the initial release, given that we only allow this capability for packages we pack in the distribution, such as http, this should be straightforward. Each package that needs this can register the function with the project API, and if the package has an import, it will call the registered function before desugar.

2. Define these transforms using a restricted subset of Ballerina.

   Each of these packages can define a .bal file with a specific name (for example, build.bal; however, user code could already have such a file, so maybe this needs to be in a specific path, such as <project-root>/build/build.bal) that supports only a subset of the language library (and perhaps only a subset of the language, so we can avoid the need to desugar it). We will expose the AST and compiler context via extern functions. Then, after generating the package AST for user code (sans build.bal), we'll pass the package AST as an argument to build.bal and interpret it.

Important

Ideally, I don't want to expose the AST and compiler context as data to the Ballerina side. This would introduce a lot of complexity in syncing data structures between the Go and Ballerina sides. Instead, I would like to expose mutation/data-access functions and handles.

3. Introduce a DSL for representing AST transforms and validations. After desugar, the frontend interprets the AST using the given DSL.

Of these options, I don't think 3 gives us anything that 2 can't give us, other than perhaps cleaner syntax. I gravitate towards starting with 1; given that, for some time, we only need to care about libraries we pack into the distribution itself, and with time, we can move towards 2. This should reduce the time we need to get an initial working version (1 needs very little work from both the project API and frontend), and later, when we do 2, we can have a very clear idea of the minimal set of APIs we need to support.

[warunalakshitha]

I believe introducing a DSL for compiler plugins on top of the Ballerina language adds unnecessary complexity.

I'm also +1 on starting with Option 1 (statically linked). It will allow us to better understand the real-world use cases and requirements for compiler plugins, which will ultimately help shape the design of Ballerina transform functions.

For Option 2 (plugin path), I suggest using the following project structure:

/compiler-plugin/<plugin_name>.bal

[sameerajayasoma]

In jBallerina, compiler plugins allow the following three tasks:

• Analyze user code and report diagnostics
• Generate new code
• Transform user code

Based on these use cases, we came up with three kinds of compiler plugins in jBallerina:

• Code analyzers
• Code generators
• Code transformers

You can find more details in the following links:

• Ballerina compiler plugin design (30,000-foot view)
• Ballerina compiler plugins
• [Ballerina compiler/platform extension design] (https://docs.google.com/document/d/1jXmFJi55RtPZYACppZ_-ikGEknXYZ7DqLuPAjT3NLJA/edit?tab=t.0#heading=h.y06xjqkf31ze)

---

@heshanpadmasiri, in your proposal, you are only supporting code mutations. Is that intentional?

[heshanpadmasiri]

My understanding is if treat a compiler plugin as function that takes a compiler context (needed for defining new symbols, types and giving compiler diagnostics) and ast.pkg and return a replacement ast.pkg (it could either mutate or create new one, I personally prefer mutation over copying) I think you can achieve all three (code generation will be restricted to the same package/module)

[heshanpadmasiri]

Difference from jBallerina design would be that we don't distinguish between the 3 different kinds of functions. Instead we expect each package to declare a single function. Then project API based on the module dependencies will build a sequence of such functions and apply them one after the other. If any of them add compiler diagnostics (should be errors only, when we distinguish warnings and errors) it will stop after that function (this is similar to how other front end stages work currently as well). Otherwise it will take the AST returned by that function and feed it to the next.

We don't validate the returned AST nor perform earlier analysis phases such as type resolution on them, instead we have defined the AST to be open enough with compiler context you can set them directly.

[heshanpadmasiri]

Potential issue with a forward pass only implementation (which I am still in favour of) is adding imports and referencing symbols in those imports in compiler plugins. I don't know if we do this already but this would require some gymnastics in the compiler plugin's code. Issue here is you are trying to refer to a symbol that don't exists in the compiler env.

[heshanpadmasiri]

This would required some way to ask the project api to compile that package, before continuing and then get the exported symbol space from that import

[heshanpadmasiri]

Of course this means that package (one that owns the compiler plugin) has a hidden dependency that will introduce all sorts of pains to us. So perhaps we shouldn't support this at all

[heshanpadmasiri]

Closed with issue #755