[RFC] dune plugins

[ghost]

This ticket describes a very simple plugin system for dune.

Overview

The aim of this proposal is to provide a simple form of extensibility to users, so that they can share common parts of their dune files in one place.

The simple is very simple and allows users to define new stanzas. User defined stanzas are translated to vanilla stanzas using functions written in OCaml. For now, the system is completely untyped, but we are hopping to evolve it into something more structured in the future.

This proposal makes sure that the use of plugins won't negatively impact the functioning of dune. In particular, building the plugins must be as fast as possible and shouldn't introduce too much staging.

Description

The system provides the following function:

val Dune_plugin.V1.define_stanza : string -> Dune_lang.Ast.t -> Dune_lang.Ast.t list

this function will be used to expand stanzas of the form (<name> ...) in dune files. It receives the whole stanzas and must return a list of expanded stanzas. The result can contain custom stanzas to be interpreted by other plugins.

Definition of plugins

Plugins are defined in files called dune.plugin.ml. Such files are plain OCaml files that have access to a special API Dune_plugin. This API will provide the types of the AST, various helper functions and the define_stanza function.

What can plugins do

They can:

• do any pure computation on the AST
• do a few FS query using the Dune_plugin API, such as listing or reading source files

They can't:

• access the FS directly using stdlib's function such as open_in
• more generally, do any system call without going through the Dune_plugin API

Limitations

Dune plugins can only be composed of a single module and cannot use libraries. They cannot use code from other plugins either. These limitations are to ensure we can build all the mappers as fast as possible and without introducing too much staging in the process. We might lift these limitations in the future if everything goes well.

Scoping

The stanzas defined in a dune.plugin.ml file have the same scoping as anything else in dune: the subtree where the file appears but without crossing project boundaries.

Implementation

Dune will compile and link all the dune.plugin.ml files found while crawling the workspace into one single executable. It will also link at the end a small driver.

Dune will then execute this executable with as argument a file containing the list of queries. A query is composed of:

• the name of the file to rewrite. This name include the directory where this file is
• the marshaled AST of the custom stanzas to expand

In return, this executable should provide a list of response. A response being:

• the name of the file that was rewritten
• the marshaled AST of the expanded stanzas
• the list of observations the OCaml did while rewriting the AST. For
  instance, if the code did Dune_mapper.V1.read_file "foo.ml", then
  this observation should be recorded here

The last point will allow dune to cache the output of plugins between runs.

Future extensions

Allow plugins to be packaged so that they can be reused by other projects.

[ghost]

Looking at this again, maybe we should just call these things plugins to be more generic. i.e. apply the following renaming:

• dune.mapper.ml --> dune.plugin.ml
• Dune_mapper.V1.register --> Dune_plugin.V1.register_mapper

We might want to add more functionalities here that are not just mappers.

[ghost]

Reading this again, we can start by not providing full file mappers. i.e. this should be enough to start with and will be easier to evolve into somethting more principled:

val Dune_plugin.V1.register_stanza : string -> Ast.t list (* arguments of the stanzas *) -> Ast.t list

[ghost]

I updated the proposal

[bobot]

Preprocessors are defined in files called dune.plugin.ml.

Does that mean at most one by directory? Why should we crawl all the workspace? Two projects could have competing plugins.

I would propose to restrict at least by project, or could we have a stanza in a dune file that tells which plugins to use for preprocessing the stanzas of the file?

[ghost]

Yes, at most one per directory. I propose to use the usual scoping: a dune.plugin.ml apply to the sub-tree rooted where the dune.plugin.ml appears, intersected with the current project. So there is no risk of conflicts between projects.

This proposal doesn't cover sharing plugins between projects. To do that, I had in mind something like this: we would declare in the dune-project file of foo the public plugins we want to share, then in another dune-project we would write: (using foo 1.0). After this, the stanzas defined by foo would be available as foo.<stanza-name>.

[ejgallego]

IMHO the proposal is good and would help solving one important issue we have in Coq: running our test suite using Dune. I'll be happy to test the PR once it is ready.

[rgrinberg]

From the dune dev meeting:

• It was pointed out that debugging plugins should be as easy as possible. The promotion mechanism was proposed as a debugging mechanism.

• Generating stanzas in multiple directories was deemed useful as well. However, this is an orthogonal feature that needs first class support in the dune language first.

[bobot]

With more though I would prefer to have a stanza (preprocess-dune-file-with dune.plugin.ml). I think it is easier to grasp a new project if things are explicit. Otherwise we are going to need to look at the existence of many different files to understand a project. Moreover we could name files in a documentative way test-generation.dune.plugin.ml. Finally we could later extend more easily the feature: (without-subdirs), (args ...), ...

[ejgallego]

I thought a bit more about this and the only limitation it would have for my test uses cases is that I cannot call coqdep to compute the dependencies of the test suite.

I guess I could indeed add a special API for this to the plugin API as part of the Coq PR.

[ghost]

I thought a bit more about this proposal and I don't think this is going in the right direction, or at least it's premature. For now, we should focus on unifying and making the dune language more well defined. Currently, a lot of things are a bit ad-hoc, such as what variables are allowed and where they are allowed. #1888 is the first step in this direction. Once we have a better and stronger basis, we will be able to safely extend the dune language to allow users to define their own stanzas and factorise a bit their dune files. For now, OCaml will be reserved for static extensions, as described in #2151.

Once we have all that, we can re-evaluate whether we need dynamic OCaml extensions or not.