Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] dune plugins #1855

Open
diml opened this issue Feb 19, 2019 · 9 comments

Comments

Projects
None yet
4 participants
@diml
Copy link
Member

commented Feb 19, 2019

This ticket describes a very simple plugin system for dune.

Overview

The aim of this proposal is to provide a simple form of extensibility to users, so that they can share common parts of their dune files in one place.

The simple is very simple and allows users to define new stanzas. User defined stanzas are translated to vanilla stanzas using functions written in OCaml. For now, the system is completely untyped, but we are hopping to evolve it into something more structured in the future.

This proposal makes sure that the use of plugins won't negatively impact the functioning of dune. In particular, building the plugins must be as fast as possible and shouldn't introduce too much staging.

Description

The system provides the following function:

val Dune_plugin.V1.define_stanza : string -> Dune_lang.Ast.t -> Dune_lang.Ast.t list

this function will be used to expand stanzas of the form (<name> ...) in dune files. It receives the whole stanzas and must return a list of expanded stanzas. The result can contain custom stanzas to be interpreted by other plugins.

Definition of plugins

Plugins are defined in files called dune.plugin.ml. Such files are plain OCaml files that have access to a special API Dune_plugin. This API will provide the types of the AST, various helper functions and the define_stanza function.

What can plugins do

They can:

  • do any pure computation on the AST
  • do a few FS query using the Dune_plugin API, such as listing or reading source files

They can't:

  • access the FS directly using stdlib's function such as open_in
  • more generally, do any system call without going through the Dune_plugin API

Limitations

Dune plugins can only be composed of a single module and cannot use libraries. They cannot use code from other plugins either. These limitations are to ensure we can build all the mappers as fast as possible and without introducing too much staging in the process. We might lift these limitations in the future if everything goes well.

Scoping

The stanzas defined in a dune.plugin.ml file have the same scoping as anything else in dune: the subtree where the file appears but without crossing project boundaries.

Implementation

Dune will compile and link all the dune.plugin.ml files found while crawling the workspace into one single executable. It will also link at the end a small driver.

Dune will then execute this executable with as argument a file containing the list of queries. A query is composed of:

  • the name of the file to rewrite. This name include the directory where this file is
  • the marshaled AST of the custom stanzas to expand

In return, this executable should provide a list of response. A response being:

  • the name of the file that was rewritten
  • the marshaled AST of the expanded stanzas
  • the list of observations the OCaml did while rewriting the AST. For
    instance, if the code did Dune_mapper.V1.read_file "foo.ml", then
    this observation should be recorded here

The last point will allow dune to cache the output of plugins between runs.

Future extensions

Allow plugins to be packaged so that they can be reused by other projects.

@diml

This comment has been minimized.

Copy link
Member Author

commented Feb 19, 2019

Looking at this again, maybe we should just call these things plugins to be more generic. i.e. apply the following renaming:

  • dune.mapper.ml --> dune.plugin.ml
  • Dune_mapper.V1.register --> Dune_plugin.V1.register_mapper

We might want to add more functionalities here that are not just mappers.

@diml

This comment has been minimized.

Copy link
Member Author

commented Feb 20, 2019

Reading this again, we can start by not providing full file mappers. i.e. this should be enough to start with and will be easier to evolve into somethting more principled:

val Dune_plugin.V1.register_stanza : string -> Ast.t list (* arguments of the stanzas *) -> Ast.t list

@diml diml changed the title [RFC] ppx rewriters for dune files [RFC] dune plugins Feb 20, 2019

@diml

This comment has been minimized.

Copy link
Member Author

commented Feb 20, 2019

I updated the proposal

@bobot

This comment has been minimized.

Copy link
Collaborator

commented Feb 22, 2019

Preprocessors are defined in files called dune.plugin.ml.

Does that mean at most one by directory? Why should we crawl all the workspace? Two projects could have competing plugins.

I would propose to restrict at least by project, or could we have a stanza in a dune file that tells which plugins to use for preprocessing the stanzas of the file?

@diml

This comment has been minimized.

Copy link
Member Author

commented Feb 25, 2019

Yes, at most one per directory. I propose to use the usual scoping: a dune.plugin.ml apply to the sub-tree rooted where the dune.plugin.ml appears, intersected with the current project. So there is no risk of conflicts between projects.

This proposal doesn't cover sharing plugins between projects. To do that, I had in mind something like this: we would declare in the dune-project file of foo the public plugins we want to share, then in another dune-project we would write: (using foo 1.0). After this, the stanzas defined by foo would be available as foo.<stanza-name>.

@ejgallego

This comment has been minimized.

Copy link
Collaborator

commented Feb 26, 2019

IMHO the proposal is good and would help solving one important issue we have in Coq: running our test suite using Dune. I'll be happy to test the PR once it is ready.

@rgrinberg

This comment has been minimized.

Copy link
Member

commented Feb 26, 2019

From the dune dev meeting:

  • It was pointed out that debugging plugins should be as easy as possible. The promotion mechanism was proposed as a debugging mechanism.

  • Generating stanzas in multiple directories was deemed useful as well. However, this is an orthogonal feature that needs first class support in the dune language first.

@rgrinberg rgrinberg added proposal and removed enhancement labels Feb 27, 2019

@bobot

This comment has been minimized.

Copy link
Collaborator

commented Feb 28, 2019

With more though I would prefer to have a stanza (preprocess-dune-file-with dune.plugin.ml). I think it is easier to grasp a new project if things are explicit. Otherwise we are going to need to look at the existence of many different files to understand a project. Moreover we could name files in a documentative way test-generation.dune.plugin.ml. Finally we could later extend more easily the feature: (without-subdirs), (args ...), ...

@ejgallego

This comment has been minimized.

Copy link
Collaborator

commented Mar 11, 2019

I thought a bit more about this and the only limitation it would have for my test uses cases is that I cannot call coqdep to compute the dependencies of the test suite.

I guess I could indeed add a special API for this to the plugin API as part of the Coq PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.