Ability to generate virtual dune files

[jordwalke]

It would be very helpful to allow a custom process to determine where dune files should exist and what their content is, so that dune files don't appear in the source tree. For simple or opinionated projects, this would be very helpful. For example, suppose a company has a particular way of laying out modules / namespaces in their repo, and you want to generate some dune files that mimic the conventions of the existing system without it being totally apparent that Dune is being used to power those conventions, and without the dune files adding to the mountain of config that this old codebase already has. This makes it a much easier sell to incrementally work Dune into an existing old project with tons of legacy build setups.

What's the alternative? A wrapper script that first generates the dune files at their correct locations in the original source tree, then runes the dune build.

What would be better: If there was some hook that allowed an executable/command to generate "virtual" dune files along with describing the locations these dune files should exist virtually. These dune files would only ever actually appear on disk in the _build directory, and would stay out of the source tree. Maybe with a simple implementation of this feature, it would not be quite as fast, but I imagine it wouldn't be hard to perform better than the alternative which would be to have a wrapper generate these dune files before each build where Dune couldn't possibly even know about them. At least with a hook that is executed before each build, Dune could be aware of which Dune configs are present, and which have changed since last time. If relying exclusively on this hook, Dune wouldn't even have to search the file system for dune files.
As a follow up, I wonder if the current Dune behavior could be implemented as a special case of this more general dune file "discovery".

let discovered_dune_files_by_location =
  if using_dune_file_discovery_hook then
    call_dune_discovery_hook ()
  else
    find_dune_files_in_source_tree ()

Where what is returned is some list of file_location * dune_file_text.

[ghost]

The more general feature here is plugins. We were originally thinking of giving access to the internal dune APIs to plugins, which is a huge commitment on our side. However for what you want to do the dune language is enough. We could imagine a plugin mechanism where plugins would just generate S-expressions.

[jordwalke]

I tried the include feature but it's limited to only being able to generate flags. I wonder if it makes sense to just special casing this one case where projects want to generate dune config out of the source of the project. It seems a lot easier, but I maybe redundant if you just end up supporting plugins anyways.

[ghost]

Indeed, however at the moment nobody is working on plugins. I suggest the following if you want a quicker solution: try to add a hook in dune that will allow to do what you want outside of the dune repository. Once you have something working, we can merge the hook in dune.

[jordwalke]

Thank you, @diml