Add support for `(include_subdirs qualified)`

With the following project:

./src:
dune       # containing (include_subdirs qualified)
x.ml

./src/a:
y.ml

./src/a/b:
z.ml

x.ml would see y.ml as A.Y and z.ml as A.B.Z. Then y.ml would see z.ml as B.Z but couldn't see x.ml.

This should allow to organize the modules using the FS layout.

I'm interested in this as it would allow to port Camlp4 to dune, which would simplify it's maintenance, and it's also a nice feature to have. However, I'm not planning to work on this in the near future, so anyone should feel free to assign themselves this feature if they are interested in it.

[rgrinberg]

There's a few complications that have kept this feature on the back burner. I'll just list them here to note that I haven't forgotten about this feature:

• Our current code relies on passing around a lot of maps keyed by the module name. Naturally, this doesn't quite work anymore since the module names are no longer unique.

• I'd like to it be possible to define a lib interface module for every subdirectory. Our codebase currently assumes there's only one such module in many places.

[Lupus]

Will it still be possible to add some custom rules in dune files within the FS hierarchy that form one dune lib? Or define some aliases? I.e. will dune files within such hierarchy be treated as normal dune files with all features available?

Also will it be possible to customize compiler flags for certain submodules? Like I want to have -open Base, but it won't work with atdgen modules, currently I'm moving them to separate lib so that I can tune the compiler flags there.

May be ability for tuning ppx-es that are used will be nice to have. When using ppx_deriving with protobuf plug-in, sexp derivers stop working, I had to move protobuf types to separate library to just change ppx set used.

Will it still be possible to add some custom rules in dune files within the FS hierarchy that form one dune lib? Or define some aliases? I.e. will dune files within such hierarchy be treated as normal dune files with all features available?

It will be the same as for (include_subdirs unqualified). You'll be able to do everything except defining libraries or executables.

Also will it be possible to customize compiler flags for certain submodules? Like I want to have -open Base, but it won't work with atdgen modules, currently I'm moving them to separate lib so that I can tune the compiler flags there.

Probably not in the first version. But in the future, why not.

May be ability for tuning ppx-es that are used will be nice to have. When using ppx_deriving with protobuf plug-in, sexp derivers stop working, I had to move protobuf types to separate library to just change ppx set used.

You can already do that: https://dune.readthedocs.io/en/stable/concepts.html#per-module-preprocessing-specification

[rgrinberg]

Just so I don't forget this later on, here's quick brain dump that's blocking
this issue. The main problem is the layout of the alias module. Consider this
directory structure:

$ ls -R .
  dune
  y.ml
  x/
    foo.ml
    bar.ml
    x.ml
$ cat dune
  (include_subdirs qualified)
  (library (name lib))

In this setup, it's important to note that x/x.ml is a lib interface module
and that y.ml may not access foo.ml and bar.ml. We might consider the
following alias map that works for such a file:

$ cat lib__.ml
module X__ = struct
  module Foo = Lib__X__Foo
  module Bar = Lib__X__Bar
end
module X = Lib__X
module Y = Lib__Y

When compiling {foo,bar}.ml, we use the flags -open Lib__ -open X__.
When compiling y.ml, we use the flags -open Lib__.

This looks great except for one little blemish: now Y can see this ugly X__
name. I can't think of a way to avoid this downside without making the alising a
lot more complicated and using multiple alias files.

@diml, @aalekseyev, perhaps you guys have a suggestion?

[aalekseyev]

Will X__ also appear in module type of Lib?
My first reaction is that multiple alias modules seems reasonable and having X__ in the module type of Lib is not reasonable.

(we use multiple alias modules when we have multiple libraries right now so I'm not surprised if we need multiple alias modules when we have multiple include_qualified directories (I think about them roughly as libraries))

[aalekseyev]

Wait, doesn't something like this work?

module Lib__ = struct
  module To_open = struct
    module X = Lib__X
    module Y = Lib__Y
  end
  module X = struct
    module To_open = struct
      module Foo = Lib__X__Foo
      module Bar = Lib__X__Bar
    end
  end
end

When compiling {foo,bar}.ml, we use the flags -open Lib__.To_open -open Lib__.X.To_open.
When compiling y.ml, we use the flags -open Lib__.To_open.

(instead of multiple opens you can put a statement include To_open on top of X.To_open, which I think should also work if you've got the compiler that allows include shadowing)

[aalekseyev]

By the way, in your original message you probably wanted to say "y.ml may not access foo.ml and bar.ml", I suggest you edit that before others are confused.

[rgrinberg]

By the way, in your original message you probably wanted to say "y.ml may not access foo.ml and bar.ml", I suggest you edit that before others are confused.

Corrected. Thanks.

Wait, doesn't something like this work?

That seems like it's worth trying, but I'm not confident that these To_open identifiers aren't going to leak when doing type inference, completion in merlin. At least with the __, merlin knows to hide these symbols from the user.

I suppose there's no harm trying this out. Generating the alias module is a fairly small piece of the work anyway.

[aalekseyev]

[To_open does not contain __]

We could call it O__, or something like that. Thinking again, it seems we don't really need the surrounding modules at all:

module To_open_in_Lib__ = struct
  module X = Lib__X
  module Y = Lib__Y
end
module To_open_in_Lib__X__ = struct
  module Foo = Lib__X__Foo
  module Bar = Lib__X__Bar
end

Or maybe I'm missing something?

[rgrinberg]

This last suggestion looks the simplest to me.

…

On Mar 6, 2020, 11:26 AM +0000, Arseniy Alekseyev ***@***.***>, wrote: > [To_open does not contain __] We could call it O__, or something like that. Thinking again, it seems we don't really need the surrounding modules at all: module To_open_in_Lib__ = struct module X = Lib__X module Y = Lib__Y end module To_open_in_Lib__X__ = struct module Foo = Lib__X__Foo module Bar = Lib__X__Bar end Or maybe I'm missing something? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.