Inconsistent behaviour for virtual libraries using melange mode

[rizo]

Expected Behavior

When creating a virtual library in native or byte modes, it is possible to assign a module to be virtual and add an implementation and interface that wraps around the virtual module in the virtual library.

Specifically the following structure illustrates this:

- vlib/
  - virt.mli     # virtual module
  - vlib.mli     # interface of vlib
  - vlib.ml      # implementation of vlib (uses virt!)
- vlib_impl_1/
  - virt.ml      # implementation of the virt module for vlib

This works as expected in native and byte modes and allows providing multiple implementations for virt.mli while offering a high-level interface in vlib.mli.

This doesn't work in melange mode.

Actual Behavior

This doesn't work when (modes melange) is used. Not only it is not possible to replicate the described workflow, but having (modes melange) also prevents the non-melange implementation to be compiled.

Error when compiling a melange target:

$ dune build @melange
File "vlib/vlib.ml", line 1:        
Error: Virt not found, it means either the module does not exist or it is a namespace

Additionally, if (modes melange) is present in vlib, the native compilation fails with:

File "src/dune_rules/virtual_rules.ml", line 11, characters 30-37:
Error: No rule found for vlib/.vlib.objs/byte/virt.cmi
File "src/dune_rules/virtual_rules.ml", line 11, characters 30-37:
Error: No rule found for vlib/.vlib.objs/byte/vlib.cmi
File "src/dune_rules/virtual_rules.ml", line 11, characters 30-37:
Error: No rule found for vlib/.vlib.objs/byte/vlib.cmo
File "src/dune_rules/virtual_rules.ml", line 11, characters 30-37:
Error: No rule found for vlib/.vlib.objs/native/vlib.cmx
File "src/dune_rules/virtual_rules.ml", line 11, characters 30-37:
Error: No rule found for vlib/.vlib.objs/native/vlib.o

Reproduction

The following repository fully replicates the above scenario based on the blackbox test included in dune:

• https://github.com/rizo/dune-melange-virt-issue

Specifications

• Version of dune: ~3.7 05797cd3f505beca4d6d62c576506d7608a6d29d
• Version of melange: f4ddebbc7755af5e983c4df2ac51498f286e1074
• Version of ocaml: 4.14.0
• Operating system (distribution and version): 22.3.0 Darwin

[jchavarri]

From the test:

Melange can't produce a .cmj solely from a virtual module .cmi, because it
needs to consult the .cmj files of dependencies to know where the require
call should be emitted

I am trying to figure out if the solution for this is something that can be solved on dune side, on melange side, or a combination of both 😅

On Melange side:

• The cmj shouldn't really require generating js requires, but I think at the moment melange embeds the resulting js output inside the cmj for simplicity?
• I believe there are other performance optimization in melange that might inline functions from other modules by reading their cmjs, not sure of any solution that removes the cmj-cmj dependency would also be affected by that

On Dune side:

• I believe there's no particular need for producing cmjs on the virtual library folder. Would it be possible to add copy rules so that the virtual lib modules and the implementation lib modules are placed in the same folder? At that point, we could proceed with regular compilation normally.

@anmonteiro @rgrinberg wdyt?

[jchavarri]

Adding a note here from @anmonteiro: solving this issue requires implementing "delayed cmj optimization" (melange-re/melange#464), so in terms of timing in won't be fixed as part of dune 3.8.

[anmonteiro]

I'm fixing this over in #10051.

@rizo if you're still interested in this fix and would like to take it for a spin, I'd appreciate your feedback.

In terms of setup, you'll need:

1. pin melange to fix: handle case of missing .cmj when trying to inline external field melange-re/melange#1067
2. pin dune to fix(melange): depend on selected impl when setting up JS rules for virtual lib #10051
3. try compiling both a) your minimal reproduction and b) your real-world library where you'd like to use this feature