feat(melange): install melange libraries

[anmonteiro]

• First part of feature(melange): installation of libraries #6600:

  • Installs melange library artifacts in $LIB_DIR/melange/*.cm{i,j,t}
  • resolves libraries with modes melange
  • melange.emit emits public library dependency JS to $emit_target/node_modules/lib_name/**/*.js

• if refactor(melange): switch --bs-package-name to library name #6841 lands, we need to adapt this PR to install node_modules/<lib_name> rather than node_modules/<pkg-name>

• figure out a solution that doesn't use Lib_info.src_dir for public libraries

Signed-off-by: Antonio Nuno Monteiro anmonteiro@gmail.com

[anmonteiro]

Currently installing melange libraries in LIB_DIR/melange/*.cm{i,j,t}. I'm not sure if this is correct / conflicts with other features. Need some guidance on whether that's what we want

[rgrinberg]

Why is value_exn okay here?

[anmonteiro]

I don't know if it is. What would you suggest we do if it's None?

This brings up another design question:

• I think I'd rather use the library public_name as the --bs-package-name .. so that we could depend on multiple public libraries within the same package. The naming scheme for those is alphanumerical-maybe-dashes.sublibrary, which I think can be a valid folder name for the node.js module resolution algorithm

[rgrinberg]

What would you suggest we do if it's None?

Work the way it worked before I guess? Right now this will not work for private libraries.

I think I'd rather use the library public_name as the --bs-package-name .. so that we could depend on multiple public libraries within the same package. The naming scheme for those is alphanumerical-maybe-dashes.sublibrary, which I think can be a valid folder name for the node.js module resolution algorithm

Better ask @jchavarri about that.

[anmonteiro]

Why not? The tests are passing. This is in the branch where Lib.Local.of_lib is None. I thought that'd be the branch for external libraries.

[rgrinberg]

I see. But then you're distinguishing between public libraries that are installed and ones that are in the workspace.

I think you should distinguish between public (external or in-workspace) and private libraries instead.

[anmonteiro]

We could also always use Lib.name now that we have the node_modules emission stuff.

[rgrinberg]

There's Lib_info.Status.t. In particular, when it's Private (_, None)

[jchavarri]

I think I'd rather use the library public_name as the --bs-package-name ..
Better ask @jchavarri about that.

I thought about this and I believe it should be fine. The only caveat is that in order to consume a public library from the same workspace, the library should be installed first, but I guess that's ok? Are workspace libraries "constantly installed" when running e.g. dune build -w?

[jchavarri]

In particular, when it's Private (_, None)

I am taking a stab at fixing this, but I am confused about this comment. If we want to distinguish between public and private, why does the second part of the payload (None, the package) matter?

I made all tests pass with something like this:

let lib_dir =
  match Lib_info.status info with
  | Installed | Installed_private | Private (_, Some _) ->
    let package_name = Option.value_exn (Lib_info.package info) in
    let bctx, _ = Path.Build.extract_build_context_dir_exn target_dir in
    Path.Build.(
      relative
        ((relative bctx) "node_modules")
        (Package.Name.to_string package_name))
  | Private (_, None) | Public _ ->
    let lib = Lib.Local.of_lib_exn lib in
    let info = Lib.Local.info lib in
    Lib_info.src_dir info

but it does not make much sense to me.

[rgrinberg]

Public libraries have the following guarantee: their behavior is the same regardless of whether they're installed or are present in the current workspace.

The second part of the payload matters because there's also private libraries with the above guarantee. Those private libraries with the package field set.

[rgrinberg]

We have Path.Build.L.relative to make this easier to read

[jchavarri]

Added in anmonteiro@4224f26, can be merged separately or together with anmonteiro#3.

[rgrinberg]

Is there a test that has (modes melange byte)? To make sure they don't step over each other.

[anmonteiro]

I think I have one untracked in git that I can add

[rgrinberg]

Perhaps remove the duplication with:

List.concat_map [ (native || byte), Ocaml Cmi ; melange, Melange Cmi ] ~f:(...

[jchavarri]

Done in anmonteiro@27905d1.

[rgrinberg]

It would also be useful to add a test for private libraries that belong to package. For example:

(library
 (name foo)
 (package bar))

The findlib name for these is mangled.

[jchavarri]

The latest from main can be picked up from anmonteiro#3.

[jchavarri]

It would also be useful to add a test for private libraries that belong to package.

I added a test for this in anmonteiro@21de584, where there is a melange.emit and library that belong to the same package, but the library does not have public name, as mentioned.

But something fundamental seems to be broken in this case because dune can't find the rules for the cmjs (unless I am doing something wrong on the test setup itself). Still investigating.

[jchavarri]

But something fundamental seems to be broken in this case because dune can't find the rules for the cmjs (unless I am doing something wrong on the test setup itself). Still investigating.

Well, this is embarrassing. I was missing the (modes melange) field on the library. I hope we can do something about #6412 🙏 because at this point I think I've spent more than 1 hour in total while debugging different instances of these very confusing "No rule found for foo.cmj" errors 😕

The private lib test seems to be passing (see anmonteiro@0e7225a), so the only 2 things remaining that I am aware of are:

• Usage of Lib_info.status: asked about it here
• Add a test with (modes melange byte), @anmonteiro you mentioned you had some test around, but I can add it myself otherwise

[anmonteiro]

Add a test with (modes melange byte), @anmonteiro you #6602 (comment) you had some test around, but I can add it myself otherwise

Please do, I lost my untracked changes in my other computer.

[jchavarri]

Please do, I lost my untracked changes in my other computer.

Done in anmonteiro#5. Added as separate test but can merge into the existing one if it's preferable.

[jchavarri]

We discussed this privately, but leaving here a note for posterity:

The approach I ended up taking is to always put js artifacts of public libs in node_modules, regardless which package or dune project they belong to:

• If two public libraries belong to the same package, relative paths like require('../foo/bar.js') will work because the original source layout is honored under the node_modules folder
• If two public libs belong to different packages, non-relative paths like require('foo/bar.js') will work because at the top level there will be a node_modules folder that includes foo.

This approach breaks only in one case: when the entries in melange.emit belong to the same package as some public library. I was able to fix this by removing the package entry from melange.emit, so the entries in emit become "private" (in dune's Lib_info.Status.t terms).

Rudi answered:

For executables, we have a package field. But this is mainly useful when you want to:

• Install the executable
• Include the executable as part of the test suite for the package. so that dune build -p pkg @runtest will see this executable

We probably want to fix this at some point in the future, either by removing package field in melange.emit (at least in the short term), or figuring out how to perform installation of emit entries.

[rgrinberg]

Public is handled differently than | Installed | Installed_private which means that installation is going to be broken. I think for Public libraries you need to remove the src_dir component. If a library is public, the user can't make assumptions as to where its located