js_of_ocaml-lwt installs incorrectly with dune 1.7.1

[glondu]

With dune 1.7.1 in ocaml 4.06.1 switch, opam install eliom=6.3.0 fails. It boils down to js_of_ocaml-lwt not being installed correctly. Indeed, the META file generated doesn't include the logger sublibrary. It seems the logger sublibrary is not built at all. With dune 1.6.2, the sublibrary is properly built, and its META stanza is properly generated.

[ghost]

This might be related to #1841. The logger sublibrary is optional and is built and installed only if its dependencies are available. What version of lwt are you using?

[glondu]

What version of lwt are you using?

3.3.0

[ghost]

Can you upgrade to Lwt>=4.0.0? Versions before 4.0.0 are still using a manual "install_filter.ml" script to rewrite the META file and reorganise the files, which confuses dune >= 1.7.

Another option would be to release a new minor version of lwt 3.x where install_filter.ml would also delete any dune-package in addition to rewriting the META file. /cc @aantron

[glondu]

Can you upgrade to Lwt>=4.0.0?

It doesn't seem possible to install eliom 6.3.0 with lwt >= 4.0.0. If I give up on eliom, and install only lwt_log and js_of_ocaml-lwt, the logger sublibrary seems to install correctly.

OK, so this is an incompatibility between lwt < 4.0.0 and dune >= 1.7.0 that is not reflected in the repository metadata.

[ghost]

Yes, it should definitely be reflected in the opam-repository.

[ghost]

I wrote a PR: ocaml/opam-repository#13485

[aantron]

We could make an Lwt 3.3.1. @glondu what would you prefer for your needs?

[avsm]

Having a split in the build universe with dune isn't ideal. A minor release of Lwt 3.3.1 would be fantastic @aantron

[aantron]

I'll make a release this week.

[glondu]

@aantron diml's PR (which has been merged) is enough, thank you.

[aantron]

Thanks, @glondu. @avsm, Could you let me know if Lwt 3.3.1 is still needed?

[aantron]

I understand this potentially affects many other packages in other ways, but I'm not as aware of the extent as you may be.

[avsm]

I think a Lwt 3.3.1 will still help. Although the constraints are now correct in opam-repository, the issue is that several packages still have a lwt<4 constraint as they migrate. Ensuring that there isnt a 'double constraint' of a dune version and an lwt version will greatly relieve the pressure on the solver to find a working solution. It's not a disaster if we don't have a lwt 3.x point release though -- the effort may be better spent looking for pre-lwt-4 packages and encouraging them to upgrade.

[aantron]

I found the following packages with an lwt < 4.0.0 constraint:

• aws
• bap-server
• core-lwt
• hardcaml-waveterm
• i3ipc
• iocaml-kernel
• learn-ocaml
• lwt-parallel
• macaque_lwt
• otetris
• pvem_lwt_unix
• qcow-format
• session-postgresql-lwt
• shared-block-ring
• skkserv-lite
• socket-daemon
• testrunner
• usb

The following packages have lwt < 4.0.0 as an optional dependency:

• bookaml
• mongo
• smtp
• unix-sys-stat

Almost all of these are leaves in the dependency graph, or very near to leaves, and have no or few monthly downloads. The only exception is session-postgresql-lwt, which is a leaf, but has ~200 monthly downloads. Some of the packages look unmaintained, perhaps deprecated.

Based on this, I suggest not to release Lwt 3.3.1 for now, but watch for any more reported problems.

[glondu]

@aantron In Debian, we have (at least) usb. And the latest lwt has a new dependency (mmap), which will take time to integrate to Debian. Therefore, I am planning to update lwt to the 3.3.x branch. Is the patch to make lwt < 4 work with dune >= 1.7 (hopefully 1.11.0) complicated?

[aantron]

Having to integrate mmap into a system package manager is regrettable, since it's just a shim that we needed due to a breaking change in the compiler. Perhaps we should instead eliminate this dependency by having Lwt detect the OCaml version (like it already does for other purposes), and adding some conditional compilation? We are using mmap for one line in an .ml file and one line in a dune file.

Would it be easier to upgrade usb instead?

Making a 3.3.x compatible with modern Dune shouldn't be too difficult, but it seems more reasonable to adapt the remaining packages to current Lwt.

[glondu]

Having to integrate mmap into a system package manager is regrettable, since it's just a shim that we needed due to a breaking change in the compiler.

I didn't realize that mmap was so trivial, as trivial as creating an opam package for it. Unfortunately, adding a package to Debian has much more overhead and goes through a rather lengthy process.

Perhaps we should instead eliminate this dependency by having Lwt detect the OCaml version (like it already does for other purposes), and adding some conditional compilation?

Yes, please do.

Would it be easier to upgrade usb instead?

This seems useful independently of the above.

[aantron]

Is it correct, then, to say that we are aiming for Lwt 4.x.x in Debian, rather than a 3.3.x release?

[glondu]

Is it correct, then, to say that we are aiming for Lwt 4.x.x in Debian, rather than a 3.3.x release?

Eventually, yes. But in the meantime, I might package an intermediate version if that makes things easier. At the moment, I am aiming at Lwt 2.7.1 with camlp4 disabled which already causes problems.

FYI, I've set up a wiki page to track down progress in updating OCaml packages in Debian.