Lwt 4.0.0 breaking changes notice

[aantron]

Lwt 4.0.0 will be released towards the end of 2017 in March 2018 with minor breaking changes to packaging, and perhaps several APIs. Hopefully, there will be less breakage than in 3.0.0.

The current release, 3.1.0, prepares to delete some modules. Another planned release, 3.2.0, will schedule additional breaking changes for 4.0.0. A comment will be added to this issue when 3.2.0 is released, with details and instructions about those additional plans.

---

• Remove package lwt.simple-top (Delete simple-top #371). This has been deprecated for years, in favor of utop. It has no users in OPAM, and users found in a GitHub search have already been notified.

---

• Remove module Lwt_chan (package lwt.unix) (Delete Lwt_chan in 4.0.0 #441). This has been deprecated for nine years in favor of Lwt_io, which subsumes it completely. GitHub and OPAM package maintainers have been notified in the linked issue.

[ELLIOTTCABLE]

Hi! Random new-OCaml-learner here, coming from the JavaScript/Node world.

Just came by to say this careful, premeditated handling of breaking changes is extremely heartening, and makes me 10x more excited to join the OCaml community.

Keep up the good work! <3

[aantron]

Well, thanks :)

[aantron]

Here are additional breaking changes, also planned for the future 4.0.0, announced and prepared by the recent 3.2.0 release. To give proper time to adapt, 4.0.0 will be released in three months, around March 2018.

Each breaking change is detailed below, with instructions on how to prepare for it. I will cc the maintainers of affected opam packages after 3.2.0 becomes installable. I've listed the names of opam packages potentially affected under each notice. See if your package is in the lists.

The scheduled breaking changes are still proposals. If you think anything below is a really bad idea, please don't hesitate to object or otherwise comment :) The three-month window is for adaptation and discussion :)

---

• Package lwt.ppx (Move the PPX to a separate package (?) #338):

  The PPX is being factored out into its own opam package, called lwt_ppx. This new opam package is available now, as of Lwt 3.2.0.

  In Lwt 4.0.0, the old method of linking with the PPX, which is to install package lwt and refer to the PPX as lwt.ppx, will no longer work. The PPX will be available only as Findlib package lwt_ppx in the new opam package of the same name.

  To adapt, if you use the PPX, depend on the new opam package lwt_ppx now, and replace lwt.ppx by lwt_ppx in your build system.

  lwt.ppx is used in packages: bitcoin bookaml brozip cstruct devkit elasticsearch-cli eliom future gamepad gdb gdbprofiler hardcaml i3ipc imaplet-lwt jsoo_router jupyter lambdoc litiom maki multipart-form-data ocsigen-start ocurl ojs-base otetris qfs rawlink rdf slacko socket-daemon spotify-cli sqlexpr stog testrunner usbmux vhd-format

---

• Package lwt.syntax (Move Camlp4 syntax to a separate package #370):

  As with the PPX, the Camlp4 syntax is being factored out into its own opam package, lwt_camlp4. This new opam package is available now, as of Lwt 3.2.0.

  In Lwt 4.0.0, it will no longer be possible to use the Camlp4 syntax by installing opam packages camlp4 and lwt, and referring to the syntax as lwt.syntax. The Camlp4 syntax will be available only as Findlib package lwt_camlp4, in the new opam package of the same name.

  The Camlp4 syntax is deprecated, and we will likely delete it completely from the Lwt repository after the 4.0.0 release (Retiring lwt_camlp4 and lwt_log from the repo #511).

  To adapt, if you use the Camlp4 syntax, we recommend replacing it by the PPX. However, if you don't wish to do that, depend on the new opam package lwt_camlp4, and replace lwt.syntax by lwt_camlp4 in your build system.

  lwt.syntax is used in packages: aws biocaml bitcoin eliom facebook-sdk hardcaml imaplet-lwt iocaml iocaml-kernel iocamljs-kernel jitsu joolog js_of_ocaml lambdoc mirage-http-xen mirari mongo ocsigenserver ojquery ojwidgets rdf release riak sqlexpr themoviedb tls usb xe xen-disk xenctrl xentropyd

---

• Package lwt.log, modules Lwt_log and Lwt_daemon (attempt to move lwt.log into separate lwt_log #484):

  The Lwt logging library, consisting of modules Lwt_log, Lwt_log_core, Lwt_log_rules, and Lwt_daemon, is being deprecated in favor of the logs library, which includes an Lwt-aware module Logs_lwt.

  As part of this, the Lwt logging library is being factored out into its own opam package, lwt_log. This new opam package is available now, as of Lwt 3.2.0.

  In Lwt 4.0.0, it will no longer be possible to use the Lwt logging library by installing opam package lwt, and linking with Findlib packages lwt.log and/or lwt.unix.

  To adapt, if you use the logging library, we recommend switching to Logs_lwt. Otherwise, depend on the new opam package lwt_log, and resolve any linking problems by adding dependencies on the Findlib package lwt_log in your build system. If you use Lwt_daemon, it is also included in the new opam package lwt_log, but see the discussion starting from this comment.

  Lwt_log or Lwt_daemon is used in packages: 0install amqp-client arakoon bap-server bt camltc charrua-client charrua-* cohttp conduit distributed eliom ezirmin gdbprofiler joolog js_of_ocaml jupyter kinetic-client lambda-term lwt-parallel message-switch mirage-conduit mirage-console mirage-http nbd nsq ocsigen-start ocsigenserver ojs-base opium ordma release shared-block-ring skkserv-lite usbmux utop uwt websocket xentropyd xen-disk yurt

---

• >> syntax in the PPX (Deprecate >> in ppx #471, Remove >> syntax from the PPX #495):

  The >> construct will be removed from the PPX in 4.0.0.

  This was originally added to ease porting to the PPX from the Camlp4 syntax. However, it is has proved troublesome – see Remove >> syntax from the PPX #495.

  To adapt, we recommend replacing all instances of foo >> bar with foo >>= fun () -> bar. Otherwise, constrain your project with {"lwt" < "4.0.0"}.

  >> is used in packages: hardcaml imaplet-lwt ocsigenserver ocurl release sqlexpr usbmux xen-api-client

---

• Package lwt.preemptive (Make lwt.preemptive an alias for lwt.unix; remove it in 4.0.0 #487):

  Findlib package lwt.preemptive is being merged into lwt.unix in Lwt 4.0.0, and the name lwt.preemptive is being retired.

  In Lwt 3.2.0, lwt.preemptive is already just an alias for lwt.unix, and the actual content has been moved into lwt.unix. So, to adapt, delete all references to lwt.preemptive in your build system, replacing them with lwt.unix in places where lwt.unix is not already listed as a Findlib dependency. This requires that you constrain your project with {"lwt" >= "3.2.0"}.

  lwt.preemptive is used in packages: 0install arakoon baardskeerder bistro camltc ctypes debian-formats flow future hvsock imaplet-lwt maki ocsigenserver ordma osx-cf osx-fsevents pvem_lwt_unix stog trakeva unix-dirent unix-fcntl unix-sys-stat unix-unistd usb usbmux vhd-format vmnet

---

• Lwt.waiter_of_wakener (Contravariance of Lwt resolvers is unsound #458):

  This function should not be used, as it makes contravariance of resolvers ('a Lwt.u) unsound. To adapt, explicitly retain references to promises that correspond to resolvers.

---

• Module Lwt_sequence (Factor out Lwt_sequence (doubly-linked list) #361):

  Lwt_sequence, an implementation of ordinary mutable doubly-linked lists, is deprecated (Factor out Lwt_sequence (doubly-linked list) #361). To adapt, switch to an implementation from Containers or Core.

  If you need to use Lwt_sequence because you are using some of the APIs in Lwt_main, you can suppress deprecation warnings in one of two ways:

  • For an expression e referring to Lwt_sequence, wrap it in (e) [@ocaml.warning "-3"].

  • To suppress the warning in an entire file, create a new, non-deprecated alias for Lwt_sequence:

    [@@@ocaml.warning "-3"]
    module Lwt_sequence = Lwt_sequence
    [@@@ocaml.warning "+3"]
    

  Lwt_sequence is used in packages: arakoon bt dns-forward lambda-term mirage-net-* mirage-solo5 mirage-tcpip-* mirage-unix mirage-xen netchannel nocrypto notty ordma release shared-memory-ring sqlexpr tcpip usb utop utp vmnet xen-evtchn xen-gnt

---

• Rare linking bug (Stop flattening the installation directory #482):

  The layout of the Lwt installation directory will change in 4.0.0. While this is not part of the Lwt API, a few packages depend (or depended) on lwt.unix incorrectly, by linking only with Findlib package lwt. Because the library files for lwt and lwt.unix are being installed in the same directory in the old layout, Findlib is still able to locate and link with them. This will no longer be the case in 4.0.0.

  Some of the affected packages are: biocaml dns irmin_watcher mirage-profile

---

• Lwt.bind (Lwt semantic improvements: exception safety and stack overflows #500): See https://discuss.ocaml.org/t/1337/13

  The semantics of Lwt.bind, and most other Lwt functions, are being adjusted for better exception and stack safety. Specifically, in 4.0.0, Lwt.bind on resolved promises will sometimes defer callbacks, instead of always running them immediately.

  This can change the order in which callbacks are executed in your program.

  The 3.2.0 release retains the "old" semantics. To test your code against the future (4.0.0) semantics, pin Lwt to the safer-semantics branch:

  opam pin add lwt https://github.com/ocsigen/lwt.git\#safer-semantics
  

  Then, rebuild your projects, and run their tests.

  If your project does not work with both Lwt 3.x.x and safer-semantics (4.0.0), the simplest thing to do is to constrain it by adding {"lwt" < "4.0.0"} to its opam file. However, we encourage adjusting it, so that it works with both 3.x.x and safer-semantics (4.0.0).

  Since this potentially affects all packages, we are announcing this change on forums, mailing lists, etc., and adding a message to Lwt's opam file.

[aantron]

cc @agarwal @akabe @andersfugmann @andrenth @andrewray @anyakichi @Armael @avsm @balat @c-cube @copy @cyberhuman @darioteixeira @delamonpansie @diml @djs55 @dominicjprice @domsj @dsheets @edwintorok @emillon @essdotteedot @fdopen @fxfactorial @gaborigloi @gildor478 @gregtatcam @haesbaert @hannesm @hhugo @ivg @johnelse @jrochel @kayceesrk @Leonidas-from-XIV @lindig @MassD @mato @metadave @mfp @mseri @msimon

[aantron]

cc @nojb @pqwy @pveber @rgrinberg @ryanslade @samoht @seliopou @smondet @talex5 @toolslive @vasilisp @vbmithr @vouillon @vzaliva @xvw @yallop @ygrek @yomimono @zoggy @zshipko

[aantron]

Please see the above comment, with instructions on how to adapt to future changes in Lwt 4.0.0, which will be made in three months. Mostly, adapting will involve adding new dependencies to opam files, due to changed packaging of Lwt helper libraries.

[gildor478]

@aantron can you list the affected packages?

What command do you use to estimate what packages are impacted by this change?

Is this something like:

$> opam list --depends-on lwt

[aantron]

@gildor478 The affected packages for what? Each specific change lists the affected packages under the description already. Are you referring to something else?

The command is

opam list --unavailable --depends-on lwt --depopts --recursive --short

[gildor478]

@aantron Thanks. The list of affected packages by breaking changes is perfect.

I have received first the comment where I was CCed and I was wondering how to act on it. It looked like a long list of sorted names. You may want to improve the CC message with something like

@gildor478 for debian-formats,
@mfp for sqlexpr,
@agarwal for biocaml,
[...]

This can help people to know what they should change.

Anyway, thanks. I'll change ocaml-debian-formats accordingly.

[aantron]

Thanks. I'll probably have to do that for the next time we have breaking changes.

[aantron]

The Lwt.bind change will most likely not happen. Please see https://discuss.ocaml.org/t/1337/13 for details. Hopefully, this makes maintenance easier for everyone, and thanks for all feedback :)

[rneswold]

That's a shame. Sounds like this change would have made bind behave consistently and correctly. Plus, bumping the major version implies API changes that may not be backward-compatible. If people are worried, they can continue to use the 3.2.x branch until they've tested thoroughly against 4.0.x. After all, OPAM reports available versions all the way back to 2.3.2 so 3.2.x isn't going away soon.

[ivg]

That's a shame. Sounds like this change would have made bind behave consistently and correctly. Plus, bumping the major version implies API changes that may not be backward-compatible. If people are worried, they can continue to use the 3.2.x branch until they've tested thoroughly against 4.0.x. After all, OPAM reports available versions all the way back to 2.3.2 so 3.2.x isn't going away soon.

The majority of big projects, that are serious stakeholders of Lwt, will then may choose to continue to use Lwt 3.x, that will eventually lead to a nasty schism, like gnome 2.x vs gnome 3.x, etc. This will, in the best case scenario, double the support burden, and, in the worst one will make many packages uninstallable, or create yet another branch in the already too complex "which async library to use" decision tree. Given that in the real life, the noted problem with the bind semantics, is not often observable (if at all), I don't think that we should make this step.

Also, I would suggest everyone to refrain yourself from using exceptions together with Lwt (or any other async computations) and use variants to denote failed computations. Under such model, the bind function has perfectly fine and sound semantics.

[aantron]

We may still be able to make the Lwt.bind change, but with more preparation, to make it easier for users to adapt. For example, we could add a testing mode to Lwt that randomizes the deferral strategy to help discover mistaken assumptions about it.

However, I'm not sure what the ultimate value of this would be, given it's still work for users, it can't be flagged statically, and the semantic problem is not a problem for users in practice.

All the semantic problem means is for developers of Lwt, we have to do extra work when editing any core Lwt API, to ensure that it interacts safely with bind. But we almost never change the core Lwt API anyway. Fixing Lwt.bind would make it possible to analyze its behavior in isolation from other APIs, which is worthwhile, but probably not worth breaking all users for.

I agree strongly with @ivg about avoiding exceptions. Reflecting a lot on experience with Lwt and JS promises, my overall opinion is that rejections and asynchronous exceptions are both pretty ill-conceived mechanisms, almost as poor as cancelation. While it would be nice if these things worked well, because exceptions need to go somewhere and we don't want to handle them manually each time, each actual design that I've seen or come up with so far seems to produce undesirable consequences.

As @ivg points out, even though Lwt supports rejection, a user that is aware of its problems is likely to be using explicit error handling, in which case the Lwt.bind problem "accidentally" becomes irrelevant.

Lwt_result can help with this.

[copy]

Async provides an Eager_deferred module that can be opened to shadow bind and friends with their eager (lwt-like) versions. Maybe a similar solution could be provided by lwt to opt-in for safe monads. That is assuming it is possible to have both styles of promises in the same process and the implementation complexity is reasonable.

[aantron]

We could eventually do that. It should be very easy to get both styles of API exposed. The changes from #500 support this.

[aantron]

Closing this, as 4.0.0 is now merged into opam (ocaml/opam-repository#11708). See also the main announcement on Discourse.

Further discussion, questions, etc., are welcome, of course :)

[vog]

Great work! However, the documentation and tutorials should reflect the new project structure as well, especially with regard to lwt.ppx versus lwt_ppx.

[aantron]

@vog thanks for catching and reporting it! I (think I) fixed all remaining lwt.ppx in 0a1863d just now. The docs site is a bit behind on picking up Lwt changes. That is being tracked in #576 for now.