[new release] conduit, conduit-mirage, conduit-lwt, conduit-lwt-unix and conduit-async (6.2.0)

[hannesm]

A network connection establishment library

• Project page: https://github.com/mirage/ocaml-conduit
• Documentation: https://mirage.github.io/ocaml-conduit/

CHANGES:

• conduit-lwt-unix: adapt tls 0.16 split: tls.lwt is now tls-lwt (conduit-lwt-unix: update to tls 0.16 package (now it is tls-lwt, no more tls.lwt) mirage/ocaml-conduit#419 @hannesm)
• conduit-mirage: adapt dns-client 7.0.0 split: dns-client-mirage (conduit-lwt-unix: update to tls 0.16 package (now it is tls-lwt, no more tls.lwt) mirage/ocaml-conduit#419 @hannesm)

[hannesm]

I struggle a bit with reverse dependency failures of sihl (e.g. 0.1.10) fails https://opam-ci.ci3.ocamllabs.io/github/ocaml/opam-repository/commit/45bdebd1ff0056182d5d140c21e7422fb978bb4d/variant/compilers,5.0,conduit-lwt-unix.6.2.0,revdeps,sihl.0.1.10 with:

# (cd _build/default && /home/opam/.opam/5.0/bin/ocamlc.opt -w -40 -g -bin-annot -I sihl/http/.sihl_http.objs/byte -I /home/opam/.opam/5.0/lib/angstrom -I /home/opam/.opam/5.0/lib/astring -I /home/opam/.opam/5.0/lib/base -I /home/opam/.opam/5.0/lib/base/base_internalhash_types -I /home/opam/.opam/5.0/lib/base/caml -I /home/opam/.opam/5.0/lib/base/shadow_stdlib -I /home/opam/.opam/5.0/lib/base64 -I /home/opam/.opam/5.0/lib/bigstringaf -I /home/opam/.opam/5.0/lib/biniou -I /home/opam/.opam/5.0/lib/bytes -I /home/opam/.opam/5.0/lib/camlp-streams -I /home/opam/.opam/5.0/lib/cmdliner -I /home/opam/.opam/5.0/lib/cohttp -I /home/opam/.opam/5.0/lib/cohttp-lwt -I /home/opam/.opam/5.0/lib/cohttp-lwt-unix -I /home/opam/.opam/5.0/lib/conduit -I /home/opam/.opam/5.0/lib/conduit-lwt -I /home/opam/.opam/5.0/lib/conduit-lwt-unix -I /home/opam/.opam/5.0/lib/conformist -I /home/opam/.opam/5.0/lib/containers -I /home/opam/.opam/5.0/lib/containers/monomorphic -I /home/opam/.opam/5.0/lib/cstruct -I /home/opam/.opam/5.0/lib/digestif -I /home/opam/.opam/5.0/lib/domain-name -I /home/opam/.opam/5.0/lib/easy-format -I /home/opam/.opam/5.0/lib/either -I /home/opam/.opam/5.0/lib/eqaf -I /home/opam/.opam/5.0/lib/ezjsonm -I /home/opam/.opam/5.0/lib/fieldslib -I /home/opam/.opam/5.0/lib/fmt -I /home/opam/.opam/5.0/lib/hex -I /home/opam/.opam/5.0/lib/hmap -I /home/opam/.opam/5.0/lib/ipaddr -I /home/opam/.opam/5.0/lib/ipaddr-sexp -I /home/opam/.opam/5.0/lib/ipaddr/unix -I /home/opam/.opam/5.0/lib/jsonm -I /home/opam/.opam/5.0/lib/jwto -I /home/opam/.opam/5.0/lib/logs -I /home/opam/.opam/5.0/lib/lwt -I /home/opam/.opam/5.0/lib/lwt/unix -I /home/opam/.opam/5.0/lib/lwt_ssl -I /home/opam/.opam/5.0/lib/macaddr -I /home/opam/.opam/5.0/lib/magic-mime -I /home/opam/.opam/5.0/lib/multipart-form-data -I /home/opam/.opam/5.0/lib/ocaml-compiler-libs/common -I /home/opam/.opam/5.0/lib/ocaml-compiler-libs/shadow -I /home/opam/.opam/5.0/lib/ocaml/compiler-libs -I /home/opam/.opam/5.0/lib/ocaml/threads -I /home/opam/.opam/5.0/lib/ocaml/unix -I /home/opam/.opam/5.0/lib/ocplib-endian -I /home/opam/.opam/5.0/lib/ocplib-endian/bigstring -I /home/opam/.opam/5.0/lib/opium -I /home/opam/.opam/5.0/lib/opium_kernel -I /home/opam/.opam/5.0/lib/parsexp -I /home/opam/.opam/5.0/lib/ppx_derivers -I /home/opam/.opam/5.0/lib/ppx_deriving/api -I /home/opam/.opam/5.0/lib/ppx_deriving/create -I /home/opam/.opam/5.0/lib/ppx_deriving/enum -I /home/opam/.opam/5.0/lib/ppx_deriving/eq -I /home/opam/.opam/5.0/lib/ppx_deriving/fold -I /home/opam/.opam/5.0/lib/ppx_deriving/iter -I /home/opam/.opam/5.0/lib/ppx_deriving/make -I /home/opam/.opam/5.0/lib/ppx_deriving/map -I /home/opam/.opam/5.0/lib/ppx_deriving/ord -I /home/opam/.opam/5.0/lib/ppx_deriving/runtime -I /home/opam/.opam/5.0/lib/ppx_deriving/show -I /home/opam/.opam/5.0/lib/ppx_deriving/std -I /home/opam/.opam/5.0/lib/ppx_deriving_yojson/runtime -I /home/opam/.opam/5.0/lib/ppx_sexp_conv/runtime-lib -I /home/opam/.opam/5.0/lib/ppxlib -I /home/opam/.opam/5.0/lib/ppxlib/ast -I /home/opam/.opam/5.0/lib/ppxlib/astlib -I /home/opam/.opam/5.0/lib/ppxlib/print_diff -I /home/opam/.opam/5.0/lib/ppxlib/stdppx -I /home/opam/.opam/5.0/lib/ppxlib/traverse_builtins -I /home/opam/.opam/5.0/lib/ptime -I /home/opam/.opam/5.0/lib/re -I /home/opam/.opam/5.0/lib/re/str -I /home/opam/.opam/5.0/lib/result -I /home/opam/.opam/5.0/lib/safepass -I /home/opam/.opam/5.0/lib/seq -I /home/opam/.opam/5.0/lib/sexplib -I /home/opam/.opam/5.0/lib/sexplib0 -I /home/opam/.opam/5.0/lib/ssl -I /home/opam/.opam/5.0/lib/stdlib-shims -I /home/opam/.opam/5.0/lib/stringext -I /home/opam/.opam/5.0/lib/tsort -I /home/opam/.opam/5.0/lib/tyxml -I /home/opam/.opam/5.0/lib/tyxml/functor -I /home/opam/.opam/5.0/lib/uchar -I /home/opam/.opam/5.0/lib/uri -I /home/opam/.opam/5.0/lib/uri-sexp -I /home/opam/.opam/5.0/lib/uri/services -I /home/opam/.opam/5.0/lib/uutf -I /home/opam/.opam/5.0/lib/yojson -I sihl/core/.sihl_core.objs/byte -I sihl/utils/.sihl_utils.objs/byte -intf-suffix .ml -no-alias-deps -open Sihl_http__ -o sihl/http/.sihl_http.objs/byte/sihl_http__Cookie.cmo -c -impl sihl/http/cookie.pp.ml)
# File "sihl/http/cookie.ml", line 78, characters 4-26:
# 78 |     Mirage_crypto.Hash.mac
#          ^^^^^^^^^^^^^^^^^^^^^^
# Error: Unbound module Mirage_crypto

Now, I've no clue which package provided the mirage-crypto dependency for sihl, and/or how to restrict it properly. Related is @kit-ty-kate comment at #23315 (comment)

Looking upstream at sihl, I'm as well not sure whether it is still active (there's a huge PR with not much activity recently), maybe @joseferben can shed some light into this?

[hannesm]

Another revdep failure is current_gitlab:

# (cd _build/default && /home/opam/.opam/5.0/bin/ocamlopt.opt -w -40 -g -I plugins/gitlab/.current_gitlab.objs/byte -I plugins/gitlab/.current_gitlab.objs/native -I /home/opam/.opam/5.0/lib/ISO8601 -I /home/opam/.opam/5.0/lib/angstrom -I /home/opam/.opam/5.0/lib/ansi -I /home/opam/.opam/5.0/lib/asetmap -I /home/opam/.opam/5.0/lib/astring -I /home/opam/.opam/5.0/lib/atdgen -I /home/opam/.opam/5.0/lib/atdgen-runtime -I /home/opam/.opam/5.0/lib/base/caml -I /home/opam/.opam/5.0/lib/base64 -I /home/opam/.opam/5.0/lib/base64/rfc2045 -I /home/opam/.opam/5.0/lib/bigarray-overlap -I /home/opam/.opam/5.0/lib/bigstringaf -I /home/opam/.opam/5.0/lib/biniou -I /home/opam/.opam/5.0/lib/bos -I /home/opam/.opam/5.0/lib/bytes -I /home/opam/.opam/5.0/lib/camlp-streams -I /home/opam/.opam/5.0/lib/cmdliner -I /home/opam/.opam/5.0/lib/cohttp -I /home/opam/.opam/5.0/lib/cohttp-lwt -I /home/opam/.opam/5.0/lib/cohttp-lwt-unix -I /home/opam/.opam/5.0/lib/conduit -I /home/opam/.opam/5.0/lib/conduit-lwt -I /home/opam/.opam/5.0/lib/conduit-lwt-unix -I /home/opam/.opam/5.0/lib/cstruct -I /home/opam/.opam/5.0/lib/csv -I /home/opam/.opam/5.0/lib/current -I /home/opam/.opam/5.0/lib/current/cache -I /home/opam/.opam/5.0/lib/current/term -I /home/opam/.opam/5.0/lib/current_git -I /home/opam/.opam/5.0/lib/current_incr -I /home/opam/.opam/5.0/lib/current_web -I /home/opam/.opam/5.0/lib/domain-name -I /home/opam/.opam/5.0/lib/duration -I /home/opam/.opam/5.0/lib/easy-format -I /home/opam/.opam/5.0/lib/eqaf -I /home/opam/.opam/5.0/lib/eqaf/bigstring -I /home/opam/.opam/5.0/lib/eqaf/cstruct -I /home/opam/.opam/5.0/lib/fmt -I /home/opam/.opam/5.0/lib/fpath -I /home/opam/.opam/5.0/lib/gitlab -I /home/opam/.opam/5.0/lib/gitlab-unix -I /home/opam/.opam/5.0/lib/inotify -I /home/opam/.opam/5.0/lib/inotify/lwt -I /home/opam/.opam/5.0/lib/ipaddr -I /home/opam/.opam/5.0/lib/ipaddr-sexp -I /home/opam/.opam/5.0/lib/ipaddr/unix -I /home/opam/.opam/5.0/lib/irmin-watcher -I /home/opam/.opam/5.0/lib/ke -I /home/opam/.opam/5.0/lib/logs -I /home/opam/.opam/5.0/lib/lwt -I /home/opam/.opam/5.0/lib/lwt-dllist -I /home/opam/.opam/5.0/lib/lwt/unix -I /home/opam/.opam/5.0/lib/macaddr -I /home/opam/.opam/5.0/lib/magic-mime -I /home/opam/.opam/5.0/lib/mirage-crypto -I /home/opam/.opam/5.0/lib/mirage-crypto-rng -I /home/opam/.opam/5.0/lib/mirage-crypto-rng/unix -I /home/opam/.opam/5.0/lib/multipart_form -I /home/opam/.opam/5.0/lib/multipart_form-lwt -I /home/opam/.opam/5.0/lib/ocaml-compiler-libs/common -I /home/opam/.opam/5.0/lib/ocaml-compiler-libs/shadow -I /home/opam/.opam/5.0/lib/ocaml/compiler-libs -I /home/opam/.opam/5.0/lib/ocaml/str -I /home/opam/.opam/5.0/lib/ocaml/threads -I /home/opam/.opam/5.0/lib/ocaml/unix -I /home/opam/.opam/5.0/lib/ocplib-endian -I /home/opam/.opam/5.0/lib/ocplib-endian/bigstring -I /home/opam/.opam/5.0/lib/parsexp -I /home/opam/.opam/5.0/lib/pecu -I /home/opam/.opam/5.0/lib/ppx_derivers -I /home/opam/.opam/5.0/lib/ppx_deriving/api -I /home/opam/.opam/5.0/lib/ppx_deriving/runtime -I /home/opam/.opam/5.0/lib/ppx_deriving_yojson -I /home/opam/.opam/5.0/lib/ppx_deriving_yojson/runtime -I /home/opam/.opam/5.0/lib/ppx_sexp_conv/runtime-lib -I /home/opam/.opam/5.0/lib/ppxlib -I /home/opam/.opam/5.0/lib/ppxlib/ast -I /home/opam/.opam/5.0/lib/ppxlib/astlib -I /home/opam/.opam/5.0/lib/ppxlib/print_diff -I /home/opam/.opam/5.0/lib/ppxlib/stdppx -I /home/opam/.opam/5.0/lib/ppxlib/traverse_builtins -I /home/opam/.opam/5.0/lib/prettym -I /home/opam/.opam/5.0/lib/prometheus -I /home/opam/.opam/5.0/lib/prometheus-app -I /home/opam/.opam/5.0/lib/re -I /home/opam/.opam/5.0/lib/result -I /home/opam/.opam/5.0/lib/routes -I /home/opam/.opam/5.0/lib/rresult -I /home/opam/.opam/5.0/lib/seq -I /home/opam/.opam/5.0/lib/session -I /home/opam/.opam/5.0/lib/session-cohttp -I /home/opam/.opam/5.0/lib/session-cohttp-lwt -I /home/opam/.opam/5.0/lib/sexplib -I /home/opam/.opam/5.0/lib/sexplib0 -I /home/opam/.opam/5.0/lib/sqlite3 -I /home/opam/.opam/5.0/lib/stdlib-shims -I /home/opam/.opam/5.0/lib/stringext -I /home/opam/.opam/5.0/lib/tyxml -I /home/opam/.opam/5.0/lib/tyxml/functor -I /home/opam/.opam/5.0/lib/unstrctrd -I /home/opam/.opam/5.0/lib/unstrctrd/parser -I /home/opam/.opam/5.0/lib/uri -I /home/opam/.opam/5.0/lib/uri-sexp -I /home/opam/.opam/5.0/lib/uri/services -I /home/opam/.opam/5.0/lib/uutf -I /home/opam/.opam/5.0/lib/yojson -intf-suffix .ml -no-alias-deps -open Current_gitlab__ -o plugins/gitlab/.current_gitlab.objs/native/current_gitlab__Api.cmx -c -impl plugins/gitlab/api.pp.ml)
# File "plugins/gitlab/api.ml", line 463, characters 2-18:
# 463 |   Ptime.of_rfc3339 str |> function
#         ^^^^^^^^^^^^^^^^
# Error: Unbound module Ptime

And I've no idea which package provided the ptime dependency (this unclear to me what to restrict) /cc @tmcgilchrist

[tmcgilchrist]

And I've no idea which package provided the ptime dependency (this unclear to me what to restrict) /cc @tmcgilchrist

That's coming from ocurrent projects (current_github and current_gitlab), which needs a lower bound on ptime. Specifically we use Ptime.of_rfc3339 which appeared in ptime.1.1.0. Incorrect, we don't specify a dependency on it at all. Preparing a metadata PR to fix the issue for ocurrent. @hannesm

[tmcgilchrist]

The deprecations that stihl is warning about were introduced in caqti.1.8.0. See https://github.com/paurkedal/ocaml-caqti/blob/v1.8.0/caqti/lib/caqti_request.mli#L285

[hannesm]

Thanks @tmcgilchrist for your ocurrent investigations.

I don't quite understand what you mean with sihl and caqti. From my impression, the error in sihl is: Unbound module Mirage_crypto -- but I can't figure out how mirage-crypto is supposed to be in the dependency chain. The deprecation warnings I do ignore....

[hannesm]

Any guidance what to do? From my perspective, it is sad that sihl 0.1.x does no longer work, but on the other hand, they specify rather weakly dependencies. Also, this is quite old, with version 3.0.5 being the last released one. So my investment in trying to find out which constraint to add to sihl 0.1.x is near its end. Would it be fine for the repo maintainers to add a "conflict: conduit-lwt-unix >= 6.2.0" (and hope that fixes it)?

[kit-ty-kate]

Would it be fine for the repo maintainers to add a "conflict: conduit-lwt-unix >= 6.2.0" (and hope that fixes it)?

yeah i agree. I've tried to figure it out but also failed to do so in a reasonable amount of time. I'll open a PR to do just that.

[hannesm]

Thanks @kit-ty-kate. IMHO this is fine to be merged! :)

[kit-ty-kate]

Sadly it's still not fine. I've spent some time tracking down the infinit loop in ocluster's tests and it seems to be coming from that error:

     capnp-rpc [WARNING] Uncaught exception: The default generator is not yet initialized. 
To initialize the RNG with a default generator, and set up entropy collection and periodic reseeding as a background task, do the following:
  If you are using MirageOS, use the random device in config.ml: `let main = Mirage.foreign "Unikernel.Main" (random @-> job)`, and `let () = register "my_unikernel" [main $ default_random]`. 
  If you are using Lwt, execute `Mirage_crypto_rng_lwt.initialize ()` at startup. 
  If you are using Async, execute `Mirage_crypto_eng_async.initialize (module Mirage_crypto_rng.Fortuna)` at startup. 
  If you are using Eio, execute in one of the fibers `Mirage_crypto_rng_eio.run (module Fortuna) env` (`env` from `Eio_main.run`).
   
  Otherwise, there is no periodic reseeding. For an initial seed from getrandom(), execute `Mirage_crypto_rng_unix.initialize ()`. You can use `Mirage_crypto_rng.accumulate` and `Mirage_crypto_rng.reseed` to reseed the RNG manually.

cc @talex5 do you know where the missing Mirage_crypto_rng_lwt.initialize () is coming from?

[kit-ty-kate]

Ok i understand what's happening (and i think it might be holding true for the rest of the revdeps failures we had previously):

conduit-lwt-unix now optionally depends on tls-lwt instead of tls.lwt. Previously this was automatically implied because those packages had lwt in their dependency tree already, but now this has to be installed manually.
For the previous revdeps failures this meant it brought all the dependencies of tls.lwt to the available modules.
And for ocluster it means the Mirage_crypto_rng_lwt.initialize isn't called by one of the dependency (previously tls.lwt) already: https://github.com/mirleft/ocaml-tls/blob/15dc528852c1cc5533535f0beb7e0a7a7ef7868a/lwt/tls_lwt.ml#L279

I'll open a PR

[kit-ty-kate]

Thanks

[hannesm]

Thanks a lot @kit-ty-kate for tracking this down! :) It shows that splitting packages is a huge pain (and still not clear whether it was worth the effort).