Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release engineering 3.7.0 #1012

Closed
41 of 46 tasks
hannesm opened this issue Oct 30, 2019 · 10 comments
Closed
41 of 46 tasks

release engineering 3.7.0 #1012

hannesm opened this issue Oct 30, 2019 · 10 comments

Comments

@hannesm
Copy link
Member

hannesm commented Oct 30, 2019 •
edited
Loading

let's have another MirageOS release. My main motivation is to push through #1004 (but also have a decent solution to the hooks, i.e. #1010). Release schedule is tentative this week (e.g. before Nov 5th)!

issues and PRs to be addressed / merged

mirage-os-shim affected changes (#1010)

in respect to #1004, here's a list of PRs that needs to be merged and released (TODO: still some are missing, most notably dns, conduit, cohttp, irmin, git) all PRed now!

libraries I won't manage to include (help is welcome!) -- since they're not required by mirage (or mirage-skeleton), they can be released after the mirageos release

  • capnp-rpc
  • dns-forward
  • nbd
  • qcow
  • hvsock
  • datakit
  • protocol-9p
  • vpnkit

PRs

anything else we should include in the release? it'll be mostly compatible with 3.6.0 (mirage-types & mirage-types-lwt are only deprecated but still around, same should be for the hooks registration in the backends) -- we can remove the deprecated things with mirage 4.0

notably other changes:

  • cstruct lower bound is now at 4.0.0
  • lwt lower bound is at 4.0.0 (EDIT: I PRed changes to lower it from 4.4.0 to 4.0.0)
  • OCaml lower bound is at 4.6.0 (finally, safe-string for everyone)
@mato
Copy link
Contributor

mato commented Oct 30, 2019

anything else we should include in the release?

From mirage/mirage-os-shim#7 (comment): I'd like to have a way for applications (e.g. Canopy) depending on packages that make use of variants (e.g. digestif, checkseum) to build out of the box with the ocamlbuild-based Mirage 3.7.0.

We could:

  1. do nothing and document/use the _tags approach suggested by @dinosaure in Update to dev version of ocaml-git + ngit branch of irmin abbysmal/Canopy#95 (comment), if this is sufficient for all cases?

  2. or, as I discussed with @hannesm yesterday, it may be worth investigating if we can use OPAM per-package variables (or other metadata) to provide information about default variants to the mirage tool so that it could generate / pass this to ocamlbuild. WDYT?

@dinosaure
Copy link
Member

From mirage/mirage-os-shim#7 (comment): I'd like to have a way for applications (e.g. Canopy) depending on packages that make use of variants (e.g. digestif, checkseum) to build out of the box with the ocamlbuild-based Mirage 3.7.0.

For this specific point, the current problem is the orchestration of the build-system to choose which implementation we want to use at the end. As far as I can know, only dune provides a mechanism (a default implementation) which is chosen at the end and avoid a non-expecting linking-error.

META file/ocamlfind and ocamlbuild is not able to do that and we still need to explicitly choose the implementation with the _tags file (the more easy way).

The proposal on mirage/mirage-os-shim#7 use the dune feature for that and mirage/mirage-os-shim#8 follows the same distribution pattern without dune. Then, the mirage tool must be updated to orchestrate by itself (#1005) or by something else (mirage + dune + duniverse) to choose the right implementation.

According the current status of MirageOS (ocamlbuild), I did not (yet!) find a proper way to automatize that, plenty of solution exist (dune/duniverse/OPAM) but we need to find a right one.

As I said into mirage/mirage-os-shim#7, virtual_library/variant/linking_trick is really interesting but it is little bit far for us and will surely overlap the dunification of MirageOS. To avoid a complexity and keep small changes for the dunification purpose, I really would like to post-pone that.

@hannesm
Copy link
Member Author

hannesm commented Oct 30, 2019

I created an opam overlay https://github.com/hannesm/mirage-dev/tree/easy which is able to build mirage-skeleton (apart from examples requiring conduit-mirage, to be fixed soon). I also started releases to opam-repository (see above checklist for the current state), and will work on more in the following evenings.

@dinosaure
Copy link
Member

I just would like to clear a situation into mirage/mirage. Currently, by the choice made into #1010, the linking-trick does not exist anymore for nocrypto. As I explained into several issues, nocrypto uses an unusual way to check that the entropy engine is properly loaded:

https://github.com/mirage/mirage/blob/master/lib/mirage.ml#L847

May be we should, for the release of MirageOS 3.7, try to see if we can delete it.

@hannesm
Copy link
Member Author

hannesm commented Nov 1, 2019

@mato I'd be open to something along those lines, but don't know how to figure out the "default variant"

@dinosaure ok, that would be nice to have removed, but how? (i.e. i still think that code is useful, but am happy to be convinced that it is not needed anymore) -- AFAIU mirage needs to emit the Nocrypto_entropy_mirage.connect () call, which is only done if the nocrypto device is in the functoria dependency graph (either explicit, as the message you point to suggests, by adding abstract nocrypto to dependencies -- or implicit by using the nocrypto_random). But: if you use the stdlib_random as random device and also depend on nocrypto/tls, nocrypto's fortuna won't get seeded unless you add the dependency manually (and nocrypto uses its own Rng directly, not the mirageos selected one (which makes sense for a crypto library to have a decent rng)). To me it feels a bit scary to have stdlib_random and nocrypto in the same unikernel, I'd be in favour of avoiding this.

@dinosaure
Copy link
Member

To me it feels a bit scary to have stdlib_random and nocrypto in the same unikernel, I'd be in favour of avoiding this.

A possible hard solution should be to add a conflict in OPAM between mirage-random-stdlib and nocrypto. Generated OPAM file by mirage will go wrong and imply the user to use the nocrypto entropy engine xor mirage-random-stdlib. However, even if the solution is possible, I'm not really convince by that.

hannesm added a commit to hannesm/mirage that referenced this issue Nov 2, 2019
@hannesm
deprecations and adjustments for mirage#1004
5b60b07 
bump versions of packages according to release schedule mirage#1012
@hannesm hannesm closed this as completed Nov 3, 2019
@hannesm
Copy link
Member Author

hannesm commented Nov 4, 2019

NOTE: from a user perspective it may have been better if e.g. mirage-time 2.0.0 would provide a Mirage_time_lwt (containing [@@@deprecated] include Mirage_time) -- this way existing unikernels would still compile (with deprecation warnings). this could be retro-fitted by drafting interface releases (though I'm pretty busy this week). removal could then be in the next major (mirageos) version.

@mato
Copy link
Contributor

mato commented Nov 4, 2019

@hannesm Is this release 3.7.1 or 3.7.0? Your mailing list announcement says 3.7 and 3.7.1. Also, given that this has breaking changes, it would be good to announce also on https://mirage.io/ and https://discuss.ocaml.org/.

@hannesm
Copy link
Member Author

hannesm commented Nov 4, 2019

release 3.7.1 or 3.7.0? Your mailing list announcement says 3.7 and 3.7.1

both. the mirage/mirage repository contains 4 opam packages (mirage, mirage-types, mirage-types-lwt, mirage-runtime) -- 3.7.0 was only a mirage-runtiime release to be able to release mirage-solo5, mirage-xen, mirage-entropy, mirage-unix which depend on mirage-runtime >= 3.7.0.

Also, given that this has breaking changes, it would be good to announce also on https://mirage.io/ and https://discuss.ocaml.org/.

The amount of breaking changes is what caught my attention as well, and this morning I figured out a solution. I went through the interface packages and implemented (and deprecated) Mirage_x_lwt modules, each of which contains only include Mirage_x. CI build it (including reverse dependencies), and there's a set of releases to opam-repository which as well removes the earlier releases (without the compatibility module):

once these are merged, the current mirage-skeleton master (without merging the above mentioned PR) builds with both mirage 3.6.0 and 3.7.1. I'll then follow up on the mailing list and announce to discourse, and prepare a PR to mirage-www.

Sorry for the hickup and for a period of ~30 hours with a big breaking changes fallout. I learned from it :)

@mato
Copy link
Contributor

mato commented Nov 5, 2019

Thanks. This is a good solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mato @hannesm @dinosaure