cppo 1.6.0 (now uses jbuilder) #10047

mjambon · 2017-08-07T18:13:41Z

No description provided.

camelus · 2017-08-07T18:18:29Z

❗ opam-lint warnings `ff9de09`

cconv.0.4 has some warnings:
- warning 41: Some packages are mentionned in package scripts of features,
  but there is no dependency or depopt toward them: "doc"
These packages passed lint tests: bisect_ppx.1.2.0, commonjs_of_ocaml.0.1.0, cppo.1.6.0, cppo_ocamlbuild.1.6.0, hdf5.0.1.2, hdf5.0.1.3, jupyter.0.0.0, jupyter.0.1.0, lilis.0.2.1, ocp-index-top.0.4.1, opam-query.1.0, opam-query.1.1, opam-query.1.2, opam-query.1.3, pla.1.0, pla.1.1, ppx_bigarray.0.0.1, ppx_bigarray.0.1.0, ppx_bigarray.2.0.0, ppx_deriving.3.3, ppx_deriving.4.0, ppx_deriving.4.1, ppx_deriving_cmdliner.0.0.0, ppx_deriving_cmdliner.0.2.0, ppx_deriving_cmdliner.0.3.1, ppx_deriving_cmdliner.0.4.0, ppx_deriving_morphism.0.4, ppx_deriving_protobuf.2.4, ppx_deriving_protobuf.2.5, ppx_deriving_yojson.3.0, ppx_getenv.1.2, ppx_import.1.1, ppx_import.1.2, ppx_include.1.1, shcaml.0.2.0, sqlexpr.0.7.1, sqlexpr.0.8.0, touist.3.1.0, touist.3.2.0, touist.3.2.1

✅ Installability check (7288 → 7290)

new installable packages (2): cppo.1.6.0 cppo_ocamlbuild.1.6.0

avsm · 2017-08-09T10:52:50Z

This is the place where the revdeps start failing i think -- do we need upper bounds on bisect_ppx on cppo?

+- The following actions failed
| - build bisect_ppx 1.2.0
+- 
+- The following changes have been performed
| - install ppx_tools 5.0
+- 
[ERROR] The compilation of bisect_ppx failed at "make build".

#=== ERROR while compiling bisect_ppx.1.2.0 ===================================#
# context              2.0.0~beta3 | linux/x86_64 | ocaml-base-compiler.4.04.2 | file:///home/opam/src
# path                 ~/.opam/ocaml-base-compiler.4.04.2/.opam-switch/build/bisect_ppx.1.2.0
# command              make build
# exit-code            2
# env-file             ~/.opam/log/bisect_ppx-78-548b09.env
# output-file          ~/.opam/log/bisect_ppx-78-548b09.out
### output ###
# ocamlbuild -use-ocamlfind -no-links -byte-plugin -plugin-tag 'package(cppo_ocamlbuild)'  src/syntax/bisect_ppx.native  src/report/report.native src/bisect.cma src/bisect.cmxa  src/ocamlbuild/bisect_ppx_plugin.cma  src/ocamlbuild/bisect_ppx_plugin.cmxa
# ocamlfind ocamlc -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild myocamlbuild.ml /home/opam/.opam/ocaml-base-compiler.4.04.2/lib/ocamlbuild/ocamlbuild.cmo -o myocamlbuild
# + ocamlfind ocamlc -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild myocamlbuild.ml /home/opam/.opam/ocaml-base-compiler.4.04.2/lib/ocamlbuild/ocamlbuild.cmo -o myocamlbuild
# ocamlfind: Package `cppo_ocamlbuild' not found
# Command exited with code 2.
# Makefile:99: recipe for target 'build' failed
# make: *** [build] Error 10

mjambon · 2017-08-09T23:20:28Z

@rgrinberg I'm creating an opam package for cppo_ocamlbuild.

mjambon · 2017-08-09T23:41:36Z

Some existing packages for ocamlbuild plugins use the form ocamlbuild-* while others use *-ocamlbuild:

base-ocamlbuild
js_of_ocaml-ocamlbuild
ocamlbuild
ocamlbuild-atdgen
ocamlbuild-pkg
ocamlbuild-protoc
ocaml-migrate-parsetree-ocamlbuild

More than consistency with other packages, I think it's important to use the same name for the ocamlfind and for the opam package. Right now, the ocamlfind package is cppo_ocamlbuild and the OCaml module it provides is Ocamlbuild_cppo (flipped).

I suggest the following plan:

create opam package cppo_ocamlbuild
edit all opam packages that depend on both ocamlbuild and cppo, adding a dependency on cppo_ocamlbuild (bonus: ensure they use cppo_ocamlbuild)
notify the maintainers of these opam packages (especially if they maintain a copy of the opam package with the sources)

I'll proceed. @rgrinberg, let me know if you think it should be done differently.

distributed with the source package `cppo`.

mjambon · 2017-08-10T01:37:25Z

37 packages use both ocamlbuild and cppo. I need to automate this. I'll resume tomorrow.

rgrinberg · 2017-08-10T03:23:15Z

@mjambon sounds good. Although I think that:

edit all opam packages that depend on both ocamlbuild and cppo, adding a dependency on cppo_ocamlbuild (bonus: ensure they use cppo_ocamlbuild)

This will over-approximate the set of packages that need cppo_ocamlbuild.

I did a brief github search instead, and this is the list of packages I got:

I say we should just start with these and proceed from here.

rgrinberg · 2017-08-10T03:30:30Z

Let me warn the maintainers of the packages above about this change: @akabe @mfp @vbrankov @whitequark @choeger @smondet @Armael @reynir

The summary of the upcoming change:

If you're using the cppo_ocamlbuild ocamlbuild plugin, you will need to add cppo_ocamlbuild opam package to your list of dependencies.

@rgrinberg

according to the list of packages provided by @rgrinberg at #10047 (comment)

mjambon · 2017-08-10T23:28:55Z

I added a "cppo_ocamlbuild" {build} line to the packages @rgrinberg listed. Let's see how that works out.

of the ocamlfind packages for cppo_ocamlbuild.

mjambon · 2017-08-11T00:00:10Z

Now all these packages depend on cppo >= 1.6, via cppo_ocamlbuild, in order to avoid a double installation of the ocamlfind packages for cppo_ocamlbuild.

avsm · 2017-08-11T13:21:23Z

Looks like zmq failed for unrelated reasons to this PR (uint64.h not found). Not a blocker

avsm · 2017-08-11T13:22:30Z

One more package: bisect_ppx needs the dependency

#=== ERROR while compiling bisect_ppx.1.2.0 ===================================#
# context              2.0.0~beta3 | linux/x86_64 | ocaml-base-compiler.4.04.2 | file:///home/opam/src
# path                 ~/.opam/ocaml-base-compiler.4.04.2/.opam-switch/build/bisect_ppx.1.2.0
# command              make build
# exit-code            2
# env-file             ~/.opam/log/bisect_ppx-79-548b09.env
# output-file          ~/.opam/log/bisect_ppx-79-548b09.out
### output ###
# ocamlbuild -use-ocamlfind -no-links -byte-plugin -plugin-tag 'package(cppo_ocamlbuild)'  src/syntax/bisect_ppx.native  src/report/report.native src/bisect.cma src/bisect.cmxa  src/ocamlbuild/bisect_ppx_plugin.cma  src/ocamlbuild/bisect_ppx_plugin.cmxa
# ocamlfind ocamlc -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild myocamlbuild.ml /home/opam/.opam/ocaml-base-compiler.4.04.2/lib/ocamlbuild/ocamlbuild.cmo -o myocamlbuild
# + ocamlfind ocamlc -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild myocamlbuild.ml /home/opam/.opam/ocaml-base-compiler.4.04.2/lib/ocamlbuild/ocamlbuild.cmo -o myocamlbuild
# ocamlfind: Package `cppo_ocamlbuild' not found
# Command exited with code 2.
# Makefile:99: recipe for target 'build' failed
# make: *** [build] Error 10

avsm · 2017-08-11T13:22:58Z

and cconv and commonjs_of_ocaml

avsm · 2017-08-11T13:23:31Z

and ppx_deriving and pla

avsm · 2017-08-11T13:23:49Z

you can see the revdeps list in https://ci.ocaml.io/ocaml/opam-repository/pr/10047?test=1%20V1.2%20Build

@rleonid

cc @rleonid

@c-cube

cc @c-cube

@AngryLawyer

cc @AngryLawyer

@whitequark

cc @whitequark

@modlfo

cc @modlfo

@Drup

cc @Drup

cc @touist

mjambon · 2017-08-11T22:06:16Z

Note to the opam package maintainers listed below:

I just modified opam files for the following opam packages, in order to add cppo_ocamlbuild as a dependency since it no longer comes with the cppo opam package. You may have to propagate the change, if you maintain the authoritative copy of the opam file elsewhere than in opam-repository.

The affected packages and maintainers are:

bisect_ppx @rleonid
cconv @c-cube
commonjs_of_ocaml @AngryLawyer
ppx_deriving @whitequark
pla @modlfo
lilis @Drup
touist @touist

Drup · 2017-08-12T10:18:58Z

Looking good, thanks!

rleonid · 2017-08-12T15:52:24Z

cc @aantron

aantron · 2017-08-12T17:31:55Z

I suspect Lwt 2.7.1 and Lwt 3.0.0 are also affected by this.

akabe · 2017-08-17T07:48:24Z

Thanks.

The cause of the travis-ci errors is reported at issuu/ocaml-zmq#43, and it was repaired in stdint 0.4.2 #10093. The constraint stdint {>= '0.4.2'} should be added to zmq packages, but installation of the latest stdint will fix the build error, i.e., you can fix it by merge the current master of this repository and restart the builds on travis-ci.

avsm · 2017-08-17T12:50:38Z

Does anything else need to happen with Lwt as per @aantron's comment? I can do some manual testing tomorrow and merge if noone else gets there first.

rgrinberg · 2017-08-17T13:16:50Z

Thanks Anil, I think that should be all that's required. I'd update this PR myself but don't have push rights to it

…

On Thu, Aug 17, 2017, 9:50 PM Anil Madhavapeddy ***@***.***> wrote: Does anything else need to happen with Lwt as per @aantron <https://github.com/aantron>'s comment? I can do some manual testing tomorrow and merge if noone else gets there first. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10047 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIe-6n06SWTJLZPLZl3drhnAqAzI-s9ks5sZDcggaJpZM4OvxOo> .

mjambon · 2017-08-17T18:04:19Z

Sorry guys. I just merged master into this branch. Let's see what Travis says.

rgrinberg · 2017-08-21T15:23:19Z

@avsm this looks good. the failure in opam-query seems unrelated.

avsm · 2017-08-22T12:20:29Z

Thanks! You may want to announce this on https://discuss.ocaml.org, where we have a Community category and an announce tag for this purpose.

mjambon · 2017-08-23T02:09:19Z

Good job everyone and thank you for your patience!

jvillard · 2017-08-23T09:54:16Z

A bit of a side-note: this change broke the compilation of some other project, and breaks the only solution I currently know of for releasing source code that would keep compiling in the face of changes to opam. I'm curious what you think could be done to avoid this, if anything.

Background: Infer uses opam-lock to freeze the versions of opam packages to some "known good" versions. (There's also an "opam" file to serve as the basis for updating the opam.lock file.) The opam.lock file was introduced in reaction to several breakages we've had where a change in opam would turn out to break compilation one way or the other. Worse, these breakages could happened in released versions of infer, which then need a rerelease just to keep them compiling after opam update (for the record, this issue prompting this release)

Puzzle for future consideration: Changing the dependencies of packages without bumping their versions breaks projects in the wild because their set of constraints may become unsatisfiable with no user action save from opam update (or getting a fresh opam) (eg, cppo.1.5.0 and ppx_deriving.4.1).

rgrinberg · 2017-08-23T14:40:39Z

Jules Villard <notifications@github.com> writes:

A bit of a side-note: this change broke the compilation of some other project, and breaks the only solution I currently know of for releasing source code that would keep compiling in the face of changes to opam. I'm curious what you think could be done to avoid this, if anything.

Sorry about that, we did try to and minimize the breakage. Would you mind pointing out the package that failed in this case?

Background: [Infer](https://github.com/facebook/infer) uses [opam-lock](https://github.com/rgrinberg/opam-lock) to freeze the versions of opam packages to some "known good" versions. (There's also an "opam" file to serve as the basis for updating the opam.lock file.) The opam.lock file was introduced in reaction to several breakages we've had where a change in opam would turn out to break compilation one way or the other. Worse, these breakages could happened in released versions of infer, which then need a rerelease just to keep them compiling after `opam update` (for the record, [this issue](facebook/infer#577) prompting [this release](https://github.com/facebook/infer/releases/tag/v0.9.5))

opam-lock was my original attempt at working around this core issue that you're pointing out. The goal was that fixing the versions would prevent the opam ground moving underneath your feet. Unfortunately, it's clearly only a partial solution. Firstly, because opam packages are very much mutable. But also because when you use opam-lock, you really want to ignore all of opam's constraint solving and give it a concrete build plan to install. The way opam-lock could really address this issue is by also snapshotting the version of opam-repository in question. I remember briefly experimenting with that, but tabled that work because I couldn't figure out a way to make it work for opam 1.2 and 2.0. In the not so distance future (I hope) when opam 2.0 is released, I'd like to revisit this.

Puzzle for future consideration: Changing the dependencies of packages without bumping their versions breaks projects in the wild because their set of constraints may become unsatisfiable with no user action save from `opam update` (or getting a fresh opam) (eg, cppo.1.5.0 and ppx_deriving.4.1).

I agree and feel very much uneasy with how things are done today. I think I'd much prefer it if opam packages were immutable and people were forced to publish new versions of packages. But note that are disadvantages in this immutable world as well. For example, upgrades such as the one in this PR would leave a good chunk of packages that depend on the ocamlbuild plugin incompatible with the new cppo release.

jvillard · 2017-08-23T15:07:32Z

Thanks for your reply @rgrinberg. Infer is the project I was referring to.

I think I'd much prefer it if opam packages were immutable and people were forced to publish new versions of packages. But note that are disadvantages in this immutable world as well. For example, upgrades such as the one in this PR would leave a good chunk of packages that depend on the ocamlbuild plugin incompatible with the new cppo release.

Is the root cause that packages are usually optimistic about being forward-compatible with their dependencies then?

Improvised idle suggestion (apologies for going off-topic): There could be packaging versions on top of software versions, e.g. ppx_deriving.4.1-1 would have become ppx_deriving.4.1-2 here. This is more or less just an implementation detail of making packages immutable, however, so I'm not sure how much that would help on its own (and I have no idea how hard it would be to have this in opam 1 or 2). On top of this, packages could be forced to always specify bounded intervals for the versions of all their dependencies (eg, cppo >= 1.4.0 && cppo <= 1.5.0-1, so no more optimism). Then this PR would have bumped the "packaging version" of all the dependencies of cppo (which is a pain, admittedly) to bump their upper bounds to cppo <= 1.6.0-1. More generally, providing a new version of any package would bump all their reverse dependencies. Not sure how realistic this trade-off would be.

rgrinberg · 2017-08-23T21:30:08Z

Jules Villard <notifications@github.com> writes:

Is the root cause that packages are usually optimistic about being forward-compatible with their dependencies then?

I wouldn't say that's the root cause, but it does contribute to the issue.

Improvised idle suggestion (apologies for going off-topic): There could be packaging versions on top of software versions, e.g. ppx_deriving.4.1-1 would have become ppx_deriving.4.1-2 here. This is more or less just an implementation detail of making packages immutable, however, so I'm not sure how much that would help on its own (and I have no idea how hard it would be to have this in opam 1 or 2). On top of this, packages could be forced to always specify bounded intervals for the versions of all their dependencies (eg, `cppo >= 1.4.0 && cppo <= 1.5.0-1`, so no more optimism). Then this PR would have bumped the "packaging version" of all the dependencies of cppo (which is a pain, admittedly) to bump their upper bounds to `cppo <= 1.6.0-1`. More generally, providing a new version of any package would bump all their reverse dependencies. Not sure how realistic this trade-off would be.

I think you just reinvented the infamous "Package versioning policy" of Hackage. I think that this approach can alleviate this particular issue somewhat but I'm a fan of it either. * It's an incomplete solution. Following semver is a manual process and is error prone. Sneaking an accidental breaking change without a major version bump is a real possibility. * It creates a ton of churn for the maintainers of opam-repository and users. Maintainers will have to bump their upper bounds on every release of one of their dependency. I think that if we look at hackage, we will see that most package authors fail to do that in a timely fashion. The result is many bad build plans and constant churn for users to update upper bounds of packages they use. In my opinion there should simply be a way to refer to a snapshot of the opam-repository when specifying your lock file. I'm thinking of something similar to stack/stackage, which solved the build reproducibility for haskell.

jvillard · 2017-08-23T22:32:10Z

Good to know, thanks! Being able to freeze more aggressively to a particular complete snapshot sounds like a good solution.

dbuenzli · 2017-08-23T22:39:00Z

It creates a ton of churn for the maintainers of opam-repository

Somehow you could argue that this could be automated based on the results of the CI.

Per ocaml#10047 (comment) Resolves ocaml/opam#2977.

cppo 1.6.0 (now uses jbuilder)

2a6fa0e

mjambon mentioned this pull request Aug 7, 2017

New release? ocaml-community/cppo#43

Closed

New, distinct opam package for ocamlfind package cppo_ocamlbuild

7ab94a8

distributed with the source package `cppo`.

avsm added the needs opam-repository maintainer action label Aug 10, 2017

avsm mentioned this pull request Aug 10, 2017

easy-format 1.3.0, now uses jbuilder #10048

Merged

Add dependency on cppo_ocamlbuild to opam packages that need it,

b6f65f1

according to the list of packages provided by @rgrinberg at #10047 (comment)

Now cppo_ocamlbuild depends on cppo >= 1.6.0 to avoid a double install

393dc7d

of the ocamlfind packages for cppo_ocamlbuild.

akabe mentioned this pull request Aug 11, 2017

Release ocaml-jupyter 1.0.0 #10071

Merged

Martin Jambon added 7 commits August 11, 2017 14:01

Enforce bisect_ppx dependency on cppo_ocamlbuild

6e28513

cc @rleonid

Specify cconv 0.4 dependency on cppo_ocamlbuild.

3845f70

cc @c-cube

Add dependency on cppo_ocamlbuild for commonjs_of_ocaml

b042b7c

cc @AngryLawyer

Add dependency on cppo_ocamlbuild for ppx_deriving (3.3, 4.0. 4.1)

fc26e22

cc @whitequark

Add dependency on cppo_ocamlbuild for pla

61748f7

cc @modlfo

Add dependency on cppo_ocamlbuild for lilis

f37118d

cc @Drup

Add dependency on cppo_ocamlbuild for touist

fda05a1

cc @touist

Merge remote-tracking branch 'official/master' into cppo.1.6.0

ff9de09

rgrinberg mentioned this pull request Aug 21, 2017

Add cppo_ocamlbuild dependency whitequark/opam-query#3

Merged

avsm merged commit 228c3a5 into ocaml:master Aug 22, 2017

mjambon deleted the cppo.1.6.0 branch August 23, 2017 02:09

jvillard mentioned this pull request Aug 23, 2017

Docker build fails with Exception: Reqs_error facebook/infer#718

Closed

jberdine mentioned this pull request Aug 25, 2017

infer: fix build Homebrew/homebrew-core#17229

Closed

4 tasks

reynir mentioned this pull request Aug 31, 2017

Use cppo_ocamlbuild reynir/ocp-index-top#12

Closed

aantron mentioned this pull request Sep 3, 2017

Can't install lwt. ocaml/opam#2977

Closed

aantron added a commit to aantron/opam-repository-fork that referenced this pull request Sep 3, 2017

Fix Lwt cppo_ocamlbuild dependency

b1a962e

Per ocaml#10047 (comment) Resolves ocaml/opam#2977.

aantron mentioned this pull request Sep 3, 2017

Fix Lwt cppo_ocamlbuild dependency #10210

Merged

jvillard mentioned this pull request Sep 13, 2017

Cannot install Infer in a Redhat-based machine: fail at step updating OCaml facebook/infer#749

Closed

This was referenced Nov 1, 2017

failed to install utop.1.19.3: cppo_ocamlbuild not found ocaml-community/utop#220

Closed

added a cppo_ocamlbuild dependency to utop #10582

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cppo 1.6.0 (now uses jbuilder) #10047

cppo 1.6.0 (now uses jbuilder) #10047

mjambon commented Aug 7, 2017

camelus commented Aug 7, 2017 •

edited

avsm commented Aug 9, 2017

mjambon commented Aug 9, 2017

mjambon commented Aug 9, 2017

mjambon commented Aug 10, 2017

rgrinberg commented Aug 10, 2017

rgrinberg commented Aug 10, 2017 •

edited

mjambon commented Aug 10, 2017

mjambon commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

mjambon commented Aug 11, 2017

Drup commented Aug 12, 2017

rleonid commented Aug 12, 2017

aantron commented Aug 12, 2017

akabe commented Aug 17, 2017

avsm commented Aug 17, 2017

rgrinberg commented Aug 17, 2017 via email

mjambon commented Aug 17, 2017

rgrinberg commented Aug 21, 2017

avsm commented Aug 22, 2017

mjambon commented Aug 23, 2017

jvillard commented Aug 23, 2017

rgrinberg commented Aug 23, 2017 via email

jvillard commented Aug 23, 2017

rgrinberg commented Aug 23, 2017 via email

jvillard commented Aug 23, 2017

dbuenzli commented Aug 23, 2017

cppo 1.6.0 (now uses jbuilder) #10047

cppo 1.6.0 (now uses jbuilder) #10047

Conversation

mjambon commented Aug 7, 2017

camelus commented Aug 7, 2017 • edited

❗ opam-lint warnings ff9de09

✅ Installability check (7288 → 7290)

avsm commented Aug 9, 2017

mjambon commented Aug 9, 2017

mjambon commented Aug 9, 2017

mjambon commented Aug 10, 2017

rgrinberg commented Aug 10, 2017

rgrinberg commented Aug 10, 2017 • edited

mjambon commented Aug 10, 2017

mjambon commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

avsm commented Aug 11, 2017

mjambon commented Aug 11, 2017

Drup commented Aug 12, 2017

rleonid commented Aug 12, 2017

aantron commented Aug 12, 2017

akabe commented Aug 17, 2017

avsm commented Aug 17, 2017

rgrinberg commented Aug 17, 2017 via email

mjambon commented Aug 17, 2017

rgrinberg commented Aug 21, 2017

avsm commented Aug 22, 2017

mjambon commented Aug 23, 2017

jvillard commented Aug 23, 2017

rgrinberg commented Aug 23, 2017 via email

jvillard commented Aug 23, 2017

rgrinberg commented Aug 23, 2017 via email

jvillard commented Aug 23, 2017

dbuenzli commented Aug 23, 2017

camelus commented Aug 7, 2017 •

edited

❗ opam-lint warnings `ff9de09`

rgrinberg commented Aug 10, 2017 •

edited