Auto-generated build-depends-preemptive, and --follow-pvp flag

[snoyberg]

Discussed via email with @23Skidoo, I'll try to capture what we wound up at, but if I put in anything you disagree with, please correct me.

The goal is this: the contentious and time-consuming part of the PVP is preemptive upper bounds: putting upper bounds to exclude versions of packages not yet released, since they might break the build. As opposed to known upper bounds, it is often desirable to relax these constraints, as a valid build may still exist. Additionally, these preemptive bounds can be fully determined by simply inspecting upload dates of packages.

So the proposal is this. In addition to having the build-depends field in the cabal file, we will have a new field with an identical syntax called build-depends-preemptive. This field will be automatically generated by cabal (question: at what point would it be generated? I'm unclear on that) using the following PVP-inspired rule:

• For each package dependency, add a requirement that the version used is less than the next major version of the package vs what's available at the time of generation.
• Example: if there's a dependency in build-depends on foo >= 1.2, and the newest version of foo on Hackage is 1.5.3, then build-depends-preemptive will have foo >= 1.2 && < 1.6.

This field will not be user editable.

We would then have an additional flag- strawman-proposal for naming being --follow-pvp - which would use the build-depends-preemptive field instead of build-depends. This leads to essentially three modes of operation regarding upper bounds:

• No extra flags: cabal will try to use newer versions of packages relative to what existed at the time the package was uploaded.
• --follow-pvp: cabal will only use versions of dependencies available at the time of upload.
• --allow-newer: completely ignore all upper bounds, whether they are preemptive or specified by the user. This would still be vital for dealing with older packages which have preemptive upper bounds in their build-depends field.

I won't include proposed changes to the PVP in this issue, but I believe this will obviously have ramifications on recommendations of the PVP.

[23Skidoo]

/cc @dcoutts

@snoyberg

Thanks for writing a summary.

(question: at what point would it be generated? I'm unclear on that)

My idea was to generate build-depends-preemptive on the server after upload (since it's possible to upload using the Web UI, we can't do it in cabal upload itself).

[snoyberg]

Ahh, very nice. That seems like it would allow for two very nice features:

• Backfilling build-depends-preemptive for existing packages.
• Allow someone to relax the preemptive bounds after upload when a new version is released.

[ygale]

Why do we require this to be generated on the server? Cabal has access to all package upload time stamps locally, as the file modification time stamps in 00-index.tar. In the past we have kept most logic inside cabal itself and kept the server protocol extremely simple, making it very easy to write useful applications like @snoyberg 's yackage. Why do we want to change that?

[ygale]

I am a big +1 on this feature, and @snoyberg 's algorithm is a great idea, even though I totally disagree with just about every statement of @snoyberg in his justification for the feature in the first post.

This feature is simply the dual of --allow-newer. As we continually improve cabal's ability to infer a correct build plan automatically, we recognize that automatic inference is not always possible for various reasons. So we also continually add features that allow developers to regain control and help cabal find a correct build plan when needed, with a minimal amount of effort. This proposal would be a powerful boost to that.

This feature would save us huge amounts of time (and pulled-out hair). It's true that we can limit versions explicitly using constraints, and record those solutions in the package-local cabal.config. But figuring out what that explicit bound should be for each of many dependencies is a daunting task. Having the ability to let cabal make an educated guess for us that will often be correct would be fantastic.

I really would want it also to be possible to specify this for individual packages - just as I would want that for --allow-newer.

I propose the following syntax:

In constraints, < inferred means less than the version inferred by @snoyberg 's algorithm for this package (also allow the common misspelling infered, or at least say "perhaps you meant...").

As an option, --infer-bounds on the command line or infer-bounds: true in cabal.config means add the additional constraint < inferred for all dependencies.

As for the admitted straw man --follow-pvp: Cabal doesn't follow or not follow PVP. It is package authors who either follow it or not. While it is true that package authors not following PVP is one of the reasons - probably the most common one - that this feature is sorely needed, it is not the only reason. Sometimes we tighten the upper bound of a dependency as a way of more easily controlling a number of its sub-dependencies at once, even though the dependency we actually constrained is perfectly compliant with PVP. This feature could be useful for that, too.

[23Skidoo]

@ygale

I really would want it also to be possible to specify this for individual packages - just as I would want that for --allow-newer.

You know about the --allow-newer=text,bytestring,mtl syntax, right?

[ygale]

No I didn't. Thanks, that's great news. We're usually not on the latest cabal - that would drive our release team crazy - and we very rarely need to ignore upper bounds rather than tighten them. But it's good to know about that, and I'm sure we'll use it. And that means there's no reason not to allow per-package inferring of upper bounds, by symmetry.

[23Skidoo]

@ygale
Regarding the < inferred proposal: I don't see why this special syntax is better than just omitting the upper bound. The idea is that --follow-pvp/--infer-bounds (whatever we choose to call it) will allow package authors to start dropping manual pre-emptive upper bounds from build-depends (PVP will have to be updated to allow that). Users will then have a choice between enabling/disabling pre-emptive upper bounds.

I agree that enabling --follow-pvp selectively could be useful. It should be also possible to ask Cabal to use inferred bounds for all packages except a given subset (e.g. --infer-bounds --no-infer-bounds=a,b,c).

In the past we have kept most logic inside cabal itself and kept the server protocol extremely simple, making it very easy to write useful applications like @snoyberg 's yackage. Why do we want to change that?

It makes sense to compute this information once and cache it. Also see the pros @snoyberg mentioned. Yackage will still work, it's just that packages uploaded to it won't have pre-emptive upper bounds inferred.

[ygale]

The term "preemptive upper bounds" is misleading and affected - it is an attempt to delegitimize PVP. Upper bounds don't "preempt" anything - they are an important part of the semantics of a dependency specification (and no, they can't be calculated automatically, although @snoyberg 's algorithm is a good heuristic that will often be correct), and it's then up to cabal what to do with that information. I am strongly opposed to changing PVP to allow omitting upper bounds. Packages with cabal files that omit upper bounds are broken: they promise that they will support all future versions of a dependency forever, no matter what changes are made to it. I won't go on any more about this here (unless you insist) because I don't think this is the forum for that discussion. The reason for adopting the proposal in this issue must not be a sneaky way to force upon the community a massive change to PVP, which is a far broader issue and certainly cannot be decided in a cabal-install GitHub issue. That would truly be "preemptive". Instead, the reason is to make cabal-install more powerful and flexible, which it would certainly do.

It makes sense to compute this information once and cache it.

Since most package authors do comply with PVP, this information would rarely be needed. We probably would use this feature as much as anyone, and we only need it for the minority of PVP-broken packages, plus a few others. This is a calculation that takes a few CPU cycles per package-version. Is it so important to save a few cycles to make such a major change to cabal/hackage architecture? There are many other simpler designs, including various kinds of local caching. My opinion is that anything other than local calculation on demand would be premature optimization. If we see that this creates a performance issue, we can think about what to do then.

see the pros @snoyberg mentioned.

Backfilling build-depends-preemptive for existing packages.

Local calculation would support all packages and versions. No backfilling is needed.

Allow someone to relax the preemptive bounds after upload when a new version is released.

The proposed feature is a cabal-install feature. It is orthogonal to the issue of allowing package descriptions on hackage to be edited. That entanglement is another good reason why this feature should have nothing to do with the server.

Yackage will still work, it's just that packages uploaded to it won't have pre-emptive upper bounds inferred.

What you are saying is that yackage will work, but cabal-install will now be somewhat broken - the inferred upper bound feature will stop working. This feature is important to us.

[snoyberg]

@ygale Not only are you misreading what I've said here, you then say that your misunderstanding somehow represents malice on my part. I will furthermore state that- as I stated initially- this proposal came from @23Skidoo after seeing my work on a Hackage mirror with added upper bounds, not something I proposed. And further, I haven't stated that this proposal is in any way dependent on PVP changes! So please take a step back and reread what I've said before making any more accusations that I'm trying to be sneaky or misleading here; I've been complete direct in all of my motivations, and it's upsetting that you're trying to imply otherwise.

Regarding the term itself: preemptive upper bounds is completely accurate. You are preemptively stating that your package won't work with a new version, even though that isn't known to be true. If you think that this term somehow is wrong, and somehow biases people in the wrong way, then provide a new term that you think makes more sense.

But your mistaken comment here demonstrates the obvious need for a separate term, as your comments confuse the discussion by talking about "removing upper bounds" in an absolute sense, when that's not all what's been proposed.

and no, they can't be calculated automatically

If that is the case, then please provide an example of a preemptive upper bound which cannot be calculated automatically.

Since most package authors do comply with PVP,

I'd like to see a stat on that, every single analysis I've ever done has said the opposite. And this is really a core component of this proposal: people looking to get benefits from the PVP require broad Hackage support of the PVP, and having the preemptive upper bounds automatically generated ensures this.

The proposed feature is a cabal-install feature.

Then you misunderstood. This is talking about allowing someone to say "package foo-1.2 wasn't available when package bar was released, but since then, foo-1.2 was released, so let's relax the automatically determined upper bound for everyone."

That's not to say that I disagree with some of your arguments for this being more logically a client-side feature, but let's not see incorrect statements accepted simply to try and win an argument.

[ygale]

@snoyberg
No no. I'm really really sorry if I came off sounding like you or anyone else intended any malice; that's not at all what I intended. Obviously, everyone's intent is to make things work better.

I haven't stated that this proposal is in any way dependent on PVP changes!

You didn't say that, but @23Skidoo did:

The idea is that --follow-pvp/--infer-bounds (whatever we choose to call it) will allow package authors to start dropping manual pre-emptive upper bounds from build-depends (PVP will have to be updated to allow that).

And I must admit that I was quite shocked by that.

You are preemptively stating that your package won't work with a new version, even though that isn't known to be true.

That is not what an upper bound means in PVP. An upper bound is an estimate - not a statement of fact - by the package author of until what version that dependency is likely safe. It is clear from the discussion of "risk" in PVP that this is the intention, not being "preemptive" in any sense. It's not forbidding you from using later versions, just giving you information about when there starts to be some risk of incompatibility. Users can then decide what to do with that information (assuming they have a recent enough version of cabal-install that allows them to have a say in the matter).

And this is really a core component of this proposal: people looking to get benefits from the PVP require broad Hackage support of the PVP

No not really. The more good information we get the better, but now that cabal-install is so much more flexible about allowing me to set my own policies, we can deal with packages that get it wrong.

and no, they can't be calculated automatically

If that is the case, then please provide an example of a preemptive upper bound which cannot be calculated automatically.

One obvious example is tagged. That library underwent major version bumps (second component) on a constant basis for a while. Those version bumps were correct. But most packages using tagged only use the Tagged newtype, whose shape hasn't changed since version 0.0 in 2009. So < 1.0 has been a reasonable upper bound all along for those packages. It's extremely unlikely that @ekmett will change that type any time in the near future, and if he does, he would almost certainly bump the first component. Libraries that use the upper bound computed by your algorithm for tagged often cause us build problems. The deepseq library is like that, too, and there are others. The same thing can happen for just about any library if your particular usage is limited to a part of the library which is more stable than the rest of the library.

The opposite problem can happen with a library that is experiencing frequent minor version bumps that matter to you - like dependency version bumps that can mess up your build plan, or like when you are using internals of the dependency. Then you may want to use the third or fourth component, for example, to determine your upper bound.

having the preemptive upper bounds automatically generated ensures this

That's not how I would use this feature; we don't need that. I would use it as a huge time saver for dealing with particular packages whose omission of upper bounds causes us trouble.

Since most package authors do comply with PVP,

I'd like to see a stat on that, every single analysis I've ever done has said the opposite.

I admit that my statement is anecdotal, based on the few hundred or so packages that I happen to use frequently. Furthermore, we use almost exclusively older versions of packages, from before all the screaming about PVP began in earnest. So things may very well have changed. But again, I don't think that's really an issue. I hope that once things settle down, and it really starts sinking in how much better cabal is now, people will start providing better upper bound estimates again.

The proposed feature is a cabal-install feature.

Then you misunderstood. This is talking about allowing someone to say "package foo-1.2 wasn't available when package bar was released, but since then, foo-1.2 was released, so let's relax the automatically determined upper bound for everyone."

Indeed, I misunderstood. I thought the goal is to make it easier to deal with packages that omit upper bounds. And then there was a proposed implementation that involved server and protocol hackery. So I suggested a simpler implementation in a more traditional client-side fashion - which might not have been obvious if you didn't think of the tar file mod date trick.

I don't like the idea of Hackage providing a modified view of packages that were uploaded. Even if the lack of upper bounds in many packages is inconvenient for me, if that's what they published, that's what I should see. But gee-whiz, I sure would like to be able to apply your algorithm automatically sometimes.

[snoyberg]

@ygale I'm sorry, I really don't see the relevance of most of your response to what I wrote. Your comments on tagged are a criticism of the PVP's rules for placing upper bounds, which this proposal merely codifies. If you're opposed to those rules being codified, then it seems like you're really opposed to the PVP's statements regarding how upper bounds are supposed to be placed on packages. Your comments about three and fourth numbers similarly conflict with the PVP. In other words, your complaints don't belong in a feature discussion for cabal, they belong on the libraries mailing list about problems with the PVP.

I also don't think your explanation of how my term "preemptive upper bound" is wrong is convincing. You also haven't provided any preferred term. So I'll continue using my term, until a different one is agreed upon.

That's not how I would use this feature; we don't need that. I would use it as a huge time saver for dealing with particular packages whose omission of upper bounds causes us trouble.

This one I can't even start on... it's not about ensuring that the PVP works by adding upper bounds where they're missing, it's about being a time saver for adding upper bounds where they're missing?!?

---

Back to the topic at hand. This feature will allow people who want to rely on PVP upper bounds to get that benefit, even when package authors don't indicate those upper bounds. Will the upper bounds sometimes be overly restrictive? Yes, that's the nature of these upper bounds. That's why the proposal provides a means of relaxing them after-the-fact. Might this feature affect future discussions of changes to the PVP? Sure, why not, but that's not what's being discussed here.

I'd like to keep this discussion on track for the feature itself, not a place to attack someone's choice of term or argue over minutiae of the exact benefits of having upper bounds. We all accept that upper bounds are useful, this proposal is about an automated way of providing them when authors (for whatever reason) choose not to.

[ygale]

First of all, just to make sure: We are talking about PVP that is defined here:

http://www.haskell.org/haskellwiki/Package_versioning_policy

At the time, Simon's write-up of PVP seemed absolutely clear. But perhaps that was only because we were in the context of what was happening at that time. I'm beginning to suspect that all the recent noise about PVP is being caused by relative newcomers to the community who were not present then and who have completely misunderstood PVP.

[ygale]

This proposal consists of a major architecture change to the cabal/hackage system, and a user-visible feature that it would make available..

I am in favor of the user-visible feature, since it would be very helpful to me. And do I highly appreciate that @snoyberg made this proposal with users like me in mind; Thank you! But that feature could be implemented simply, without the re-architecture. The only claimed significant justification for the re-architecture depends on one particular view on a highly controversial topic. I completely agree that this is not the place to discuss controversial topics and make decisions on them.

So if we are not going to consider just the feature on its own, I don't see much point in continuing.

[dcoutts]

See also #3729 for a similar idea though with a different mechanism.

[jneira]

I guess this somehow RFC was not formally or informally accepted nor rejected.
A whole new build tool was implemented by the proposer thereafter and one of its characteristics was just make (upper) bounds less necessary, one of the goals of this issue.
So I am gonna label this as historical interestengi and close it.