Impact assement of breakage caused by mtl-2.3 #136
Comments
|
There are also these affected packages which have patches in head.hackage: |
|
|
Yes, but I think we can support the migration effort. |
|
Is the README out of date? I'm pretty sure I'm reading
|
|
To be precise, CLC governs but not maintains It was known for a long time (e. g., haskell/mtl#74 (comment)) that this is indeed a very significant breakage. Judging from my experience it is even more painful for applications than for libraries. @david-christiansen it's a nice opportunity for HF to shine and help with development of migration tools, because sponsors of HF are most certainly to benefit greatly from them. |
|
I've opened ndmitchell/hlint#1460 to add a feature to ease stuff like this. I probably won't have the capacity to implement it though. |
|
@Bodigrim So the CLC is on-board and takes responsibility with shipping |
|
@mpickering As I said, CLC does not approve changes to To be fair, this is not a last-minute breakage at all. The change has been proposed back in 2019 and Please raise an issue at https://github.com/haskell/mtl/issues, that's a better place to discuss opportunities and solutions. |
|
Also to harp on one of the things that I often harp on, we should strive not to bundle breaking changes together! Questions:
One should be able to upgrade compiler without breaking changes to
Absolutely agreed. And I think the lower hanging fruit is this ensuring one can use either compiler with either library. |
It's already the case unless you depend on |
Personal opinion, but I'd advise against shipping a new major version of a library with a minor release of the compiler ;-) |
|
I'm confused as to what the issue is here @mpickering , since
We released a fixed version ( |
|
I want to highlight one odd (to me) thing from the initial post to this thread:
I mean yes, of course, the packages that will be affected most by a breaking change are definitionally those which have not been tracking any changes or getting any updates or attention from their maintainers! So I think that these qualities of the enumerated packages do not make the breakage more or less significant. Rather, they are just the qualities which, almost by the laws of nature, will necessarily be possessed by those packages which have not been updated to be compatible with a relatively long-released (but not shipped with ghc) update to a library near the root of the dependency tree. I draw no conclusions from this about any choice of what library to ship or anything else. I just want to note this observation. |
|
The critical question is which version of Recently we have been under a lot of pressure to reduce breaking changes with releases. The key point is that the choice that the choice of Therefore, we have to carefully consider whether the benefit of upgrading to @Bodigrim The scale of the impact of this change has been shielded until now because stackage snapshots fix the mtl version to the version shipped by GHC. I suppose I can only speak for myself but I wasn't aware of the impact this would have until a few months ago. @emilypi I don't feel like it's a very pragmatic viewpoint to take, if we are changing fundamental things in the ecosystem then we have to be proactive about understanding and mitigating the impact. @Ericson2314 We can't bump to a major version of a library in a point release for precisely the reason due to the amount of breakage it causes. @gbaz I think my point is Gershom is that if there are long-standing stable parts of the ecosystem (such as mtl) then changing those has to be taken with utmost care as otherwise the impact can be much more significant than modifying something which frequently changes. |
|
People are getting way too conservative. Anyone who wants to keep using unmaintained packages is welcome to keep using an old GHC version as well. That said, I'm a bit curious why |
|
I welcome that team GHC is being proactive about issues related to breaking changes. What would happen if we shipped [*] Perhaps the problem is related to Stackage specifically: it doesn't allow choice of versions, which superficially sounds like a good idea but in practice causes problems like this. |
|
Yes, Stackage is the problem. I don't think waiting another five years to upgrade |
|
Yes, this is exactly the point @tomjaguarpaw, we have to engage with this issue now so that when the release happens there's a record about this choice being carefully considered. and yes, that's also exactly right, you are only forced into |
I may still be missing something. It seems like it should be possible to patch all packages currently in Stackage so that they compile with both |
I think this is a great point. We could have pre-release meetings with CLC and some of the relevant library maintainers to decide on which boot libraries to bump and document that decision. I think this is a good idea in general, since I also remember some hiccups wrt the last unix pin in GHC. Formally, I think CLC and both core lib maintainers can only give suggestions, since it's ultimately GHC that bundles those libraries. Since none of those breaking changes to core libs require a proposal, CLC also can't demand the maintainers to perform any specific work wrt the breaking changes. From what I understand, you're asking for two things:
Wrt 1. I think you're clear if the mtl maintainers recommend to bump it for the next GHC release. Wrt 2. I think we could coordinate this on the stackage issue tracker maybe. That has worked well for the aeson migration, afair. @juhp? |
|
I don't think anyone has set out to systematically patch affected packages, since this is not an issue with |
OK, I think that's something that should be done. Perhaps no individual or body has formal responsibility for it, in which case we should try to find someone or some body who is willing. "We" here is "individuals who care" (including myself) since neither has anyone responsibility for appointing a responsible person. If others agree with me then perhaps we can generate enough momentum to gather volunteers, appoint leadership, etc.. |
|
Could this problem be solved by:
|
They won't. The problem is packages which are not actively maintained. Remember how many pings is sometimes required to make some (trivial) changes in a dependency. |
|
Ok, but it gives them a lot longer to be updated. So that will fix a portion of them. And the change in mtl is not exactly urgent. One might even argue that it can comply with the PVP (as was the original request) just using deprecation warnings and never removing the re-exports :-) |
The I say if GHC won't bundle |
|
FWIW I think the key thing to ease something like this is a quick-fix type thing rather than a warning. Another thing to keep in mind is that external tooling becomes available to people a lot sooner than GHC versions. LTS is still on 9.2 and some people still use 8.10. My personal view on all this as someone with no authority is that we should keep mtl-2.3 for 9.6 and drum up some volunteers to submit patches to libraries before Stackage upgrades nightly. I'd be happy to help with this. I've already submitted patches for most of the libraries in my work apps footprint. |
I think this is a great idea, but unfortunately I must point out it's been a great idea that people have had over at least the decade that I have been using Haskell, and it has never happened. My guess is it's not likely to happen, and certainly not soon enough to fix any particular example of breakage. |
Indeed, often in discussions like this hypothetical solutions are offered without anyone following through. I feel as well that often the effort to develop a quickfix can far outweigh the effort to go without it and write patches by hand. Though as someone who isn't familiar with the internals of HLS/hlint/retrie I can't really judge. Though in this instance I feel like it shouldn't take too much effort. HLS already has refine imports, which is a superset of this and HLint already has warning on badidents. |
|
@dcoutts Many people seem content to ignore warnings altogether until they turn into errors. A library that's not actively maintained won't produce any of those until someone not responsible for maintaining it runs into a problem. |
As the maintainer by default of HPDF, that's probably what will happen for this package. I don't even use it so I won't see any potential warning. |
|
@chshersh I understand your points, but hackage is not everyone's private repository. It's a shared community space. Sure, it doesn't have strong policies or guarantees, but the existence of hackage trustees and their power to apply GHC compat patches in maintainers absence is a very clear signal that this is true: hackage is a shared space. I don't know if that allows us to draw any conclusions or make any actual changes, but my opinion is that users who don't intend to maintain their packages anymore should either:
The main problem is that hackage trustees can't really work in batch-mode. It would be cool to have a proper notification system and be able to just apply the patches after a notice period runs out. Then we could get all those in before the deprecation period is over and schedule things accordingly. But I'm expecting this to be rather controversial. |
I wish more maintainers would follow my lead and define specific transition policies in advance of them actually being needed: https://github.com/tomjaguarpaw/haskell-opaleye/#backup-maintainers |
This is flawed reasoning. GHC itself is a package that has version bounds like any other package. There can be multiple supported solver plans for any given version of a package. It doesn't even require even different minor versions, so long as both The "one version of GHC <==> one bindist per platform <==> one build plan" thinking is precisely why we get these thundering herds of simultaneous version bumps. We need to break it. Ship more GHC bindists with multiple different library build plans, and this problem goes away entirely. |
|
I'm inclined to think @Ericson2314 is correct here. Forcing things to upgrade in lockstep is a source of huge fragility. I'm beginning to understand that Stackage itself, whilst seeking to avoid fragility due to failing build plans is adding another form of fragility in the form of being unable to upgrade some parts of the ecosystem without upgrading others. If we managed to decouple things more we'd have less of these problems. |
|
Though in fairness, the fact that there are at least multiple stackages per compiler is still a good first step. We need to start applying that same mentality to the packages which ship in a GHC bindist (which is a mini-stackage of sorts). And then can we get proper tick-tock where either libraries change, or the compiler changes, but never both! |
Hey... wait a moment. It's not like there's no work involved 😅 |
I completely agree here. However, I don't believe that the problem is shipping more binary distributions; to me that seems like an unsustainable path, both for users (which suddenly need to worry about which of the many subtly-different binary distributions they should be using) and GHC maintainers (which need to somehow exhaustively enumerate possible core library build plans). I rather think that the solution here is to loosen the requirement that packages linking against the |
|
Unless I'm misunderstanding something it's not linking to |
|
Oh for sure, users can just build what they need themselves is tantamount to making the cost per bindist quite low :).
However, while it is not feasible or desire to CI every possible build plan, the ones that are needed to do the "incremental breakge chain" I think are important. E.g. per this thread, I would definitely want to CI 9.6 with and without MTL 2.3. |
|
Conversely, I'm basically being driven out of the community by us not
updating things - very specifically mtl. You lose people either way, and if
you update we end up with better software and keeping active people.
…-davean
On Fri, Feb 24, 2023 at 6:55 AM Dmitrii Kovanikov ***@***.***> wrote:
Even if it means staying on mtl-2.2 for another 15 years.
—
Reply to this email directly, view it on GitHub
<#136 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABZLS5XA2DRIVE6ZU3ZFTLWZCOUJANCNFSM6AAAAAAVFS3CZA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Just as a datapoint, I checked the list at the top of against reverse-deps, and discovered that out of the 32 packages listed, roughly 19 had any current rev-deps (i.e. rev-deps bounded against at least the current version of the package). Of those, only 5-8 seemed to have a significant quantity of rev-deps, and so I think patches against solely those would resolve most of the potential issues.
(note these are revdeps from the haskellers site, not hackage, which also has revdeps. calculation differences mean the two won't tie out exactly -- eg hackage calculates hedis at 56 and sydtest at 24 -- in the latter case this is because hackage doesn't include testsuite-deps as revdeps, i believe) |
|
FWIW hedis, sydtest, and postgresql-persistent already have patches waiting to be merged. |
Thanks for sharing your perspective Davean. Could you elaborate a bit? Are you using Stackage or the |
|
@tomjaguarpaw I think @davean was replying by email to an older message in this thread (by email). I hope it is clear that the more reinstallability / interpolated bindists as needed thing I am thinking is supposed to allow us to have our cake and eat it too: lot's of being able to make breaking improvements, few individually painful migrations. |
|
Oh I can and do - I don't touch stackage because it locks things in and
makes me do more work. But I was waiting a large portion of a decade for
fixes that are now in mtl.
I needed those fixes, and carried duplicated code to get them. Those fixes
in mtl aren't from nowhere. The state of affairs where we don't update
things is quite bad.
And yes, I was replying via email.
…-davean
On Fri, Feb 24, 2023 at 12:43 PM tomjaguarpaw ***@***.***> wrote:
I'm basically being driven out of the community by us not updating things
- very specifically mtl
Thanks for sharing your perspective Davean. Could you elaborate a bit? Are
you using Stackage or the ghc package? If not, can't you use whatever
version of mtl you want, regardless of what GHC ships with?
—
Reply to this email directly, view it on GitHub
<#136 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABZLS4YGII3FZEDXRZZTF3WZDXM3ANCNFSM6AAAAAAVFS3CZA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
It's more common than one might think. They primarily fall into two categories: tools and plugins. In the former category we have things like In the latter we have things like
I agree that
As far as I can tell this isn't a GHC issue but rather a Stackage one. By its nature, Stackage currently requires a completely consistent package set. As long as this is the case, user packages link against Yes, although it seems to me that this it should be possible (not having discussed this idea with Stackage maintainers yet) for Stackage to slightly weaken their consistency requirement. That is, instead of having precisely one version per package, there could be a small set (e.g. |
Though it is occurring to me that getting CI going with e.g. multiple versions of
That sounds possible, I would think sticking to the more bindists / more reinstallability vision is simpler. Make it really easy for stackage to create their own bindists (or ghc-up to do so on their behalf) and the problem is also solved without slightly weakening anything. |
I'm not sure there's much we can do here. Afterall,
Perhaps. I don't have a strong opinion on this. |
|
Yeah that's the beauty of it, there is nothing but the "last mile" to do with MTL: CI wrangling to split out more build artifact variations. Maybe we can get it easy enough for users to build their own copies of GHC soon, but not in time for GHC 9.6. And this capability for GHC's own CI is something we'd want anyways for e.g. a post split- |
Yes, I agree. Sorry if my comment came across as implying something else. |
|
Dear all. I believe the questions directed to CLC have been sufficiently answered in #136 (comment) and #136 (comment). Please move to more appropriate venues to explore further opportunities and solutions. If you want to discuss the evolution of If you want to discuss migration efforts and strategy, please collaborate with HF Stability Working Group at https://github.com/haskellfoundation/stability. If you want to discuss which version of |
To what degree do we need to be proactive, @mpickering? This is currently an undefined process. If an upstream maintainer announces that breaking changes are coming down the pipeline a year+ in advance, has an open issues policy in which to discuss the changes and impact, has a release candidate policy, a migration guide, and a meaningfully distant stretch of time for people to migrate within, then they have done all they can do to mitigate impact. What you seem to be implying with this thread, and the way in which you guys are discussing solutions, is that no breaking changes can be made if they're sufficiently deep enough without doing the legwork to move the entire impacted package set along with it. This is unscalable, and untenable. You cannot maintain people's packages for them, or submit patches to all your downstream dependencies without incurring an insane burden. That attitude is what contributes to people burning out every other year - you're doing an insane amount of work for other people to make up for the fact that they're not maintaining their packages! Tell me this: what is lost by letting the Hackage Trustees revise the affected packages by putting a correct upper bound on them? If Stackage is the issue, then Stackage has curators that can do roughly the same thing and coordinate changes to their stable set, as they did for This is not to belittle or get in your way here. I sympathize, but to be quite honest, we're not placing enough trust in the maintainers of these packages or existing processes that take care of these kinds of issues. |
I'm worried it might be GHC devs burning out, because they:
|
|
I'm talking about them specifically here. |
|
In light of:
we have concluded that this issue does not rise to the level where a reversion is necessary. We will move forward as originally planned, with 9.6.1 shipping |
This new release has caused major ecosystem breakage which I believe the CLC should be aware of.
A list of known affected packages are:
The migration is quite simple but I believe the breakage to be significant because many of these packages
Can the CLC please analyse this information and take ownership for the changes to this core library in the 9.6 release?
The text was updated successfully, but these errors were encountered: