Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add midstream bindist proposal #61

Merged
merged 3 commits into from Apr 23, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
281 changes: 281 additions & 0 deletions proposals/0000-midstream-bindists.md
@@ -0,0 +1,281 @@
# Midstream bindists

## Abstract

Bindists for the Haskell toolchain have been produced by upstream (the developers of each respective tool) for
a long time and many tools rely on these "official" bindists (e.g. GHCup and stack).

We propose here that bindists are built and maintained by the GHCup project, which
provides the main installation experiences in the Haskell ecosystem, removing the hard
dependency on upstream bindists entirely.

## Background

Historically, installers like GHCup and stack have used upstream bindists for mainly one reason: it's easy to do
so and doesn't require further efforts.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a pretty key point. ghcup has historically been a volunteer-run project, and so building all the bindists for all the tools would have been an unreasonable amount of work. Historically therefore the effort has been to push work upstream, with upstream projects building bindists for ghcup and contributing to ghcup-metadata.


However, using upstream bindists directly is extremely rare in the Linux world of distribution. Most distributions
build, package, test and curate binary packages themselves, not only because they have custom formats, but for
reasons of control, trust and quality.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add also for reasons of relationship. Most packagers for linux distributions do not have close relationships with the upstream projects. They don't generally have the option of working with upstream to get better bindists.

This is very different for ghcup, which is tightly embedded in the Haskell ecosystem and has a small set of tools that are distributed. In this setting it seems much more plausible to me to work with upstream to improve their bindists, rather than having two sets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most packagers for linux distributions do not have close relationships with the upstream projects.

I'm not sure. I've sent hundreds of build system patches to upstream projects during my Gentoo days and even got full commit rights to some of them.

And yet, we never used their bindists (because Gentoo is a source distro). So we focussed on what was actually relevant: improving their build systems, so we have an easier time packaging.


The relationship of GHCup and bindists has also been described in the blog post
[GHCup is not an installer](https://hasufell.github.io/posts/2023-11-14-ghcup-is-not-an-installer.html) recently.

## Problem Statement

From the perspective of a GHCup developer, there are several issues with relying on upstream bindists.

### Platform support

GHC and other tools have in the past dropped support for certain platforms either entirely or requested
the community to step up and do the work (e.g. on GHC CI).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be explicit: normally in an open-source project this is the default. Someone who cares has to step up.

GHC has paid release management from WT, so perhaps a more precise version of the complaint is "the GHC team decided not to ask their paid release manager to do this but instead requested that the community do it". So there's a disagreement about priorities.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree very strongly. Dropping an entire platform is not something trivial and needs:

  • a long grace period
  • a clear call for help, indicating support will be dropped unless someone stepped up

None of that happened. It was rather done ad-hoc/silently. That's not something GHCup considers good user experience. So here again seem to be very different goals/priorities.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was talking about the decision to not put effort towards it, not the communication issue. That I agree is problematic.


E.g. [GHC ARMv7 support was dropped silently without any call for help](https://gitlab.haskell.org/ghc/ghc/-/issues/21177#note_470440). Similarly, FreeBSD support just ceased to exist when the GHC FreeBSD CI stopped working. Later the community asked for a [revival](https://gitlab.haskell.org/groups/ghc/-/epics/5), but nothing signifcant has happened so far.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linked issue notes three (maybe four?) people who are interested, none of whom have cared enough to follow up except for @hasufell . My perception has been that the person who mostly wants FreeBSD support is @hasufell , and this doesn't convince me that there's much community demand.

Copy link
Contributor Author

@hasufell hasufell Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a chicken and egg problem: is the demand low, because the tooling is totally busted and end-users just use a linux VM?

Are windows users at 15%, because the tooling experience is great or because they switch to WSL and don't bother?

Why don't we just drop windows?

Again: I do not subscribe to this approach. We clearly have different goals/perceptions here.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What approach do you think we should use for deciding what platforms to support?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point is to have any approach at all.

My approach would be very conservative. Look, we still support i386, but suddenly decided to drop armv7? Despite clear evidence that there are still a lot of devices out there that run on armv7.

Similarly, FreeBSD is by far the most popular BSD. It was dropped not because there are inherent issues with supporting it code-wise, but due to CI shenanigans.

So we could start with:

  • don't suddenly drop platforms, ever
  • if a platform is difficult to support, evaluate what about it precisely is the issue
    • if it's CI -> there is a full-time devops working on GHC
    • if it's resources -> ask HF
    • if it's expertise (that's actually more a problem on windows) -> ask for help, find more devs
  • if none of the above worked, communicate the intent to drop the platform to the community and ask for volunteer help
  • if no one steps up, drop the platform in ~3 major releases time (1.5 years)

GHCup still produces bindists from time to time for FreeBSD, but e.g. the [HLS release manager for 2.5.0.0 recently refused to add FreeBSD bindists](https://github.com/haskell/ghcup-metadata/pull/159).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without getting into the details, my perception of the HLS situation was that FreeBSD support was provisional. We can argue about whether it should have been provisional or whether we tried hard enough to keep it going, but I think it's reasonable to expect low availability or abrupt removal of provisional things.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The situation was a bit more: I fixed the situation (missing GHC bindists, which is why CI was stale and buggy), but HLS release management was not interested in using my work.

This again shows different goals/priorities.


Similarly, stack used to have issues with Aarch64 bindists:

* https://github.com/commercialhaskell/stack/issues/5709
* https://github.com/commercialhaskell/stack/issues/5854
* https://github.com/commercialhaskell/stack/issues/5540
* https://github.com/commercialhaskell/stack/issues/5610
* https://github.com/commercialhaskell/stack/issues/5619
* https://github.com/commercialhaskell/stack/issues/6141
* https://github.com/commercialhaskell/stack/issues/6142

Recently, cabal-install had issues with i386 binaries and alpine, delaying a GHCup metadata PR:

* https://github.com/haskell/ghcup-metadata/pull/127#issuecomment-1766020410

These issues are frequent and so far the GHCup developers used to single handedly fix all those missing bindists manually
Copy link

@michaelpj michaelpj Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bracketing the quality issue for now, it seems to me that there is an issue of decision-making and control. That is, who decides what platform tool X supports?

Specifically, it seems like we sometimes have a situation like:

  • Tool X does not want to support platform Y
  • GHCup wants tool X to support platform Y

If upstream gets to decide what platforms it supports, it makes no sense to talk about GHCup "fixing" "missing" bindists. Upstream doesn't support it, you can be annoyed about that but it's their decision. The current situation where GHCup decides to add platforms that upstream does not want to support is risky: upstream will not be testing on those platforms nor likely to be responsive to issues on those platforms.

I'm sure it is not unheard of for packagers to add platform support, particularly if adding support doesn't require much modification to what upstream provides. But again, it seems strange to me in the context of GHCup, where there is a much closer relationship between the packager and upstream.

There certainly is an issue that Haskell tooling as a whole is inconsistent and wavering on what platforms it supports. I do think it would be good to have a shared policy on platform support that was changed more transparently. And again, since we are in a situation where our distributor (GHCup) has a close relationship with upstream, GHCup is certainly a stakeholder in platform support discussions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If upstream gets to decide what platforms it supports, it makes no sense to talk about GHCup "fixing" "missing" bindists. Upstream doesn't support it, you can be annoyed about that but it's their decision. The current situation where GHCup decides to add platforms that upstream does not want to support is risky: upstream will not be testing on those platforms nor likely to be responsive to issues on those platforms.

Yes, this proposal includes the testing of bindists (including unofficial ones). The idea would be to:

  • run the tests in CI and upload the results somewhere for end users to see (whether there are test failures or not)
  • have a mechanism to report these results to upstream developers
  • communicate test results to end-users in a compact way
  • but more importantly: have a way to run the entire test suites for all tools on the end-users system (which doesn't even work for GHC correctly... this is where the test suite must pass)

There certainly is an issue that Haskell tooling as a whole is inconsistent and wavering on what platforms it supports. I do think it would be good to have a shared policy on platform support that was changed more transparently. And again, since we are in a situation where our distributor (GHCup) has a close relationship with upstream, GHCup is certainly a stakeholder in platform support discussions.

So far, GHC developers have not asked the community about any of their platform decisions (dropping FreeBSD or armv7), nor were GHCup developers consulted. There was no call for help.

And yet, the unix package still supports these platforms, because there even have been OpenBSD users opening bug reports to make unix buildable again. There are clearly different perceptions and goals here and I've been communicating these for the past 2 years with zero improvement in platform support. So no, I do not believe that any other approach works, other than doing it downstream/midstream.

I have much more anecdotal evidence that there are widely diverging goals and perceptions, but I don't think these will aid to give more clarity to the proposal.

and provide them here: https://downloads.haskell.org/~ghcup/unofficial-bindists/

This is unfunded and significant work.

### Bindist maintenance

Sometimes, bindists are broken, e.g. for GHC there are a couple of instances:

* 9.0.2 shipping without profiling info: https://gitlab.haskell.org/ghc/ghc/-/issues/21841
* DESTDIR variable ignored by `make install`: https://gitlab.haskell.org/ghc/ghc/-/issues/19646

Sometimes, bindists have been built for very old version of linux distros and won't run well on newer linux versions.
This is currently a problem since Debian has removed ncurses5:

* https://github.com/haskell/ghcup-hs/issues/902
* https://answers.launchpad.net/ubuntu/+source/ncurses/+question/707838
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025964

For cabal, there has been the infamous hLock issue:

* https://github.com/haskell/cabal/issues/7313
* https://github.com/haskell/cabal/issues/7950

This shows that bindists, for current and historical versions, need continuous maintenance. However, upstream developers
so far have very rarely engaged in this type of maintenance work, pushing it down to GHCup. As an example, here are all the

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to invest this work, why not do it on the side of fixing the upstream bindists, given that we're working within the Haskell community? Reading this with my outsider hat on, this sounds crazy. Someone in the community is fixing the bindists but not pushing them upstream?

Additionally, fixing upstream bindists has a much bigger impact than just on GHCup, because upstream's bindists are used by other packagers, e.g. linux distributions, nixpkgs, etc. GHCup is not the only packager to end up in the situation of accumulating patches that somehow don't quite get upstream, see e.g. haskell.nix's patch list. Directing additional effort to getting this stuff properly pulled upstream benefits everyone distributing GHC and other tools.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone in the community is fixing the bindists but not pushing them upstream?

I tried to get i386 alpine bindists into GHC: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5213

After 2 years, it still hasn't happened yet (the issue got stuck while Ben was debugging some issues with statically linked GHCs, which I did not request at all).


Instead with the proposed framework, I believe that upstream developers would actually benefit from this, because:

  • they don't have to build releases at all anymore
  • if they still build releases, they can focus on smaller coverage (e.g. not build for every esoteric linux distro)
  • test results for "unsupported" or tier3 platforms can flow back to upstream developers, allowing ad-hoc fixes even if there's no continuous GHC CI support for said platform

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, fixing upstream bindists has a much bigger impact than just on GHCup, because upstream's bindists are used by other packagers, e.g. linux distributions, nixpkgs, etc.

FWIW I think all the packagers you mention build their own binary distribution, they don't repackage the upstream bindist.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, fixing upstream bindists has a much bigger impact than just on GHCup, because upstream's bindists are used by other packagers, e.g. linux distributions, nixpkgs, etc.

Yeah, I totally missed that. It's mostly incorrect. Linux distributions either don't use upstream bindists at all or only for bootstrapping.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think you're right. Nixpkgs at least only uses them for bootstrapping. That's not an unimportant use-case, though, and you certainly have a problem if your bootstrapping bindists don't work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not an unimportant use-case, though, and you certainly have a problem if your bootstrapping bindists don't work.

But for them it really doesn't matter where the bootstrap bindists come from and given that GHCup supports more platforms and architectures, they can just use those.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At that point it seems like we've removed all reason for upstream to bother producing bindists at all. Maybe we should say that - it would remove some repeat labour. But if nothing else it's probably good for upstream to at least notice some packaging problems quickly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, but that's up to upstream. We don't want to tell them what to do.

manually patched and re-packaged bindists that fix the DESTDIR bug outlined above: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc/curated/

This type of work also requires significant time.

### Quality gateway

There's something pleasing about upstream providing bindists: there's a perceived trust about it in the community,
e.g. the [haskell.org committee expressed concerns about GHCup changing bindists in the past](https://github.com/haskell-infra/www.haskell.org/issues/212#issuecomment-1272312911).

However, rarely do upstream developers have signifcant experience in redistribution, nor do they have the time to focus
on all the issues that come with it. Bindists are mostly provided "as-is" and support beyond what the release CI outputs is
left to midstream (GHCup, stack, ...).

Conceptually, it is a good idea to separate concerns: upstream provides the sources and has tested it. Distributions build the binaries,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like an argument against upstream bindists full stop? I don't hate this idea, except that it leads to duplication of work across packagers. While some packagers do use GHC's binary distributions (and they are very useful for bootstrapping), many build it from source. But it's very helpful to at least have the scripts for building binary distributions maintained and distributed by upstream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't hate this idea, except that it leads to duplication of work across packagers.

It is very trivial duplication if upstream has a good build system.

I can say this confidently from a decade of experience of being a linux distro developer. If the build system is wonky, the workload increases. Hadrian does not really shine well here (for people who are just interested in building release binaries).

validate that the program can be successfully built from source and make sure that the final artifacts pass the test suite.
This is good, because we want to know whether end users can build e.g. a functioning GHC from source. If only the release CI outputs
a GHC that passes the test suite, then something is fundamentally broken.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely agree that a bindist should not be a "big deal". Ideally you should be able to easily make your own that's just as functional as upstream's.


Distributors are often closer to the end-users and can provide additional support and efforts for the installation experience.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... but again, I do think this should go upstream as much as possible!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can. One of the goals of this proposal is to feed back such information (e.g. test reports) back to upstream, but in a more structured manner, while still maintaining strict separation of the projects.


### Security backports

Cabal has recently found to be vulnerable to [HSEC-2023-0015](https://github.com/haskell/security-advisories/blob/main/advisories/hackage/cabal-install/HSEC-2023-0015.md).

This vulnerability [had not been communicated to the GHCup team](https://github.com/haskell/security-advisories/issues/129) prior to disclosure,
causing high distress for a backport, since at the time of disclosure, cabal-3.6.2.0
was still 'recommended' by GHCup, since [cabal-3.10.2.0 is still broken on windows](https://github.com/haskell/cabal/issues/9334).

As such, GHCup developers needed a quick and efficient way to:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, there's a responsibility issue here. GHCup only "needs" to do this if it is GHCup's responsibility to fix it. If it's upstream's job, then it's reasonable for GHCup to mark the offending release as insecure and direct any ire to upstream. (This is what nixpkgs does, for example, and users can choose to ignore it if they want)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not align with my understanding of good re-distribution, so we have more evidence of diverging ideas.

The options were:

  • install a vulnerable version by default and warn the user
  • install a version by default that's broken on windows

I think those are poor choices and the security fix was simple enough to backport.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those are certainly poor choices, but I think it would be fair to say that's upstream's fault. It's only if GHCup regards it is its responsibility to provide a working recommended version of cabal that it should feel like it has to step up when upstream is failing to do so.

IMO it's is upstream's job to provide the working software. If they can't do that then we should direct users to yell at them, not try to fix things up afterwards. But perhaps this is a difference of perspective 🤷

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's only if GHCup regards it is its responsibility to provide a working recommended version of cabal that it should feel like it has to step up when upstream is failing to do so.

Yes, this is how most (if not all) distributions understand their work. ArchLinux may be a notable exception.

IMO it's is upstream's job to provide the working software.

If it was for upstream, they'd have GHCup recommend 3.10 by default and just expose users to known regressions. Again there is a clash of perspectives.


* patch cabal
* build release binaries for cabal
* ship the binaries

This had been done as a downstream release `3.6.2.0-p1` roughly a week after the disclosure.

Meanwhile, cabal upstream still has not finished their backport due to issues
with hackage dependencies: https://github.com/haskell/cabal/pull/9457
Comment on lines +112 to +113

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW there are no plans for a 3.6 or 3.8 patch release

https://mail.haskell.org/pipermail/cabal-devel/2023-November/010578.html

I am greateful for your patch release 3.6.2.0-p1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this highlights different priorities.

GHCup does not need to care about hackage dependencies etc and can just build an ad-hoc release with a backport, whether or not that exists on hackage.


This shows that a certain amount of independences from upstream CI and upstream workflow
is essential to fulfill swift security backports to potentially under-maintained
branches/versions.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, if we have the capacity to do this, and we're in a context where we can work with upstream, why would we not do it upstream?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because upstream had different priorities (making the backport release buildable via hackage), which impeded my priority: ship the fix as fast as possible.


### GHC nightlies

As a special case, I want to point out that GHC nightlies have been frequently broken beyond repair:
Copy link

@michaelpj michaelpj Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were GHC nightlies advertised as anything other than a provisional feature? If not, I don't see the problem with them having problems or low availability to begin with.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, but afaik, nightlies were funded (not sure whether by HF or as a courtesy of WT), but the project as a whole can be considered failed, since availability and platform issues were not designed properly.

The GHCup developers had spent a non-trivial amount of time to prepare ghcup for this feature (implementation), discuss the user interface design, help with testing and advertise it to the public, just to see that it was all botched in the end.

This did not reinforce my motivation to work on such joint projects in the future, when upstream (GHC) can just call the entire thing "off". So doing this midstream seems much more appealing.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not saying the nightlies went well, I'm just not sure how it supports your argument.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It supports my argument in so far that we have indication that GHCup and some upstream projects have very different ideas of:

  • what is a PoC deliverable
  • what is "best effort"/provisional

This has become even more clear with the last HLS release.


* https://gitlab.haskell.org/ghc/ghcup-metadata/-/issues/2
* https://gitlab.haskell.org/ghc/ghc/-/issues/24000

The breakage was left unattended and some bindists completely vanished from gitlab CI artifacts, because of misconfiguration.
Here's a graph of nightlies availability: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availabilitie-of-ghc-nightlies-via-ghcup?orgId=2

## Prior Art, Related Efforts and alternative solutions

So far, GHCup developers have tried to close the gap, doing signifcant work on upstream CIs and building bindists manually where
necessary.

Bindists for cabal-install are now produced by GHCup's own CI: https://github.com/haskell/ghcup-metadata/blob/develop/.github/workflows/cabal-release.yaml

HLS will likely follow shortly.

As mentioned before, there have been attempts to improve the coordination and collaboration across the entire Haskell
toolchain, view the tooling end-user experience in a holistic way and make decisions based on that end-user experience: https://github.com/haskellfoundation/tech-proposals/issues/48

The proposers believe that the community structure at the moment does not allow such an approach and there needs to be
significant work to align goals, perception and priorities. Otherwise there will be too much friction.

The most important currency in the open source volunteer world is **energy**. It is not code or technical effort. As such we
believe that the amount of saved energy by being more independent of upstream release processes and decisions far outweighs
potential costs of technical redundancy/duplication.

A future proposal may very well attempt to create a unified user experience across the entire Haskell toolchain
through joint management and collaboration. But that is not in the scope of this proposal and we have no concrete
idea how to achieve that.

## Technical Content

We propose here to start with the smallest step possible, to build the entire Haskell toolchain autonomically.
The way this will be implemented is to start a central GitHub repository that builds bindists for releases of:

- GHC
- HLS
- cabal
- stack

For the following platforms:

- FreeBSD x86_64
- Linux i386
- Linux x86_64
- Linux armv7
- Linux aarch64
- Darwin x86_64
- Darwin aarch64
- Windows x86_64

For the following Linux x86_64 distros:

- Debian
- Ubuntu
- Mint
- Fedora
- CentOS
- RedHat
- Rocky Linux
- Void Linux
- Amazon Linux
- Alpine

Linux i386, armv7 and aarch64 will be confined to Debian or Ubuntu.

These private runners will be made available to the whole Haskell GitHub org and as such benefit
other projects there as well (like HLS, Cabal, bytestring, etc.).

## Future work

The following ideas and goals are outside of the scope of this proposal, but are essential
to understand the broader mission and roadmap that motivated this proposal.

### Enhancements to bindist quality and installation experience

Further goals are:

* enhance the quality of the bindists by
- running the entire test suite for all of the tools
- having a mechanism to report test failures back upstream
- publishing test failures for end users to see
- communicte test status of bindists clearly through e.g. GHCup
- resolve GHC issues related to test bindists:
* https://gitlab.haskell.org/ghc/ghc/-/issues/22726
* https://gitlab.haskell.org/ghc/ghc/-/issues/22723
* https://gitlab.haskell.org/ghc/ghc/-/issues/22727
* make fixing bindists easier
- implement revisions in GHCup: https://github.com/haskell/ghcup-hs/issues/361
- make it easy to update an older GHC branch and re-run the release pipeline
- Make building upstream release binaries easier
* https://github.com/haskell/cabal/issues/9461
* https://github.com/haskell/haskell-language-server/issues/3878

One main idea is that bindists should be primarily tested **on the users system**, because that is where they're going to run.
It is great to know that e.g. the test suite passes on GHC CI, but that may have little value in different environments.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Additionally, issues with tests can flow back to upstream developers and we may develop workflows and processes to streamline
this type of feedback. Early release candidates can assist with this workflow.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯


Another perception shift necessary is that upstream projects should consider that their build system
are end-user interfaces, making it easier for both distributors and end-user to build release binaries correctly themselves.

### Nightlies

We also want to make nightlies available for GHC and cabal and HLS. This will require
coming up with a permanent storage solution and very robust nightly pipelines.

## Timeline

* 6 months: proof of concept of a central GitHub CI building most of the toolchain
* 12 months: building GHC via github actions

## Funding

### Who and what

The GHCup project requests funding for private GitHub CI runners to power the midstream bindist release pipelines.
It will receive and manage the funding in strict collaboration with the Haskell Foundation.

Volunteers who want to collaborate in midstream bindists are welcome to look at the project structure and collaboration guidelines:

* https://www.haskell.org/ghcup/about/#team
* https://www.haskell.org/ghcup/dev/#contribution-process-and-expectations

### Budget

The following is an example/estimate of a budget (prices are in USD). The HF and the proposer will negotiate the exact terms in private.

- Linux/FreeBSD x86_64 runner on Hetzner (AX52)
* monthly: $62.24
* yearly: $746.88
- Linux aarch64 runner on Hetzner (RX170)
* monthly: $181.73
* yearly: $2,180.76
- Darwin runner on Hetzner (Mac Mini M1)
* monthly: $56.59
* yearly: $679.08

To host one private runner per all these platform, the yearly cost would be: **$3,606.72**

There will likely also be an initial setup cost (as is usual for Hetzner).

We may request more runners depending on the demand, so it may very well be 3 runners per platform, resulting in yearly cost of: $10,820.16

**As such, the budget estimated is between 4 to 10k USD per year.**

## Stakeholders

* GHCup developers, who receive funding
* GHC developers
* cabal developers
* stack developers
* HLS developers
* VSCode Haskell developers
* Haskell toolchain end users

## Success

* reliable, continuously maintained bindists, readily available