Provide Nix expressions

[domenkozar]

They currently live in https://github.com/hercules-ci/ghcide-nix, however, this has a drawback that they have to be manually updated and thus can't use just any commit from ghcide and get a binary.

Once Nix files are in place here, I recommend setting up CI to provide binaries for each commit:

• github actions + cachix.org for serving the binaries
• https://hercules-ci.com/ agent running on someone's infrastructure (happy to maintain it myself)

[garyverhaegen-da]

Please excuse my ignorance here, but:

• What are these nix expressions for? Is that for people who want to use ghcide on a nixOS machine? For any Haskell project set up to build through nix? Is providing those nix expressions the only way to use ghcide from a Haskell project that uses nix as its build tool?
• What, if any, are the benefits to the rest of the ecosystem? (I have no sense of how prevalent nix is as a build tool for Haskell projects.)
• What, if any, are the costs to the rest of the ecosystem?
• Why do we need to change CI? Is it not possible to create nix-compatible binaries from a "normal" machine? We seem to be able to do it on the daml repo.

[cocreature]

Let me try to answer some of @garyverhaegen-da’s questions and hopefully @domenkozar can weigh in on the others:

• The nix expressions are for people that use Nix for Haskell development both on nixOS and on other operating systems. You might be able to get away with building ghcide without nix while using nix for Haskell development in some cases but that’s just asking for issues to pop up since ghcide needs to be built against the same GHC version as your project (not necessarily the same binary but there are more constraints than just the version).
• Nix is very prevalent as a build tool for Haskell tools. So people will want and create these Nix expressions either way (as shown by the fact that @domenkozar has already created them). The obvious advantage of having these expressions live in this repository is that they don’t get out of sync.
• To ensure that they don’t get out of sync, we have to test them in CI. We could do that by just calling nix-build on Azure pipelines similar to how we handle this on the DAML repo.
• We probably want to cache Nix stuff in some way for two reasons: 1. It makes our CI jobs faster. 2. It’s nice for external contributors to not have to rebuild the world.
  • Achieving 1 is easy: We can just use Azure’s cache to cache the nix store.
  • Achieving 2 is a bit more tricky: There are two options here that I see: 1. Host our own binary cache similar to the DAML repo (probably a bad idea since that might get us back to not being able to run automatic CI jobs for external contributors) 2. Use cachix which is a fairly popular host for Nix caches that is easy to use (I’ve used it myself before) (disclaimer: It’s created and maintained by @domenkozar). 2 will probably still require some auditing since we have to have the credentials for our pushing to Cachix in our CI jobs which is an additional attack vector.
• Hercules is a CI service created by @domenkozar that supposedly integrates nicely with Cachix and Nix in general. If we don’t host the infrastructure ourselves, I believe that we probably don’t need new credentials but just a Github webhook so I’m not sure it introduces any additional attack vector. That said (please correct me if I’m wrong here @domenkozar) Hercules is fairly new, much less popular than cachix and I’ve never used it myself which makes me a bit sceptical (but I’m happy to be convinced). It will also never (or at least not in the foreseeable future) be our primary CI service so I don’t think we will profit much from whatever fancy features it has and it will mostly be equivalent to calling nix-build on Azure and pushing to cachix (again please correct me here @domenkozar)

I haven’t given this too much thought so far but atm I’m leaning towards doing nix-build + using Azure’s nix cache at least for now and see if we can use Cachix or host a binary cache ourselves after that.

[domenkozar]

What are these nix expressions for?

They make up for deterministic installation of ghcide on linux and macos. No matter which environment or distribution someone is using, it's always going to work.

Binary caches (cachix) provide binaries to avoid compilation from source, making the whole installation a matter of downloading ~100MB binaries and unpacking them.

This is convenient if you want ghcide-ghc865, ghcide-ghc864 ... ghcie-ghc841

Nix is just a build tool here, Haskell dependencies are mapped 1-to-1 from stack.yaml plan.

What, if any, are the benefits to the rest of the ecosystem? (I have no sense of how prevalent nix is as a build tool for Haskell projects.)

It's quite prevalent, although I have no numbers to back that off. At most Haskell hackathon there's a group of people hacking on Nix tooling.

The benefit is that it simplifies the installation of ghcide, which is one part of why is editor integration hard in Haskell compared to the rest of the languages.

What, if any, are the costs to the rest of the ecosystem?

I see no other costs rather than maintenance.

I believe that we probably don’t need new credentials but just a Github webhook so I’m not sure it introduces any additional attack vector.

It's using GitHub apps, so you grant access via GitHub to a single repository and tie that to a token which agent uses to run jobs.

Hercules is fairly new, much less popular than cachix and I’ve never used it myself which makes me a bit sceptical (but I’m happy to be convinced)

Both are written in Haskell and other boring technologies, so the confidence is pretty high. So far there were a few downtimes, mostly handled within an hour or two. Cachix is older though, more than 1000 people rely on it. Hercules CI has over 100 users, growing each day.

---

I'd like to also add that we're going for the commercial success of Nix, so I am biased here.

I don't think you need to do any of the infrastructure if you don't want to. I'd be OK just maintaining Nix expressions myself in ghcide and giving access to our CI to build it.

I used to maintain https://github.com/Infinisil/all-hies for HIE itself, and people really liked the simple installation method.

No hard feelings if you decide to go a different route, it would be just less maintenance for me having all of this upstream.

[garyverhaegen-da]

To clarify, any change of infrastructure, including a move to an external CI or the addition of a webhook, needs to go through an audit by our security team. So I'm very keen on getting as much value as we can without doing that.

From the above, it seems to me that we can achieve almost all we want by simple changes to our existing build process and not introducing any new credentials: having the nix expressions in this repo, building them as part of CI, and keeping a cache of /nix/store at the Azure level all require no sensitive change at all. If I understand correctly that would result in everything working for everyone:

• Our CI would guarantee the nix expressions don't get out of date, and
• Nix users could therefore rely on those expressions.

The only downside left would be that nix users would need to build those expressions instead of being able to download the binaries. Am I understanding correctly so far?

If the only missing piece after that is the discovery of new commits by Hercules so it can pre-build packages, could it just poll this repo, say once a day? That way we would not need to know about it: no credential, no webhook, no external service we need to know about, and therefore no need to go through a security audit.

[cocreature]

Let me summarize the current state: We’ll happily accept the nix expressions in this repo with an Azure-based CI setup but relying on an external service is not something we want to do right now.

[domenkozar]

keeping a cache of /nix/store at the Azure level

That's the very first thing one would do, but it brings a bunch of problems:

• /nix/store/ is ever growing if you don't garbage collect: you can garbage collect before building, but that wastes time and bandwidth
• copying whole closure for each build is quite time consuming - CI build times will be slow

Either way it seems like you've made the decision so I won't dwell on this further, let's go step by step.

I'll prepare expressions to be upstreamed here.

[cocreature]

The way azure caches work, this shouldn’t ever grow. Downloading the closure should be relatively fast (we have a significantly larger closure on the daml repo and it takes ~2 minutes to download). Uploading is slow but that only happens if the cache key (which will include things like the nix files and the cabal files) which isn’t all that often usually.

[garyverhaegen-da]

Regarding the nix cache, we do something similar for the daml repo, it may be worth taking a look. Essentially, the cache is recreated from scratch every time the files that define it (in this case it would be the set of nix expressions) change, which hopefully doesn't happen too often. I guess this counts as GC, in a very naive way.

[drewboardman]

Is this still being considered to be merged into the ghcide repo?

[cocreature]

Right now, no. I think it’s fine for this to be in a separate repo.