Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Haskell package set to Stackage Nightly (plus other fixes) #103740

Merged
merged 43 commits into from Nov 20, 2020
Merged

Conversation

@peti
Copy link
Member

@peti peti commented Nov 13, 2020

This PR is test-built by Hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. I'll fix up the remaining errors and merge it on Friday, 2020-11-20 20:00 +01:00. You can watch this live on Twitch at https://www.twitch.tv/peti343. In addition to the chat features offered by Twitch, there is also a voice conference at https://discord.gg/YTEa3XR that viewers can use to chat with me and with each other.

@peti peti requested a review from cdepillabout as a code owner Nov 13, 2020
@peti peti self-assigned this Nov 13, 2020
@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Nov 14, 2020

@peti Why did you decide to update the package set to Nightly (instead of an LTS)?

Also, when LTS-17 comes out, are you planning on switching back to track it?

@peti
Copy link
Member Author

@peti peti commented Nov 14, 2020

The discussion took place at http://www.twitch.tv/peti343/v/801715350?sr=a&t=3625s.

We felt that Nightly is better for us in general. Release branches could follow some LTS set, of course, but we actually don't update the package set in release branches, so I guess it doesn't make any practical difference.

@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Nov 14, 2020

@peti Thanks for the link to the conversation.

I think I agree with most of the points brought up. It sounds like Nightly might actually be a better fit for nixpkgs, since we also provide many packages from Hackage that are not in Stackage. Packages from Hackage are sometimes hard to get working with the slightly older stuff from Stackage.

One concern I have is how often Nightly drops "important" packages. I remember from about a year or so ago I was quite surprised when Nightly dropped Servant, which has quite a few dependencies in Stackage. However, I don't actually have any idea how frequent or widespread this kind of breakage is. Servant is somewhat of a fringe package (even though it has a lot of dependencies), but I wonder what the story is for the more "core" packages, like lens, aeson, conduit, etc.

On Twitch @maralorn suggested that we should do another call for help with making sure our various Haskell packages compile on Nightly / ghc-8.10.2. I think this sounds like a good idea.

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 16, 2020

Okay, I got this down to zero eval errors.

We are now at 5933 building packages, down from 6044 before the switch to stackage nightly. This is not so bad.

Sadly right now some of the most important packages are still broken. e.g. cabal-install, stack, hls, git-annex, neuron …

So, still a lot to do.

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 17, 2020

I will just very informally document here, what packages are still on my list to be fixed:

  • cabal-install
  • hls
  • stack
  • neuron
  • stan
  • git-annex
  • password
  • snap

Of course, I don‘t know if I can fix them all.

@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Nov 18, 2020

@srid

neuron isn't building because the following packages are broken on this branch:

  • reflex
  • aeson-gadt-th
  • rib-core
  • dependent-sum-aeson-orphans
@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Nov 18, 2020

I've fixed Spago in #104164.

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 18, 2020

@srid

neuron isn't building because the following packages are broken on this branch:

* `reflex`

* `aeson-gadt-th`

* `rib-core`

* `dependent-sum-aeson-orphans`

I have fixed all of them locally. But most of them with simple dontCheck and doJailbreak, so I want to clean that up and make a pretty PR with patches or at least issues out of it.

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 18, 2020

I have fixed all neuron deps other than neuron itself. It fails with this error message: srid/neuron#475

@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Nov 19, 2020

@maralorn The fix in 80d6b9a for pcg-random was all that was needed to get password building. I'll check it off on your list above.

@expipiplus1
Copy link
Contributor

@expipiplus1 expipiplus1 commented Nov 19, 2020

streamly-0.7.2 doesn't compile with primitive-0.7.1 (even after jailbreak) (hence pkgs.nix-linter is broken)

PR here #104232

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 19, 2020

As soon as we merge master into haskell-updates (or the other way around). hls will break. :-(

#104275

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 19, 2020

That has been fixed. \o/

I think it's a bit late now to merge master in, if we did that now, we won‘t have reasonable hydra infos tomorrow.

I wish we had a way for hydra to tell us which jobs fail now which built successfully last week. So we could have a look at which of them should also be fixed …

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 20, 2020

@peti You obviously didn‘t read what I said or decided to disagree. 😆

@peti
Copy link
Member Author

@peti peti commented Nov 20, 2020

I don't see an alternative. We have to merge master in, no? What else could we do?

@peti peti force-pushed the haskell-updates branch from 18fa823 to 3278408 Nov 20, 2020
peti added 3 commits Nov 13, 2020
This means we'll switch our default compiler from ghc-8.8.4 to 8.10.2.
This update was generated by hackage2nix v2.15.5-28-gd194902 from Hackage revision
commercialhaskell/all-cabal-hashes@cb01aa6.
expipiplus1 and others added 14 commits Nov 19, 2020
The build was broken because our `primitive` version was too high.

Using the patch from composewell/streamly#565
This bumps aeson-gadt-th, which will be pulled in by the hackage update
this night anyways.
This update was generated by hackage2nix v2.15.5-27-g11a103b from Hackage revision
commercialhaskell/all-cabal-hashes@fa72061.
This update was generated by hackage2nix v2.15.5-28-gd194902 from Hackage revision
commercialhaskell/all-cabal-hashes@8c0e05f.
@Kiwi, please check the state of glirc. matterhorn, mattermost-api, and mattermost-api-qc.

@roberth, please check hercules-ci-agent, hercules-ci-api-core, and hercules-ci-api-agent.
This update was generated by hackage2nix v2.15.5-28-gd194902 from Hackage revision
commercialhaskell/all-cabal-hashes@8c0e05f.
@peti peti force-pushed the haskell-updates branch from 0cf6d0c to c55cdb4 Nov 20, 2020
The cachix build is broken with ghc-8.10.x and recent versions of
its dependencies, particularly servant, swagger2, etc. We cannot
easily fix that with overrides -- it would be best if upstream
released a new version that builds successfully in the Stackage
Nightly environment. A simple way to ensure that would be to add
cachix to Stackage.

Ping @domenkozar.
@domenkozar
Copy link
Member

@domenkozar domenkozar commented Nov 20, 2020

I've only gotten the notification that decision was made to switch to nightly. Can I get like a week before this is merged? Otherwise cachix will be broken and I literally had no time to prepare without knowing about the decision.

@turboMaCk
Copy link
Member

@turboMaCk turboMaCk commented Nov 20, 2020

how about this quickfix:

  # in all-packages.nix
  cachix = haskell.lib.justStaticExecutables haskell.packages.ghc883.cachix;

this builds for me and might be good enough perhaps?

@peti
Copy link
Member Author

@peti peti commented Nov 20, 2020

@domenkozar, there are plenty of ways to fix cachix quickly. One has been pointed out by @turboMaCk. Another option might be to use overrideScope to re-create the environment (set of versions of its dependencies) in which it builds successfully. I don't know. It doesn't feel like a big deal if cachix is broken for a few hours or even days on the unstable development branch. I'd rather not abort all our efforts here to avoid a problem that doesn't feel particularly major to be honest.

@peti peti merged commit 4f2880b into master Nov 20, 2020
13 of 16 checks passed
13 of 16 checks passed
tests tests
Details
action action
Details
Wait for ofborg This failed status will be cleared when ofborg finishes eval.
Details
grahamcofborg-eval Calculating Changed Outputs
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e681190"; rev="e6811900dddcf6e5c16556e8a41a28abadf25dc2"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@domenkozar
Copy link
Member

@domenkozar domenkozar commented Nov 20, 2020

@peti you're a true gentlement :D

@maralorn
Copy link
Member

@maralorn maralorn commented Nov 20, 2020

For what it's worth. It is really annoying that we don‘t have notifications for failed hydra builds. We should have an automated way to notify devs about pending broken merges as early as possible.

Our current workflow is lacking in that regard and we will need to improve on that.

@peti
Copy link
Member Author

@peti peti commented Nov 20, 2020

We'll merge #104424 next week.

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Nov 20, 2020

@maralorn we have release blockers, but it doesn't help if someone removes your package from it 20min before merging the PR...

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Nov 20, 2020

And disabling servant means disabling A LOT of packages.

@expipiplus1
Copy link
Contributor

@expipiplus1 expipiplus1 commented Nov 21, 2020

For what it's worth. It is really annoying that we don‘t have notifications for failed hydra builds. We should have an automated way to notify devs about pending broken merges as early as possible.

Our current workflow is lacking in that regard and we will need to improve on that.

Can you open an issue for this? (It's also been discussed elsewhere, for instance the static pkgs maintainer issue)

@nixos-discourse
Copy link

@nixos-discourse nixos-discourse commented Nov 21, 2020

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-ghc-default-version-8-10-2-package-versions-now-from-stackage-nightly/10117/4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants