Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unbreak ghcide and bump haskell-lsp version #75449

Closed
wants to merge 4 commits into from

Conversation

@turion
Copy link
Contributor

@turion turion commented Dec 10, 2019

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

- haskell-lsp ==0.15.0.0
- haskell-lsp-types ==0.15.0.0
- haskell-lsp ==0.17.0.0
- haskell-lsp-types ==0.17.0.0
Comment on lines 868 to 869

This comment has been minimized.

@cdepillabout

cdepillabout Dec 11, 2019
Member

In general, the Haskell stuff in nixpkgs tracks the latest LTS release. We don't override these package versions with our own updated versions (since this could break a lot of stuff).

If you look at the top of this list here, you can see that these packages are locked to versions from the latest LTS. (peti updates these weekly, you can watch live if you're interested: https://www.twitch.tv/peti343)

It appears that the latest version of ghcide requires haskell-lsp ==0.17.*, haskell-lsp-types ==0.17.*, which is later than the version from LTS-14.

However, hackage2nix generates the latest versions of packages in the LTS as well, so haskellPackages.haskell-lsp_0_18_0_0 and haskellPackages.haskell-lsp-types_0_18_0_0 should be available.

I suggest trying to override ghcide to use these two package versions and then jailbreaking the cabal file.

Here's an example of hnix overriding package versions as well as using doJailbreak:

hnix =
generateOptparseApplicativeCompletion "hnix" (
dontCheck (doJailbreak (super.hnix.override { these = self.these_0_7_6; }))
);

Could you update this PR to remove the above lines from configuration-hackage2nix.yaml, and instead adding an entry for ghcide to configuration-common.nix?

This comment has been minimized.

@cdepillabout

cdepillabout Dec 11, 2019
Member

I suggest trying to override ghcide to use these two package versions and then jailbreaking the cabal file.

Oh, and let me know if this doesn't work. There is a way to force hackage2nix to generate arbitrary package versions (so we could make it generate haskellPackages.haskell-lsp_0_17_0_0 for ghcide).

This comment has been minimized.

@turion

turion Dec 11, 2019
Author Contributor

Thanks for your explanations! Do I have to run hackage2nix locally to test my changes?

This comment has been minimized.

@cdepillabout

cdepillabout Dec 11, 2019
Member

@turion No, peti runs hackage2nix locally, so you don't have to do it in this PR.

This comment has been minimized.

@turion

turion Dec 11, 2019
Author Contributor

Unfortunately ghcide does not build with haskell-lsp-0.18. We probably need to make a haskellPackages.haskell-lsp_0_17_0_0 , but I don't know how. In parallel, I'll start and (send a PR to) upgrade ghcide to build with a newer haskell-lsp.

This comment has been minimized.

@turion

turion Dec 12, 2019
Author Contributor

I see. I'm working towards 1. then first :)

This comment has been minimized.

@turion

turion Dec 12, 2019
Author Contributor

ghcide with haskell-lsp-0.18 is released. I don't really understand how this change gets here. Do I just wait and rebase?

This comment has been minimized.

@cdepillabout

cdepillabout Dec 12, 2019
Member

@turion

I don't really understand how this change gets here.

I think there is an automatic script that updates the haskell-updates branch with recent package versions from Hackage. (Or maybe it is semi-automatic and peti has to run it by hand)

You can see the current state of haskell-updates here (including a bunch of commits with the message hackage-packages.nix: automatic Haskell package set update):

#75429

At some point (maybe today or tomorrow?) one of these Haskell package set updates should contain the new version of ghcide. At that point you should be able to rebase this PR on haskell-updates and use the new version of haskell-lsp.

This comment has been minimized.

@turion

turion Dec 13, 2019
Author Contributor

Ok. Thanks for all your explanations!

Is this whole workflow documented somewhere? Just in case peti goes on a 2 month vacation and someone needs to pick up the reign ;)

This comment has been minimized.

@cdepillabout

cdepillabout Dec 13, 2019
Member

It is somewhat documented in the hackage2nix Readme:

https://github.com/NixOS/cabal2nix/tree/master/hackage2nix

There is also a youtube video showing the steps:

https://www.youtube.com/watch?v=qX0mgtSm360

Also, peti streams live these updates, so if you watch him doing it once or twice, you should get a good idea of how to do it:

https://discourse.nixos.org/t/please-join-the-new-weekly-haskell-updates-video-livestream/4420

@cdepillabout cdepillabout mentioned this pull request Dec 11, 2019
5 of 10 tasks complete
@peti peti force-pushed the NixOS:haskell-updates branch 2 times, most recently from 14b0fb3 to 9c01e12 Dec 13, 2019
Copy link
Member

@peti peti left a comment

Can you please re-base this PR on top of a current version of haskell-updates to resolve the merge conflicts?

cdepillabout and others added 2 commits Dec 16, 2019
This update was generated by hackage2nix v2.15.0-9-g650d530 from Hackage revision
commercialhaskell/all-cabal-hashes@dc5b0e7.
@peti peti force-pushed the NixOS:haskell-updates branch from 3814616 to 8bee1b9 Dec 18, 2019
@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

Can you please re-base this PR on top of a current version of haskell-updates to resolve the merge conflicts?

@peti Yes, since the conflicts are only internal to haskell-updates. But it seems that you added a commit that obsoletes my PR. Also, it seems that it doesn't work anymore because haskell-lsp has advanced to 0.19.0.0 in the meantime.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

Another dependency, hie-bios, is broken on haskell-updates. Something with the test suite. I'll see whether I can fix it easily.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

Another dependency, hie-bios, is broken on haskell-updates. Something with the test suite. I'll see whether I can fix it easily.

Seems nontrivial to fix. I'll add dontCheck for the time being. This should be removed once there is an acceptable answer to mpickering/hie-bios#25 (comment)

@turion turion force-pushed the turion:haskell-updates_ghcide branch from 7461625 to 6417091 Dec 18, 2019
@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

Fixing hie-bios is separately in #75869.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

I'm a bit out of ideas. The upstream version for haskell-lsp is 0.19 by now, and ghcide is once again behind. It does not compile with the newer haskell-lsp, that's the latest thing I tried now. Do I have to pin the version of haskell-lsp at 0.18 somewhere? Is this a manual process that gets stuck every time upstream versions get out of sync?

@mpickering
Copy link
Contributor

@mpickering mpickering commented Dec 18, 2019

@turion You are discovering why people use haskell.nix.

@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Dec 18, 2019

@turion What you can do is send a PR adding a line to this list:

extra-packages:
- aeson < 0.8 # newer versions don't work with GHC 7.6.x or earlier
- aeson-pretty < 0.8 # required by elm compiler
- apply-refact < 0.4 # newer versions don't work with GHC 8.0.x
- aws ^>= 0.18 # pre-lts-11.x versions neeed by git-annex 6.20180227
- binary > 0.7 && < 0.8 # keep a 7.x major release around for older compilers
- binary > 0.8 && < 0.9 # keep a 8.x major release around for older compilers
- blank-canvas < 0.6.3 # more recent versions depend on base-compat-batteries == 0.10.* but we're on base-compat-0.9.*
- Cabal == 2.2.* # required for jailbreak-cabal etc.
- Cabal == 2.4.* # required for cabal-install etc.
- colour < 2.3.4 # newer versions don't support GHC 7.10.x
- conduit >=1.1 && <1.3 # pre-lts-11.x versions neeed by git-annex 6.20180227
- conduit-extra >=1.1 && <1.3 # pre-lts-11.x versions neeed by git-annex 6.20180227
- containers < 0.5 # required to build alex with GHC 6.12.3
- control-monad-free < 0.6 # newer versions don't compile with anything but GHC 7.8.x
- dbus <1 # for xmonad-0.26
- deepseq == 1.3.0.1 # required to build Cabal with GHC 6.12.3
- generic-deriving == 1.10.5.* # new versions don't compile with GHC 7.10.x
- gloss < 1.9.3 # new versions don't compile with GHC 7.8.x
- haddock == 2.22.* # required on GHC 8.0.x
- haddock-api == 2.22.* # required on GHC 7.8.x
- happy <1.19.6 # newer versions break Agda
- happy == 1.19.9 # for purescript
- haskell-gi-overloading == 0.0 # gi-* packages use this dependency to disable overloading support
- haskell-src-exts == 1.19.* # required by hindent and structured-haskell-mode
- hinotify == 0.3.9 # for xmonad-0.26: https://github.com/kolmodin/hinotify/issues/29
- hoogle == 5.0.14 # required by hie-hoogle
- html-conduit ^>= 1.2 # pre-lts-11.x versions neeed by git-annex 6.20180227
- http-conduit ^>= 2.2 # pre-lts-11.x versions neeed by git-annex 6.20180227
- inline-c < 0.6 # required on GHC 8.0.x
- inline-c-cpp < 0.2 # required on GHC 8.0.x
- lens-labels == 0.1.* # required for proto-lens-descriptors
- mainland-pretty == 0.6.2.* # required for tensorflow-opgen-0.1.0.0
- mtl < 2.2 # newer versions require transformers > 0.4.x, which we cannot provide in GHC 7.8.x
- mtl-prelude < 2 # required for to build postgrest on mtl 2.1.x platforms
- network == 2.6.3.1 # newer versions don't compile with GHC 7.4.x and below
- network == 3.0.* # required by network-bsd, HTTP, and many others (2019-04-30)
- parallel == 3.2.0.3 # newer versions don't work with GHC 6.12.3
- patience ^>= 0.1 # required by chell-0.4.x
- persistent >=2.5 && <2.8 # pre-lts-11.x versions neeed by git-annex 6.20180227
- persistent-sqlite < 2.7 # pre-lts-11.x versions neeed by git-annex 6.20180227
- primitive == 0.5.1.* # required to build alex with GHC 6.12.3
- proto-lens == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
- proto-lens-protobuf-types == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
- proto-lens-protoc == 0.2.* # required for tensorflow-proto-0.1.x on GHC 8.2.x
- QuickCheck < 2 # required by test-framework-quickcheck and its users
- resolv == 0.1.1.2 # required to build cabal-install-3.0.0.0 with pre ghc-8.8.x
- resourcet ==1.1.* # pre-lts-11.x versions neeed by git-annex 6.20180227
- seqid < 0.2 # newer versions depend on transformers 0.4.x which we cannot provide in GHC 7.8.x
- seqid-streams < 0.2 # newer versions depend on transformers 0.4.x which we cannot provide in GHC 7.8.x
- split < 0.2 # newer versions don't work with GHC 6.12.3
- tar < 0.4.2.0 # later versions don't work with GHC < 7.6.x
- these == 0.7.6 # required by hnix 0.6.1
- transformers == 0.4.3.* # the latest version isn't supported by mtl yet
- vector < 0.10.10 # newer versions don't work with GHC 6.12.3
- xml-conduit ^>= 1.7 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod ^>= 1.4 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod-core < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod-form < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod-persistent < 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod-static ^>= 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227
- yesod-test ^>= 1.5 # pre-lts-11.x versions neeed by git-annex 6.20180227

like

- haskell-lsp == 0.18.0.0 # for ghcide-0.4 (or whatever is the correct ghcide version)

Once this gets merge into haskell-updates and the update script run, it will create a haskell-lsp_0_18_0_0 derivation. This will exist as long as the haskell-lsp == 0.18.0.0 line exists.

You should be able to use this package in an override to ghcide. That will hopefully get ghcide building.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

@turion You are discovering why people use haskell.nix.

Hehe, yes ;) but I haven't given up on nixpkgs becoming good enough for standard usage yet.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

@cdepillabout I guess I could, but this is quite a manual and tiresome cycle. I'll just wait until upstream has released a new version. haskell/ghcide#277

This whole workflow feels like it's going to break every time e.g. haskell-lsp releases a newer version than ghcide can handle. Is that correct? How would this situation ever stabilize?

@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Dec 18, 2019

@turion Sorry this is somewhat of a manual/annoying process.

This whole workflow feels like it's going to break every time e.g. haskell-lsp releases a newer version than ghcide can handle. Is that correct?

For the most part, yes.

How would this situation ever stabilize?

There are some cases where the situation would stabilize:

  • On release branches of nixpkgs.

    release branches of nixpkgs don't get updates to the Haskell package versions. So if you get ghcide working on one of the release branches, it will probably stay working forever.

  • If ghcide and haskell-lsp are released to stackage (and end up in an LTS release).

    nixpkgs tracks the latest LTS release, so packages from the LTS releases are almost always working, because they are all "known compatible".

    If the upstream authors of ghcide and haskell-lsp don't want to get compatible versions into stackage, maybe you could?

  • If you add specific versions of haskell-lsp and ghcide to extra-packages in configuration-hackage2nix.nix as suggested above.

    This will generate versions of haskell-lsp and ghcide like haskell-lsp_0_18_0_0 and ghcide_0_4_0_0 (or whatever) that you can use in overrides and are known to work together.

    These should continue working at least until other dependencies get too new.

  • If ghcide and haskell-lsp stop frequently releasing new versions to Hackage.


Unfortunately, the Haskell stuff in nixpkgs doesn't work too well for new packages that aren't in Stackage and have frequent releases to Hackage.

As suggested by mpickering, creating an overlay with known package versions (or using a tool like haskell.nix to do this for you) might be the easiest solution.

Creating a ghcide.nix github repo that other people could also use might be helpful for the community as well.

@turion
Copy link
Contributor Author

@turion turion commented Dec 18, 2019

@cdepillabout Thank again you for your detailed explanations! It certainly helps me to understand the process and to more effectively contribute.

If ghcide and haskell-lsp are released to stackage (and end up in an LTS release).

Ah, good point! So LTS in Haskell on nixpkgs always refers to stackage LTS, I'm assuming now. There is a ticket tracking that: haskell/ghcide#144
This sounds like the best option for me then. I'll see whether I can help with that.

@peti
Copy link
Member

@peti peti commented Dec 20, 2019

I have attempted to fix the ghcide build in Nixpkgs, too, but it is quite a mess. Honestly, I think it's much easier to work with upstream to fix ghcide's build with recent versions of its dependencies than trying to build it in Nixpkgs with old ones.

@peti peti force-pushed the NixOS:haskell-updates branch from 6484351 to 1f79abf Dec 20, 2019
@cdepillabout
Copy link
Member

@cdepillabout cdepillabout commented Dec 21, 2019

I think @mdorman might have gotten ghcide working in #76103.

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Dec 21, 2019

This is why I maintain https://github.com/hercules-ci/ghcide-nix

@peti peti force-pushed the NixOS:haskell-updates branch 2 times, most recently from 1f7024e to 7d2cc64 Dec 26, 2019
@peti
Copy link
Member

@peti peti commented Dec 27, 2019

@turion, do you still intend to update this PR? Or can it be closed?

@peti peti mentioned this pull request Dec 27, 2019
3 of 3 tasks complete
peti added a commit that referenced this pull request Dec 27, 2019
We made an effort to support ghcide in Nixpkgs, but the complexity of the
problem is a bit too high, IMHO. We need to keep older versions of several
packages around in order to satisfy the build requirements, and some of those
older packages don't even build themselves (like hie-bios). We had ghcide
working at some point, but then it was broken again right away after a couple
of days. I fear that we'll run into that issue again and again with a setup of
that complexity.

Instead, I'd propose that we work with upstream to fix their build, i.e. let's
make sure that the proper ghcide build works with recent versions of its build
inputs.

Closes #75449.
Closes #76103.
peti added a commit that referenced this pull request Dec 27, 2019
We made an effort to support ghcide in Nixpkgs, but the complexity of the
problem is a bit too high, IMHO. We need to keep older versions of several
packages around in order to satisfy the build requirements, and some of those
older packages don't even build themselves (like hie-bios). We had ghcide
working at some point, but then it was broken again right away after a couple
of days. I fear that we'll run into that issue again and again with a setup of
that complexity.

Instead, I'd propose that we work with upstream to fix their build, i.e. let's
make sure that the proper ghcide build works with recent versions of its build
inputs.

Closes #75449.
Closes #76103.
@peti peti force-pushed the NixOS:haskell-updates branch from d3a751a to aa46cdd Dec 27, 2019
@peti peti force-pushed the NixOS:haskell-updates branch from aa46cdd to fdd64fb Dec 27, 2019
@peti peti closed this in 4d403a3 Dec 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.