New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixpkgs is increasing with each Haskell LTS release for 400+ Kb #14897

Closed
danbst opened this Issue Apr 22, 2016 · 17 comments

Comments

@danbst
Contributor

danbst commented Apr 22, 2016

nixpkgs is increasing with each Haskell LTS release for 400+Kb and now takes 50Mb out of 128 Mb of checkout of nixpkgs. Are people aware of this? What are the policies on repo size?

Issue should be tagged as closure-size, I think (nixpkgs checkout resides in my /nix/store)

@abbradar

This comment has been minimized.

Show comment
Hide comment
@abbradar

abbradar Apr 22, 2016

Member

I remember there was a discussion about having only revision and checksum of a Hackage snapshot inside Nixpkgs and building hackage-packages.nix on Hydra instead (and using Nix's import-from-derivation). Was there some sort of conclusion reached?

Member

abbradar commented Apr 22, 2016

I remember there was a discussion about having only revision and checksum of a Hackage snapshot inside Nixpkgs and building hackage-packages.nix on Hydra instead (and using Nix's import-from-derivation). Was there some sort of conclusion reached?

@danbst

This comment has been minimized.

Show comment
Hide comment
@danbst

danbst Apr 22, 2016

Contributor

My concern is not about hackage, but files like https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-lts-1.5.nix
That's around of 9k lines "library" = dontDistribute super."library"

  1. Maybe dontDistribute should be default and LTS configuration should include only those to distribute?
  2. LTS releases should be based on on some basis LTS config using Nix merge capabilities
Contributor

danbst commented Apr 22, 2016

My concern is not about hackage, but files like https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-lts-1.5.nix
That's around of 9k lines "library" = dontDistribute super."library"

  1. Maybe dontDistribute should be default and LTS configuration should include only those to distribute?
  2. LTS releases should be based on on some basis LTS config using Nix merge capabilities
@copumpkin

This comment has been minimized.

Show comment
Hide comment
@copumpkin

copumpkin Apr 22, 2016

Member

@abbradar I think we need one more nix feature for that to work well (NixOS/nix#52), but that feature needs work and someone to work on it.

Member

copumpkin commented Apr 22, 2016

@abbradar I think we need one more nix feature for that to work well (NixOS/nix#52), but that feature needs work and someone to work on it.

@ericsagnes

This comment has been minimized.

Show comment
Hide comment
@ericsagnes

ericsagnes Apr 23, 2016

Contributor

Related ML thread: Publish All of Hackage

Contributor

ericsagnes commented Apr 23, 2016

Related ML thread: Publish All of Hackage

@vcunat

This comment has been minimized.

Show comment
Hide comment
@vcunat

vcunat Apr 23, 2016

Member

I thought importing e.g. from fetchurl works fine, only you have that consequence of building the imported stuff during evaluation already.

Member

vcunat commented Apr 23, 2016

I thought importing e.g. from fetchurl works fine, only you have that consequence of building the imported stuff during evaluation already.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar May 30, 2016

Member

This is becoming more and more annoying, nixpkgs is 100MB while haskell is 10MB and LTS packages are 40MB.

Member

domenkozar commented May 30, 2016

This is becoming more and more annoying, nixpkgs is 100MB while haskell is 10MB and LTS packages are 40MB.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar May 30, 2016

Member

Could builtins.fetchTarball be used here? Then we could have a separate job for haskell packages and people could still get LTS packages.

We would need to be very careful introducing new fetchTarball calls, but I think this could be an exception.

cc @peti

Member

domenkozar commented May 30, 2016

Could builtins.fetchTarball be used here? Then we could have a separate job for haskell packages and people could still get LTS packages.

We would need to be very careful introducing new fetchTarball calls, but I think this could be an exception.

cc @peti

@copumpkin

This comment has been minimized.

Show comment
Hide comment
@copumpkin

copumpkin May 30, 2016

Member

I don't think we need fetchTarball here. It doesn't seem terrible for a particular nixpkgs version to be tied to a particular haskellPackages version, so updating the ref and sha256 every so often (with e.g., import (fetchFromGitHub { owner = "peti"; repo = "haskellPackages"; ... })) doesn't seem too bad.

Ideally, I'd like there to be a hard rule that fetchTarball appear nowhere in nixpkgs.

Member

copumpkin commented May 30, 2016

I don't think we need fetchTarball here. It doesn't seem terrible for a particular nixpkgs version to be tied to a particular haskellPackages version, so updating the ref and sha256 every so often (with e.g., import (fetchFromGitHub { owner = "peti"; repo = "haskellPackages"; ... })) doesn't seem too bad.

Ideally, I'd like there to be a hard rule that fetchTarball appear nowhere in nixpkgs.

@vcunat

This comment has been minimized.

Show comment
Hide comment
@vcunat

vcunat May 30, 2016

Member

We would want to use immutable references here. Apart from that, AFAIK the main difference is whether you need to compile curl, unzip, etc. during evalutation (e.g. --dry-run).

Member

vcunat commented May 30, 2016

We would want to use immutable references here. Apart from that, AFAIK the main difference is whether you need to compile curl, unzip, etc. during evalutation (e.g. --dry-run).

@peti peti self-assigned this Jun 5, 2016

@peti peti added this to the 16.09 milestone Jun 5, 2016

@peti

This comment has been minimized.

Show comment
Hide comment
@peti

peti Jun 5, 2016

Member

I'll remove almost all Stackage-related package sets from Nixpkgs soon. Give me a few more days, then I'll have a posting for ready nix-dev that explains my plans.

Member

peti commented Jun 5, 2016

I'll remove almost all Stackage-related package sets from Nixpkgs soon. Give me a few more days, then I'll have a posting for ready nix-dev that explains my plans.

@peti

This comment has been minimized.

Show comment
Hide comment
@peti

peti Jun 8, 2016

Member

http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

tl;dr: once the LTS 7.x package set comes out, I intend to make the following changes in "master":

  • All haskell.packages.lts.* package sets will disappear.
  • haskellPackages will loosely follow the most recent LTS release,
Member

peti commented Jun 8, 2016

http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

tl;dr: once the LTS 7.x package set comes out, I intend to make the following changes in "master":

  • All haskell.packages.lts.* package sets will disappear.
  • haskellPackages will loosely follow the most recent LTS release,

peti added a commit to peti/nixpkgs that referenced this issue Jul 19, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first stop towards dropping Stackage support. We keep LTS
6.x around because I don't want to downgrade our default compiler to GHC
7.x again, but once LTS 7.x comes out we'll switch our main package set
to that and drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant: NixOS#16130.

peti added a commit to peti/nixpkgs that referenced this issue Jul 20, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first stop towards dropping Stackage support. We keep LTS
6.x around because I don't want to downgrade our default compiler to GHC
7.x again, but once LTS 7.x comes out we'll switch our main package set
to that and drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant: NixOS#16130.

peti added a commit to peti/nixpkgs that referenced this issue Jul 20, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first stop towards dropping Stackage support. We keep LTS
6.x around because I don't want to downgrade our default compiler to GHC
7.x again, but once LTS 7.x comes out we'll switch our main package set
to that and drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant: NixOS#16130.

peti added a commit to peti/nixpkgs that referenced this issue Jul 20, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first step towards dropping Stackage support. We keep LTS 6.x
around because I don't want to downgrade our default compiler to GHC 7.x,
but once LTS 7.x comes out we'll switch our main package set to that and
drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant:

 - NixOS#16130
 - commercialhaskell/stack#2259

@peti peti closed this in 641fc0e Jul 21, 2016

peterhoeg added a commit to peterhoeg/nixpkgs that referenced this issue Jul 22, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first step towards dropping Stackage support. We keep LTS 6.x
around because I don't want to downgrade our default compiler to GHC 7.x,
but once LTS 7.x comes out we'll switch our main package set to that and
drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant:

 - NixOS#16130
 - commercialhaskell/stack#2259

vrthra added a commit to vrthra/nixpkgs that referenced this issue Jul 24, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first step towards dropping Stackage support. We keep LTS 6.x
around because I don't want to downgrade our default compiler to GHC 7.x,
but once LTS 7.x comes out we'll switch our main package set to that and
drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant:

 - NixOS#16130
 - commercialhaskell/stack#2259

kampfschlaefer added a commit to kampfschlaefer/nixpkgs that referenced this issue Jul 26, 2016

haskell: remove all but the latest LTS package sets (version 6.7)
This is the first step towards dropping Stackage support. We keep LTS 6.x
around because I don't want to downgrade our default compiler to GHC 7.x,
but once LTS 7.x comes out we'll switch our main package set to that and
drop Nightly.

More details are at:

  http://permalink.gmane.org/gmane.linux.distributions.nixos/20505

Closes NixOS#14897.

Also relevant:

 - NixOS#16130
 - commercialhaskell/stack#2259
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Sep 17, 2016

Member

@peti can I ask you to write a changelog entry for this PR?

Member

domenkozar commented Sep 17, 2016

@peti can I ask you to write a changelog entry for this PR?

@peti

This comment has been minimized.

Show comment
Hide comment
@peti

peti Sep 17, 2016

Member

Yeah, you are right. The Haskell situation needs some documentation. I'll see what I can do. I'll be traveling next week, so I might not get to it until the week after that.

Member

peti commented Sep 17, 2016

Yeah, you are right. The Haskell situation needs some documentation. I'll see what I can do. I'll be traveling next week, so I might not get to it until the week after that.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Sep 30, 2016

Member

@peti could we document these changes today before the release? :)

Member

domenkozar commented Sep 30, 2016

@peti could we document these changes today before the release? :)

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Sep 30, 2016

Member

@peti I've been talking to various people in Haskell community and this brings major pain to them. Could we keep more than just latest LTS package set?

This would give them a timespan of for example one year to migrate. Currently when they update nixpkgs, they get a new TLS set and they're forced to also upgrade at the same time.

I think keeping latest or a few latest sets would yield very little change to nixpkgs size.

Member

domenkozar commented Sep 30, 2016

@peti I've been talking to various people in Haskell community and this brings major pain to them. Could we keep more than just latest LTS package set?

This would give them a timespan of for example one year to migrate. Currently when they update nixpkgs, they get a new TLS set and they're forced to also upgrade at the same time.

I think keeping latest or a few latest sets would yield very little change to nixpkgs size.

@domenkozar domenkozar reopened this Sep 30, 2016

@peti

This comment has been minimized.

Show comment
Hide comment
@peti

peti Sep 30, 2016

Member

@domenkozar, I'll add an entry to the release notes in a few minutes.

About offering LTS package sets again: I don't want to do it. My life as a maintainer for Haskell packages has become much easier since we're following the latest LTS release only. I already threw all the obsolete code out of my generator tools and everything is so much simpler and nicer now! I realize it's inconvenient for some, but I just don't want to bother worrying about ancient LTS package sets anymore, especially not in a situation where we don't have the bandwidth to do test builds for them. If someone else wants to maintain that stuff, I am all for it, but I don't want to any more.

Member

peti commented Sep 30, 2016

@domenkozar, I'll add an entry to the release notes in a few minutes.

About offering LTS package sets again: I don't want to do it. My life as a maintainer for Haskell packages has become much easier since we're following the latest LTS release only. I already threw all the obsolete code out of my generator tools and everything is so much simpler and nicer now! I realize it's inconvenient for some, but I just don't want to bother worrying about ancient LTS package sets anymore, especially not in a situation where we don't have the bandwidth to do test builds for them. If someone else wants to maintain that stuff, I am all for it, but I don't want to any more.

@peti peti closed this in 6e785be Sep 30, 2016

peterhoeg added a commit to peterhoeg/nixpkgs that referenced this issue Oct 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment