Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update haskellPackages #122286

Merged
merged 31 commits into from May 11, 2021
Merged

Update haskellPackages #122286

merged 31 commits into from May 11, 2021

Conversation

maralorn
Copy link
Member

@maralorn maralorn commented May 9, 2021

This Merge

This is the regularly happening merge of the haskell-updates branch into master.

This branch is being continually build and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.

I will aim to merge this PR until 2021-05-14. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @cdepillabout will continue these merges from 2021-05-15 to 2021-05-28.

haskellPackages Workflow Summary

Our workflow is currently described at #121140 (comment).

The short version is this:

  • We regularly update the stackage and hackage pins on haskell-updates (normally at the beginning of a merge window).
  • The community fixes builds of Haskell packages on that branch.
  • We aim at at least one merge of haskell-updates into master every two weeks.
  • We only do the merge if the mergeable job is succeeding on hydra.
  • If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

This is the follow-up to #122117.


Scan for unbroken packages

This pull request is special in the sense that I haven‘t bumped stackage or hackage at the moment. I don‘t want anything confounding happening right now.

I have thrown away the broken list and now we will try to let hydra build all hackage packages once. When this evaluation is over I will mark all failing builds broken again.

After that we can go back to business as usual.

During this time merging fixes for broken packages is okay but we should try not to touch anything that is deep down in the dependency chain so that we don‘t case an unnecessary huge number of hydra builds.

A script used at test time in HTF assumes that you have GNU coreutils
installed in a manner where all executables are prefixed with a `g`.
Since we use GNU coreutils in the stdenv anyways, this is not necessary
and needs to be patched out.
haskell-ci pins a specific version of ShellCheck as ShellCheck adds new
checks with new minor versions which may break CI checks.
@maralorn
Copy link
Member Author

maralorn commented May 9, 2021

haskell-updates build report from hydra

evaluation 1669027 of nixpkgs commit 39c0dd9 as of 2021-05-11 00:19 UTC

Build summary

Platform Failed ❌ DependencyFailed ❗ TimedOut ⌛🚫 Canceled 🚫 HydraFailure 🚧 Success ✔️
aarch64-linux 📱 49 94 5 6519
x86_64-darwin 🍎 80 51 3 1 15 6477
x86_64-linux 🐧 3 6698

Maintained packages with build failure

Maintained packages with failed dependency

Unmaintained packages with build failure

128 job(s)

Unmaintained packages with failed dependency

140 job(s)

Report generated with maintainers/scripts/haskell/hydra-report.hs

@maralorn
Copy link
Member Author

maralorn commented May 9, 2021

Okay. So we see ~5200 that I will mark broken again.

I am confused about llvm. Apparently that package exists as a haskellPackage but is broken as such? Anyways the not working build will be disabled again and llvmPackages.llvm will be alright. So I think there is nothing to do there.

@sternenseemann
Copy link
Member

I am confused about llvm. Apparently that package exists as a haskellPackage but is broken as such? Anyways the not working build will be disabled again and llvmPackages.llvm will be alright. So I think there is nothing to do there.

Yes, we reexpose the LLVM version used by GHC. We should probably filter this out in the build report, seems like it has been broken in master for some time.

@maralorn
Copy link
Member Author

maralorn commented May 9, 2021

Okay. This is going great! We have gained about 700 more working packages. 🎉.

  • On x86-linux we are left with 20 to 30 build errors.
  • All those packages are not marked broken right now, because they do not fail itself but have a failing dependency.
  • The transitive broken script does not detect them because for the failing dependency there is no hydra job.

Until this point I am certain.

I looked into some of them and I have the feeling the biggest reason for this are packages that have license unknown. They seem to get a hydra-platform = none from hackage2nix.

This is not a huge problem either way. In the worst case I just mark them manually broken.
But I am not sure what the most principled way forward is. It strikes me as a bit weird anyways that we don‘t have jobs with unknown license but then we take no issue in building them for reverse dependencies?

Any opinions?

@sternenseemann
Copy link
Member

This is not a huge problem either way. In the worst case I just mark them manually broken.
But I am not sure what the most principled way forward is. It strikes me as a bit weird anyways that we don‘t have jobs with unknown license but then we take no issue in building them for reverse dependencies?

To be honest the correct way forward is to patch cabal2nix to emit lib.licenses.unfree instead of unknown. Unknown literally means that and as well that we can't know if we'll run into legal issues if we build such packages on hydra and redistribute them in binary form.

@cdepillabout
Copy link
Member

To be honest the correct way forward is to patch cabal2nix to emit lib.licenses.unfree instead of unknown

This seems like a completely reasonable thing to do.

I guess it would be great if we could report them upstream (or add new licenses to Nixpkgs), but that is quite a lot of annoying work.

@maralorn
Copy link
Member Author

maralorn commented May 9, 2021

This is not a huge problem either way. In the worst case I just mark them manually broken.
But I am not sure what the most principled way forward is. It strikes me as a bit weird anyways that we don‘t have jobs with unknown license but then we take no issue in building them for reverse dependencies?

To be honest the correct way forward is to patch cabal2nix to emit lib.licenses.unfree instead of unknown. Unknown literally means that and as well that we can't know if we'll run into legal issues if we build such packages on hydra and redistribute them in binary form.

I think it’s not that easy. I have looked at a few of those packages. (I count 496 with LICENSE unknown.) Unknown means that hackage displays the license as LicenseRef-OtherLicense. Which means "custom". All licenses I looked at were open source. Some even CC0.

Also consider this: https://hackage.haskell.org/upload

The code and other material you upload and distribute via this site must be under an open source license.

So IANAL but I assume we are pretty safe with assuming that all packages on Hackage are somehow Open Source licensed.

I am curious about other opinions but (until I here new arguments) I vote for enabling builds of those packages.

I am totally happy with maintaining a deny-list of packages where we know that we don‘t have the right to distribute them. But since we are processing packages in an automated fashion I can‘t imagine that we will get in a lot of trouble if we assume that it’s not the case.

@sternenseemann
Copy link
Member

Also consider this: https://hackage.haskell.org/upload

The code and other material you upload and distribute via this site must be under an open source license.

So IANAL but I assume we are pretty safe with assuming that all packages on Hackage are somehow Open Source licensed.

I'm no lawyer, so I am unsure whether this actually enough assurance for us. Hackage does not actively verify that this is the case as far as I am aware.

I am curious about other opinions but (until I here new arguments) I vote for enabling builds of those packages.

I am totally happy with maintaining a deny-list of packages where we know that we don‘t have the right to distribute them. But since we are processing packages in an automated fashion I can‘t imagine that we will get in a lot of trouble if we assume that it’s not the case.

I also would think that it is an unreasonable amount of work to figure out unclear and inconsistent license identifiers. In cases where we care about it, I think it is more productive to convince upstream to finally switch to SPDX-license identifiers which we can process properly and are not ambiguous.

@maralorn
Copy link
Member Author

The current state has been like this for years. So it will probably fine if we keep it for a while longer. Considering that we have 2 weeks left until branch-off now is not the time to make rush decisions breaking packages.

So I suggest to let this rest until after branch-off and then we find a more principled solution.

I fear there is a consensus that we somehow need to default to regard this packages as unfree but should find an easy solution to override them with their correct license, once we have figured the license out.

sternenseemann and others added 21 commits May 10, 2021 13:39
… to Maybe Int

The buildstatus may be `null` from Hydra if the Build hasn't finished
running yet.
Co-authored-by: Dennis Gosnell <cdep.illabout@gmail.com>
This commit has been generated by maintainers/scripts/haskell/mark-broken.sh
This commit has been generated by maintainers/scripts/haskell/mark-broken.sh
@maralorn
Copy link
Member Author

@thoughtpolice I tried fixing tamarin-prover, which hadn‘t built since January.

The problem is that tamarin-prover 1.6.0 is not compatible with maude 3.1.0 which we have in nixpkgs now. I can get it to build by disabling the checks, but it looks like the program won‘t work correctly then. So I opted to err on the save side and marked the package broken. Please feel free to choose a different solution at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants