Automatically mark packages broken #185049
Assignees
Comments
The idea here was to reuse the way we handle it for aliases. I'm open to other solutions of course. |
|
I missed a thing here:
I think we could do the same as with aliases here, with a clear timestamp that's easily greppable it could make it easier to do cleanups, I guess. |
|
Oooh, I love this idea ❤️! To avoid duplication, would it make sense to base nixpkgs-mark-broken off of the json output of hydra-check? Also, what should we do in the rare cases where the package can build locally, but fails to build in hydra for whatever reason. For example, Responses to Open questions
I think this is reasonable, this is better than the situation where you manually mark it as
I think that should be ok, this could happen anyway with manually marking a package as broken, right? Maybe it could be useful to have a bot that checks when a PR touches a package that has been marked as broken after the merge base? |
|
Not that I think it will be that helpful for the general case, but we have an elaborate system to automatically mark Haskell packages as broken based on the build results from Hydra.
We have a document describing how we maintain the Haskell stuff in Nixpkgs (although only part of it is about marking things as broken): https://github.com/NixOS/nixpkgs/blob/35e26e2fea230ec7325e1811faa86b6072133927/pkgs/development/haskell-modules/HACKING.md Here's an example build status report where you can see some Haskell packages (that have maintainers) have failed to build, so they need to either be fixed or marked broken: #180512 (comment) After using this system for about a year or so, the biggest pain point is that Hydra is quite flaky, especially for darwin and aarch64-linux builds. It is very common for us to see build failures on darwin and aarch64-linux that are fixed just by restarting the build on Hydra. It is very frustrating (especially as someone who doesn't use other of those systems). |
Maybe it makes sense to set |
|
Oh yeah that could work! It might be a bit awkward for pygls though because it builds on Linux, but I guess we could just use something like |
Thanks for mentioning your tools! I think the way I'd like to address flakiness is to keep track of how long builds are broken and not marking one-offs as broken. If a build consistently fails for > x time, only then it should be marked broken, I think. |
1 task
|
See #203997 for a first PR related to this. Tool is very much WIP still, but hopefully you find it a good start. I also generated a list: List of broken packages (sorted by broken since) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Intro
I'm considering building some kind of tool that can automatically mark a package broken if it fails to build.
I'd like some feedback as to how such a tool should work and what it should do exactly.
Requirements
meta.brokenfor the platforms it's supported formeta.brokenfor the platforms where the build fails, e.g. iflinuxis supported andaarch64-linuxbuilds butx86_64-linuxdoesn't, the result should bemeta.broken = (stdenv.isLinux && stdenv.isx86_64);(or something that has the same meaning)Open questions
meta.broken = true;to mark broken for all platforms? I think it makes sense since we otherwise cannot track it it's healthy anyway.Feedback (from matrix)
git blame? It has a timestamp by itself. Anyway, should also be easy to add a date after thebroken = <platform>.Initial draft
FYI, I already made an initial implementation that can make a list of all packages on trunk with their build status and can then filter the broken packages out of that list: https://github.com/Mindavi/nixpkgs-mark-broken. Doesn't have any documentation yet, still hacking on it. But I thought it might be useful to discuss with the community if they have some ideas/opinions before submitting any PRs with this.
Related issues
The text was updated successfully, but these errors were encountered: