hydra-unstable: 2020-06-23 -> 2020-07-28 #93945

ivan · 2020-07-27T02:45:56Z

Motivation for this change

This fixes hydra-unstable on master, which currently does not build.

This adds a nixHydra because Hydra does not build with nixStable or nixUnstable: NixOS/hydra#794

Things done

edolstra · 2020-07-27T07:27:44Z

I would prefer to remove hydra-unstable from Nixpkgs. I don't think we should maintain yet another version of Nix in Nixpkgs just for a pretty niche use case.

ivan · 2020-07-27T07:52:25Z

I think it might be too late to leave it unfixed? The hydra module and release notes for 20.03 already strongly encouraged a migration to hydra-unstable, where hydra-unstable is

  hydra-unstable = callPackage ./common.nix {
    version = "2020-06-23";

I don't know how, or if, I could migrate back to an older Hydra.

Also, we could remove or alias nixHydra after NixOS/hydra#794 gets fixed.

cc @Ma27

Ma27 · 2020-07-27T11:24:18Z

@edolstra we had this discussion in the past and while I understand your concerns, I'm still against it (for now at least) for the following reasons:

Hydra doesn't have any release model. This package in nixpkgs is maintained (by me currently), so people don't have to regularly check for new revisions from the git repo. Also, the implementation in nixpkgs has an upgrade-path for Hydra's schema change.
I know that hydra can (and probably should) be distributed using flakes. I actually use flakes for a while now and don't get me wrong, I really started to like this. However this feature isn't stable yet and I occasionally hit some problems when using nixUnstable by default, so the majority of average users can't really use this IMHO (I'd reconsider this as soon as there's a stable nix release including flakes (not hidden behind an experimental flag) which landed in a stable NixOS release).

I don't know how, or if, I could migrate back to an older Hydra.

That's not possible due to the DB schema changes from ~February 2020. But AFAIU Eelco prefers to distribute Hydra using flakes (as mentioned above, this is perfectly understandable, but IMHO not viable for the average users at the moment), not a downgrade to an older Hydra rev.

Also, we could remove or alias nixHydra after NixOS/hydra#794 gets fixed.

Actually, I'm not sure if we really want to do that: whenever nixUnstable gets bumped, Hydra tends to break. This is definetely not a complaint though, I really appreciate your work @edolstra (and I know how time-consuming OSS work can be), but for me as a maintainer of this (legacy) package, such a mechanism would be fairly helpful.

However there are two more things to discuss before merging:

I think it's better to expose nixHydra as hydra-unstable.passthru.nixHydra, using this helper-package for anything else must not be advertised (which is somehow the case by adding a top-level attribute) IMHO.
I really need to test if this causes conflicts between the Nix used by Hydra and the Nix used as daemon. @edolstra do you think this can potentially create issues?

edolstra · 2020-07-27T11:34:51Z

Note that I'm not talking about removing hydra, just hydra-unstable. The number of people who need to run the bleeding edge is probably pretty small. And if they can't or don't want to use flakes, they can use a git checkout of hydra master. (That's what we used to do for hydra.nixos.org before flakes.)

Ma27 · 2020-07-27T11:37:03Z

Well, but Hydra master is actually the one with flakes, right? And what would you choose in our case then? There are still no releases.

This fixes the build: config.status: creating hydra-config.h config.status: executing depfiles commands config.status: executing libtool commands config.status: executing executable-scripts commands building build flags: -j8 -l8 SHELL=/nix/store/c4wxsn4jays9j31y5z9f83nr2cp7l4pa-bash-4.4-p23/bin/bash make all-recursive make[1]: Entering directory '/build/source' Making all in src make[2]: Entering directory '/build/source/src' Making all in hydra-evaluator make[3]: Entering directory '/build/source/src/hydra-evaluator' g++ -DHAVE_CONFIG_H -I. -I../.. -std=c++17 -I/nix/store/2xhb4hlskn33pbyph36v4wxcan56dnrw-boehm-gc-8.0.4-dev/include -I/nix/store/5rjgsqjdm71cflfb68q7m771a1rqcsk3-nix-2.4pre20200719_a79b6dd-dev/include/nix -Wall -I ../libhydra -Wno-deprecated-declarations -g -O2 -std=c++17 -include nix/config.h -c -o hydra_evaluator-hydra-evaluator.o `test -f 'hydra-evaluator.cc' || echo './'`hydra-evaluator.cc hydra-evaluator.cc:27:27: error: template argument 1 is invalid 27 | std::unique_ptr<Config> config; | ^ hydra-evaluator.cc:27:27: error: template argument 2 is invalid hydra-evaluator.cc: In constructor 'Evaluator::Evaluator()': hydra-evaluator.cc:61:56: error: base operand of '->' is not a pointer 61 | , maxEvals(std::max((size_t) 1, (size_t) config->getIntOption("max_concurrent_evals", 4))) | ^~ hydra-evaluator.cc:60:44: error: invalid user-defined conversion from 'std::_MakeUniq<Config>::__single_object' {aka 'std::unique_ptr<Config, std::default_delete<Config> >'} to 'int' [-fpermissive] 60 | : config(std::make_unique<::Config>()) | ~~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /nix/store/3krz9s8ni3bqy4hy35ycmq8assrrb4f6-gcc-9.3.0/include/c++/9.3.0/memory:80, from /nix/store/d97d0wixvlprz59z57maqj1pmda55r3b-libpqxx-6.4.5/include/pqxx/binarystring.hxx:17, from /nix/store/d97d0wixvlprz59z57maqj1pmda55r3b-libpqxx-6.4.5/include/pqxx/binarystring:4, from /nix/store/d97d0wixvlprz59z57maqj1pmda55r3b-libpqxx-6.4.5/include/pqxx/pqxx:3, from ../libhydra/db.hh:3, from hydra-evaluator.cc:1: /nix/store/3krz9s8ni3bqy4hy35ycmq8assrrb4f6-gcc-9.3.0/include/c++/9.3.0/bits/unique_ptr.h:374:16: note: candidate is: 'std::unique_ptr<_Tp, _Dp>::operator bool() const [with _Tp = Config; _Dp = std::default_delete<Config>]' <near match> 374 | explicit operator bool() const noexcept | ^~~~~~~~ /nix/store/3krz9s8ni3bqy4hy35ycmq8assrrb4f6-gcc-9.3.0/include/c++/9.3.0/bits/unique_ptr.h:374:16: note: return type 'bool' of explicit conversion function cannot be converted to 'int' with a qualification conversion make[3]: *** [Makefile:440: hydra_evaluator-hydra-evaluator.o] Error 1 make[3]: Leaving directory '/build/source/src/hydra-evaluator' make[2]: *** [Makefile:360: all-recursive] Error 1 make[2]: Leaving directory '/build/source/src' make[1]: *** [Makefile:414: all-recursive] Error 1 make[1]: Leaving directory '/build/source' make: *** [Makefile:344: all] Error 2 builder for '/nix/store/g967cc3j6rc3nnpx2s4klpr03ig9zzyp-hydra-2020-06-23.drv' failed with exit code 2

ivan · 2020-07-28T23:48:14Z

@edolstra thank you very much for the fix in Hydra. I have updated this PR to just bump hydra-unstable.

Ma27 · 2020-07-30T22:08:56Z

Thanks! Ported to stable as 38516a2.

thefloweringash · 2020-08-03T17:14:20Z

Did this change the setup requirements for signing? I have a hydra running on 20.03 and I’m seeing failures when building on remote builders due to paths lacking signatures. See example build and hydra-queue-runner log.

The bump includes a change to how paths are copied, which looks related.

Ma27 · 2020-08-03T17:27:52Z

Thanks for your report. Will take a look today or tomorrow.

Ma27 · 2020-08-03T21:12:44Z

@thefloweringash is it possible that this is related to the NoCheckSigs-argument in the old behavior? If I understand this correctly, you could check if adding a single path like this to the store might fix the issue:

destStore->addToStore(info, from, NoRepair, NoCheckSigs);

For further reference you may want to look at the "old" behavior: https://github.com/NixOS/nix/blob/699fc89b394ef5347ff0508abe389e77a7cde09e/src/libstore/export-import.cc#L91

If this fixes your issue, please file a PR against NixOS/hydra (I'd also apply this as patch here if needed).

NixOS/nixpkgs#93945 (comment)

edolstra · 2020-08-04T08:54:23Z

@Ma27 Thanks, I've added the NoCheckSigs back.

Ma27 · 2020-08-04T08:56:40Z

Awesome, thanks! Will update the package later today :)

Moving just one patch forward to fix importing store-paths from `hydra-queue-runner`[1]. The other patches[2] require a newer `pkgs.nixUnstable` and can't be used currently due to this. [1] #93945 (comment) [2] NixOS/hydra@77c33c1...4b58130

Moving just one patch forward to fix importing store-paths from `hydra-queue-runner`[1]. The other patches[2] require a newer `pkgs.nixUnstable` and can't be used currently due to this. [1] #93945 (comment) [2] NixOS/hydra@77c33c1...4b58130 (cherry picked from commit ae9ab32)

Ma27 · 2020-08-04T15:20:16Z

@thefloweringash fixed on master in ae9ab32 and on release-20.03 in b8151a4.

thefloweringash · 2020-08-04T15:33:41Z

Thanks for the quick fix!

ofborg bot added the 8.has: package (new) This PR adds a new package label Jul 27, 2020

ofborg bot requested a review from edolstra July 27, 2020 02:53

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Jul 27, 2020

ivan force-pushed the hydra-unstable-fix branch from 0389167 to 0d7bdd6 Compare July 28, 2020 23:41

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1 and removed 10.rebuild-darwin: 1 10.rebuild-darwin: 1-10 labels Jul 28, 2020

ivan changed the title ~~hydra-unstable: 2020-06-23 -> 2020-07-14 and add nixHydra~~ hydra-unstable: 2020-06-23 -> 2020-07-28 Jul 29, 2020

Ma27 approved these changes Jul 30, 2020

View reviewed changes

Ma27 added the 9.needs: port to stable A PR needs a backport to the stable release. label Jul 30, 2020

Ma27 merged commit 8211932 into NixOS:master Jul 30, 2020

Ma27 added 8.has: port to stable A PR already has a backport to the stable release. and removed 9.needs: port to stable A PR needs a backport to the stable release. labels Jul 30, 2020

edolstra added a commit to NixOS/hydra that referenced this pull request Aug 4, 2020

Restore NoCheckSigs

77c33c1

NixOS/nixpkgs#93945 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hydra-unstable: 2020-06-23 -> 2020-07-28 #93945

hydra-unstable: 2020-06-23 -> 2020-07-28 #93945

ivan commented Jul 27, 2020

edolstra commented Jul 27, 2020

ivan commented Jul 27, 2020 •

edited

Loading

Ma27 commented Jul 27, 2020

edolstra commented Jul 27, 2020

Ma27 commented Jul 27, 2020

ivan commented Jul 28, 2020

Ma27 commented Jul 30, 2020

thefloweringash commented Aug 3, 2020

Ma27 commented Aug 3, 2020

Ma27 commented Aug 3, 2020

edolstra commented Aug 4, 2020

Ma27 commented Aug 4, 2020

Ma27 commented Aug 4, 2020

thefloweringash commented Aug 4, 2020

hydra-unstable: 2020-06-23 -> 2020-07-28 #93945

hydra-unstable: 2020-06-23 -> 2020-07-28 #93945

Conversation

ivan commented Jul 27, 2020

Motivation for this change

Things done

edolstra commented Jul 27, 2020

ivan commented Jul 27, 2020 • edited Loading

Ma27 commented Jul 27, 2020

edolstra commented Jul 27, 2020

Ma27 commented Jul 27, 2020

ivan commented Jul 28, 2020

Ma27 commented Jul 30, 2020

thefloweringash commented Aug 3, 2020

Ma27 commented Aug 3, 2020

Ma27 commented Aug 3, 2020

edolstra commented Aug 4, 2020

Ma27 commented Aug 4, 2020

Ma27 commented Aug 4, 2020

thefloweringash commented Aug 4, 2020

ivan commented Jul 27, 2020 •

edited

Loading