Better 'required' job that includes everything #976

michaelpj · 2020-12-17T17:25:51Z

This requires explicitly identifying the things that don't work and
removing them. I used our own disabled attribute, because broken is
weird and breaks evaluation, which isn't really what we want.

This requires explicitly identifying the things tht don't work and removing them. I used our own `disabled` attribute, because `broken` is weird and breaks evaluation, which isn't really what we want.

michaelpj · 2020-12-17T17:25:56Z

bors try

michaelpj · 2020-12-17T17:31:50Z

bors try

iohk-bors · 2020-12-17T17:31:52Z

try

Already running a review

michaelpj · 2020-12-17T17:33:46Z

bors try

iohk-bors · 2020-12-17T17:33:47Z

try

Already running a review

michaelpj · 2020-12-17T17:43:10Z

bors try-

michaelpj · 2020-12-17T17:50:34Z

bors try

iohk-bors · 2020-12-17T18:28:11Z

try

Build failed:

buildkite/haskell-dot-nix

rvl

Looks good to me -- if it builds!

rvl · 2020-12-18T02:41:10Z

ci.nix

+  isDisabled = d:
+    let meta = d.meta or {};
+    in meta.disabled or false;


This is a good idea - I was thinking about doing the same thing in cardano-wallet.

rvl · 2020-12-18T02:45:21Z

test/builder-haddock/default.nix

-    meta.disabled = stdenv.hostPlatform.isMusl;
+    meta = {
+      platforms = platforms.all;
+      disabled = stdenv.hostPlatform != stdenv.buildPlatform || stdenv.hostPlatform.isMusl;


I previously had a horrible time doing these sort of platform checks in order to remove jobs from the Hydra jobset.

What I think was happening, is that Hydra was getting the job list by evaluating with its native linux platform. So I was unable to remove jobs for e.g. macOS only.

But I was probably doing something wrong - is this working alright on Hydra?

I think it was working before! The platform it's evaluating on shouldn't matter (unless you use buildins.currentSystem - this is checking the host/build platforms from stdenv, so should be okay.

michaelpj · 2020-12-18T09:57:40Z

bors try

iohk-bors · 2020-12-18T10:33:32Z

try

Build failed:

buildkite/haskell-dot-nix

michaelpj · 2020-12-18T11:03:36Z

bors try

michaelpj · 2020-12-18T11:23:52Z

bors try

iohk-bors · 2020-12-18T11:23:55Z

try

Already running a review

michaelpj · 2020-12-18T11:36:39Z

bors try-

michaelpj · 2020-12-18T11:36:42Z

bors try

This doesn't actually matter, not sure why I was changing it.

iohk-bors · 2020-12-18T14:46:06Z

try

Build failed:

ci/hydra:Cardano:haskell-nix:required

michaelpj · 2020-12-18T17:16:03Z

bors try

iohk-bors · 2021-01-05T14:39:55Z

try

Timed out.

michaelpj · 2021-01-05T16:03:17Z

Interestingly, the commit status for the previous commit is okay, so it actually evaluated there. I guess it's just right on the edge of what's manageable? I guess I'm just not sure how to make it use less memory...

hamishmack · 2021-01-05T22:58:25Z

bors try

iohk-bors · 2021-01-05T23:01:20Z

try

Build failed:

buildkite/haskell-dot-nix

hamishmack · 2021-01-05T23:30:58Z

bors try

hamishmack · 2021-01-06T01:06:37Z

bors try

iohk-bors · 2021-01-06T01:06:39Z

try

Already running a review

hamishmack · 2021-01-06T01:07:31Z

bors try-

hamishmack · 2021-01-06T01:07:37Z

bors try

hamishmack · 2021-01-06T02:57:55Z

bors try

iohk-bors · 2021-01-06T02:57:57Z

try

Already running a review

hamishmack · 2021-01-06T02:58:08Z

bors try-

hamishmack · 2021-01-06T02:58:14Z

bors try

iohk-bors · 2021-01-06T03:29:32Z

try

Build succeeded:

hamishmack · 2021-01-06T03:41:16Z

OMG NixOS/hydra#715 worked! It seems to fix the OOM issue with having one massive required job.

This PR now also cuts the number of jobs from over 2000 to 216. This seems to greatly speeds up the time it takes small changes (where most stuff remains the same) to be processed by hydra.

michaelpj · 2021-01-06T09:59:52Z

🥇

Hamish you are a champion!

This requires explicitly identifying the things tht don't work and removing them. I used our own `disabled` attribute, because `broken` is weird and breaks evaluation, which isn't really what we want. Cuts the number of jobs from over 2000 to 216. This seems to greatly speeds up the time it takes small changes (where most stuff remains the same) to be processed by hydra. Uses NixOS/hydra#715 to reference the jobs in required by name avoiding OOM issues. Co-authored-by: Hamish Mackenzie <Hamish.Mackenzie@iohk.io>

michaelpj added 2 commits December 17, 2020 17:25

Better required job that includes everything

56a1d88

This requires explicitly identifying the things tht don't work and removing them. I used our own `disabled` attribute, because `broken` is weird and breaks evaluation, which isn't really what we want.

IFD level 1

1a67449

iohk-bors bot added a commit that referenced this pull request Dec 17, 2020

Try #976:

bad51cb

IFD level 2

6d65be0

iohk-bors bot added a commit that referenced this pull request Dec 17, 2020

Try #976:

7a5056b

rvl approved these changes Dec 18, 2020

View reviewed changes

iohk-bors bot added a commit that referenced this pull request Dec 18, 2020

Try #976:

50869da

Try to use smaller collect invocations

ca27186

iohk-bors bot added a commit that referenced this pull request Dec 18, 2020

Try #976:

77b4a77

try less

dcb15c8

iohk-bors bot added a commit that referenced this pull request Dec 18, 2020

Try #976:

208c074

IFD level 3

a42f978

This doesn't actually matter, not sure why I was changing it.

Turn off plan-nix for setup-deps too

eb73f94

iohk-bors bot added a commit that referenced this pull request Jan 5, 2021

Try #976:

f2988fd

Just two of the required jobs again

efab9f5

iohk-bors bot added a commit that referenced this pull request Jan 5, 2021

Try #976:

adc1e26

All jobs

019ba55

iohk-bors bot added a commit that referenced this pull request Jan 5, 2021

Try #976:

80fe336

hamishmack added 2 commits January 6, 2021 13:20

Add latestJobs to select those we want broken out

cd07ad4

Fix for buildkite

8a1310e

iohk-bors bot added a commit that referenced this pull request Jan 6, 2021

Try #976:

2f86466

Try NixOS/hydra#715

147b4f0

iohk-bors bot added a commit that referenced this pull request Jan 6, 2021

Try #976:

3f10e7e

hamishmack merged commit f6bc2f7 into master Jan 6, 2021

iohk-bors bot deleted the imp/better-release branch January 6, 2021 03:44

michaelpj mentioned this pull request Mar 3, 2021

GHC 9.0.1 #1032

Merged

Better 'required' job that includes everything #976

Better 'required' job that includes everything #976

Uh oh!

Conversation

michaelpj commented Dec 17, 2020

Uh oh!

michaelpj commented Dec 17, 2020

Uh oh!

michaelpj commented Dec 17, 2020

Uh oh!

iohk-bors bot commented Dec 17, 2020

try

Uh oh!

michaelpj commented Dec 17, 2020

Uh oh!

iohk-bors bot commented Dec 17, 2020

try

Uh oh!

michaelpj commented Dec 17, 2020

Uh oh!

michaelpj commented Dec 17, 2020

Uh oh!

iohk-bors bot commented Dec 17, 2020

try

Uh oh!

rvl left a comment

Choose a reason for hiding this comment

Uh oh!

rvl Dec 18, 2020

Choose a reason for hiding this comment

Uh oh!

rvl Dec 18, 2020

Choose a reason for hiding this comment

Uh oh!

michaelpj Dec 18, 2020

Choose a reason for hiding this comment

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

iohk-bors bot commented Dec 18, 2020

try

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

iohk-bors bot commented Dec 18, 2020

try

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

iohk-bors bot commented Dec 18, 2020

try

Uh oh!

michaelpj commented Dec 18, 2020

Uh oh!

iohk-bors bot commented Jan 5, 2021

try

Uh oh!

michaelpj commented Jan 5, 2021

Uh oh!

hamishmack commented Jan 5, 2021

Uh oh!

iohk-bors bot commented Jan 5, 2021

try

Uh oh!

hamishmack commented Jan 5, 2021

Uh oh!

hamishmack commented Jan 6, 2021

Uh oh!

iohk-bors bot commented Jan 6, 2021

try

Uh oh!

hamishmack commented Jan 6, 2021

Uh oh!

hamishmack commented Jan 6, 2021

Uh oh!

hamishmack commented Jan 6, 2021