azure: split slow linux builders

[pietroalbini]

This PR splits into two the 6 Linux builders that regularly approach or go over the 3 hours mark on Azure, using the make ci-subset-{1,2} already used by other builders on Windows.

r? @alexcrichton
cc #61185

[pietroalbini]

Fixed the build failure.

[alexcrichton]

I'm somewhat tempted to hold off on this if we can, it seems like it's going to be hard to manage what's split and what isn't over time. Additionally in terms of resources this is wasteful in the sense that the split builds have the same first halves.

I think we should probably first explore disabling assertions where possible and see if we can recover enough time that way. Distcheck for example has no need to turn on assertions, same with builders like cargo. I was hoping to look more into this today but it's looking like I may not have enough time today

[alexcrichton]

Did we forget to include this from before?

[pietroalbini]

No, on Travis x86_64-gnu-cargo is included inside x86_64-gnu-aux. Instead on AppVeyor, to speed things up, cargo is tested inside its own job. I replicated the same thing we do on AppVeyor in Azure Linux.

[pietroalbini]

Ok, let's wait on the results of disabling assertions.

@rustbot modify labels: +S-blocked -S-waiting-on-review

[alexcrichton]

Ok so to say again that I haven't forgotten about this:

• I took the current master branch, the merge of Fix duplicated bounds printing in rustdoc #59033
  • Travis - https://travis-ci.com/rust-lang/rust/builds/114110767
  • AppVeyor - https://ci.appveyor.com/project/rust-lang/rust/builds/25004114
• I scheduled a build on azure which is the full suite on try (to let everything finish)
• I scheduled a second build with llvm/debug assertions turned off, also on try

I'll circle back tomorrow with timings after that all runs overnight.

[alexcrichton]

OK scratch that. More spurious failures basically invalidated that run, so there's...

• a new full try build
• a new try build with assertions disabled
• A new merge base - Make cannot move errors more consistent with other borrowck errors #61136
  • https://travis-ci.com/rust-lang/rust/builds/114222742
  • https://ci.appveyor.com/project/rust-lang/rust/builds/25025060

[alexcrichton]

Ok got some good builds, although there's some outliers we've got a good set of data:

https://gist.github.com/alexcrichton/4350118b27fa2ae356cfde3d21f6e45d

The build with assertions had to rebuild some containers (namely the huge linux ones) so a few 6 hour builds are outliers. Relative times are relative to the first travis/appveyor column.

Some things:

• Something feels wrong about builders like i686-gnu. 4 cores on Travis is equivalent to 2 cores on Azure. The CPUs don't even look that far off from one another there (same clock rates). I think we really need to plumb through @johnterickson's script to get CPU usage over time across all builders.
• Same-host dist builds are all through the roof on AppVeyor. We're looking at a blanket 20 minute slowdown up to 40 minutes for dist-x86_64-musl
• Disabling assertions has a massive impact on Windows tests, often saving 30-40 minutes. (x86_64-mingw-2)
• Some windows tests remain stubborn (even after assertions are disabled) and execute for quite some time. (i686-msvc-2 and i686-mingw-2)

As for builders affected by this PR specifically:

	Travis CI/AppVeyor	Azure Pipelines (assertions)	Azure Pipelines (no assertions)
i686-gnu	2:28:47	3:19:05 (+0:50:18)	2:29:24 (+0:00:37)
i686-gnu-nopt	2:30:52	3:08:23 (+0:37:31)	2:21:25 (-0:09:27)
x86_64-gnu	2:25:52	2:53:57 (+0:28:05)	2:27:42 (+0:01:50)
x86_64-gnu-nopt	2:10:02	3:08:37 (+0:58:35)	2:25:52 (+0:15:50)
x86_64-gnu-distcheck	2:26:33	3:05:53 (+0:39:20)	2:28:32 (+0:01:59)

It looks like assertions makes up the lion's share of time differences for these builders, except maybe the x86_64-gnu-nopt one which may benefit from splitting.

---

The conclusions I'm drawing here are that we should disable LLVM/debug assertions by default on CI since we can't afford them and get such huge wins without them. We should still have a builder or two with them enabled, but perhaps don't run the full test suite and just a few things here and there. (also across platforms to cover OSX/Linux/MSVC at least).

Additionally we probably don't want to start the splitting as proposed in this PR just yet. I think disabling assertions gets us almost the way there. The last major hurdle will be getting the dist builders under control, which I'll be taking a look into more shortly.

[bors]

☔ The latest upstream changes (presumably #61772) made this pull request unmergeable. Please resolve the merge conflicts.

[pietroalbini]

Rebased, and also disabled Windows builders on auto and Windows and macOS builders on try.

r? @alexcrichton

[alexcrichton]

I talked with @pietroalbini on discord briefly about this, but I think we may not want to land this just yet. In our transition plan to Azure I think we're just going to eat a regression in compile times until the 4 core machines come online, and we're transitioning everything to Azure so speeding up some of the builders here won't translate to build time wins as long as at least one builder is lagging behind. I think the try changes can land at any time, but I've also included those in #62142 since they're required there as well.

[pietroalbini]

Yeah, closing this.