Check rustbook links on all platforms when running locally

[mati865]

cc #62739

[mati865]

cc @mark-i-m

[rust-highfive]

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

[rust-highfive]

⚠️ Warning ⚠️

• These commits modify submodules.

[alexcrichton]

We have historically disabled this because of the significant amount of time it takes to execute on CI. Is there a good grasp on how much time this adds to CI builders to execute this step?

[mati865]

Is there a good grasp on how much time this adds to CI builders to execute this step?

It's hard to tell because it's not a separate step (it's part of rustbook).

---

@ehuss had troubles with current approach:

this is especially awkward for me to deal with, because it only runs on linux, which is not my native platform.

---

I haven't thought about it earlier but maybe it could be disabled on CI (except 1 job) and still ran locally?

My idea is to make linkcheck a feature of rustbook, it would be enabled unless some kind of environment variable or configuration option is present.

[alexcrichton]

That seems reasonable to me, but I think it'd also be reasonable to do a little work and basically execute this locally to see how long it takes to see whether it's necessary.

[mati865]

On my box (lldb tests are failing so I excluded them), ./x.py clean was ran before every test:

• with linkcheck ./x.py test --exclude src/test/debuginfo 10944,00s user 875,25s system 707% cpu 27:51,09 total
• without linkcheck ./x.py test --exclude src/test/debuginfo 10623,22s user 868,37s system 698% cpu 27:25,33 total

Linkcheck was nuked out of existence with this diff: https://gist.github.com/mati865/00fd7281723a4922a719902ad5c3d4ce

So it will increase build time by ~1.6%.

[alexcrichton]

That looks like linkcheck is taking 300s (5 minutes), is that right? If so that's definitey much too long to execute on each builder running tests.

[mati865]

I don't know about Rust CI but on my machine it increased build time by 1.6% (25,7 seconds) to 27 minutes 51 seconds total time.

[alexcrichton]

Sorry I'm having trouble parsing this. Is the data here showing that your build increased by 30 seconds after enabling this? If that's the case this seems fine to enable, that's just margin of error.

[mati865]

Generally you want to look for real in bash or total in zsh (this case).
10944,00s user 875,25s system 707% cpu 27:51,09 total means the command spent 10944s CPU time in user, 875s CPU time in kernel mode, avg CPU usage was 707% and clock time was 27 minutes 51 seconds.

Ryzen 2700X is much more powerful CPU than what Rust have as the CI so it will definitely take longer but I'm not able tell how much longer.

[alexcrichton]

So.... your local build took 30 seconds more, right?

If that's true then my thinking is the same, 30 seconds of a 30 minute build is margin of error which means this has basically no real impact. I'm just trying to confirm because I literally didn't understand "27,51s" before, I haven't seen that syntax to denote "27 minutes and 51 seconds" before myself.

[mati865]

Right, it should have been: "Increased build time by 1.6% (25,7 seconds) from 27 minutes 25 seconds to 27 minutes 51 seconds total time.

[alexcrichton]

@bors: r+

[bors]

📌 Commit 2282c213e126362433f2b3a1a9434433b14fdb4a has been approved by alexcrichton

[mark-i-m]

Sorry, I'm a bit late to the game. The CI time for the guide is about 2.5m right now. I actually would prefer not to run linkcheck on all platforms because it increases the odds of getting a spurious failure or rate limiting. I like the idea mentioned above of turning it off with an env var for most builders.

…

On Fri, Aug 2, 2019, 11:29 AM Alex Crichton ***@***.***> wrote: @bors <https://github.com/bors>: r+ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62950?email_source=notifications&email_token=ACDLHQFPCV42WVAEYBGNUU3QCROG3A5CNFSM4IGTKIW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OHJOY#issuecomment-517764283>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACDLHQEM5LI6DWE66NB64BLQCROG3ANCNFSM4IGTKIWQ> .

[alexcrichton]

@bors: r-

[mati865]

Should be ready for review.

[mati865]

Changed the way the feature is enabled and improved CI detection tu reuse bootstrap check in the last commit.
I'll squash the commits when this is accepted.

[alexcrichton]

@bors: delegate+

Looks good to me!

[bors]

✌️ @mati865 can now approve this pull request

[mati865]

Squashed and rebased.

@bors r=alexcrichton

[bors]

📌 Commit c7e16c5 has been approved by alexcrichton

[mark-i-m]

cc @ehuss