Move coverage tests from run-make-fulldeps to run-make #85007

richkadel · 2021-05-06T20:30:06Z

The first commit was migrated from another PR that failed because CI had errors likely resulting from trying to run the coverage tests in run-make. (See: #84797 (comment))

So moving the tests should be done as it's own separate PR.

To attempt to resolve those CI errors, this PR also updates bootstrap to add LLVM library link path to run-make.

When moving coverage tests from run-make-fulldeps to run-make,
some targets failed in CI with an obscure message:

failed to execute command: "musl-g++" "-ffunction-sections"
"-fdata-sections" "-fPIC" "-m32" "-march=i686" "-Wl,-melf_i386"
"-static" "-Wa,-mrelax-relocations=no" "-print-file-name=libstdc++.a"

error: No such file or directory (os error 2)

The coverage tests include # needs-profiler-support and these are the
first run-make tests to require it (as far as I can tell).

There is a special case in bootstrap for adding the LLVM library link
path, and it applies to run-make-fulldeps. This commit adds it for
run-make as well.

r? @tmandry
cc: @wesleywiser @cuviper (regarding the change in bootstrap/tests.rs

Mark-Simulacrum · 2021-05-06T20:35:17Z

I forget - maybe we've already had this conversation - but while this doesn't seem like a bad step, it may also make sense to move the coverage tests into compiletest itself (similar to how we have mir-opt tests, in theory). I'm not quite sure what is too unique about them that they need to be driven by Make, but my guess is writing the driver in Rust is likely going to help avoid some of the problems that've cropped up over time, and will help avoid tying coverage tests to any 'related' dependencies (e.g., run-make tests depend on rustdoc, but coverage tests likely don't need rustdoc).

richkadel · 2021-05-06T20:40:16Z

Making a coverage-specific compiletest has been discussed a couple of times. It's a good idea, but there just hasn't been time (especially for me to get ramped up on how to write a compiletest, let alone migrate all of the logic from the Makefiles, which is non-trivial).

I don't think I've ever filed an Issue, though, so I'll do that at least.

cuviper · 2021-05-06T20:50:54Z

The first commit was migrated from another PR that failed because CI had errors likely resulting from trying to run the coverage tests in run-make. (See: #84797 (comment))

I don't understand why that would need the LLVM library path.

richkadel · 2021-05-06T20:51:01Z

Filed #85009

richkadel · 2021-05-06T20:54:07Z

I don't understand why that would need the LLVM library path.

To be honest, I don't know if it will. No one so far has been able to explain why the MUSL build target fails in CI (very early in the build) with:

failed to execute command: "musl-g++" "-ffunction-sections"
"-fdata-sections" "-fPIC" "-m32" "-march=i686" "-Wl,-melf_i386"
"-static" "-Wa,-mrelax-relocations=no" "-print-file-name=libstdc++.a"

error: No such file or directory (os error 2)

I don't know anything about the MUSL target, but I think libstdc++.a may be in the LLVM library path, and the coverage tests did not fail under run-make-fulldeps but they do fail under run-make; and I can't find any other substantial differences.

richkadel · 2021-05-06T20:55:37Z

Also, it's not 100% certain that the error: error: No such file or directory (os error 2) refers to libstdc++.a ... just a guess.

bors · 2021-05-07T10:39:48Z

☔ The latest upstream changes (presumably #85022) made this pull request unmergeable. Please resolve the merge conflicts.

Fixes: rust-lang#83830 The first commit was migrated from another PR that failed because CI had errors likely resulting from trying to run the coverage tests in run-make. (See: rust-lang#84797 (comment)) So moving the tests should be done as it's own separate PR. To attempt to resolve those CI errors, this PR also updates bootstrap to add LLVM library link path to run-make. When moving coverage tests from run-make-fulldeps to run-make, some targets failed in CI with an obscure message: failed to execute command: "musl-g++" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m32" "-march=i686" "-Wl,-melf_i386" "-static" "-Wa,-mrelax-relocations=no" "-print-file-name=libstdc++.a" error: No such file or directory (os error 2) The coverage tests include # needs-profiler-support and these are the first run-make tests to require it (as far as I can tell). There is a special case in bootstrap for adding the LLVM library link path, and it applies to run-make-fulldeps. This commit adds it for run-make as well.

When moving `coverage` tests from `run-make-fulldeps` to `run-make`, some targets failed in CI with an obscure message: failed to execute command: "musl-g++" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m32" "-march=i686" "-Wl,-melf_i386" "-static" "-Wa,-mrelax-relocations=no" "-print-file-name=libstdc++.a" error: No such file or directory (os error 2) The coverage tests include `# needs-profiler-support` and these are the first `run-make` tests to require it (as far as I can tell). There is a special case in `bootstrap` for adding the LLVM library link path, and it applies to `run-make-fulldeps`. This commit adds it for `run-make` as well.

tmandry · 2021-05-07T21:53:55Z

@bors r+ rollup=never

bors · 2021-05-07T21:53:56Z

📌 Commit 4881fc7 has been approved by tmandry

bors · 2021-05-08T05:38:45Z

⌛ Testing commit 4881fc7 with merge 8f0f5468b2b5cb2a26d03fbd3e99dd47d4568e6a...

rust-log-analyzer · 2021-05-08T05:42:57Z

The job dist-i586-gnu-i586-i686-musl failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

DirectMap1G:    53477376 kB
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s


failed to execute command: "musl-g++" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m32" "-march=i686" "-Wl,-melf_i386" "-static" "-Wa,-mrelax-relocations=no" "-print-file-name=libstdc++.a"
error: No such file or directory (os error 2)

failed to run: /checkout/obj/build/bootstrap/debug/bootstrap --stage 2 test --host= --target i586-unknown-linux-gnu,i686-unknown-linux-musl
Build completed unsuccessfully in 0:00:00

bors · 2021-05-08T05:44:20Z

💔 Test failed - checks-actions

richkadel · 2021-05-08T06:01:23Z

Well, I did try to test this using the target Docker image and run.sh, and some networking limitations prevented me from seeing the entire suite succeed, but it seemed to build way past the error shown here.

I don't know what else to try.

ghost · 2021-05-08T15:32:25Z

Well, I did try to test this using the target Docker image and run.sh, and some networking limitations prevented me from seeing the entire suite succeed, but it seemed to build way past the error shown here.

I'm able to reproduce this with DEPLOY=1, but I'm also not able to reproduce this without that environment variable.

ghost · 2021-05-08T16:13:57Z

Also, it's not 100% certain that the error: error: No such file or directory (os error 2) refers to libstdc++.a ... just a guess.

I guess it refers to musl-g++, because that builder builds LLVM only for x86_64-unknown-linux-gnu (there's no other "Building LLVM" message in the log I linked), not for musl, so musl-g++ could be absent.

richkadel · 2021-05-08T16:33:50Z

Thanks for the insights!

I did see a different error message (different from CI) in my Docker repro that seems to confirm musl-g++ is missing.

So I don't know if run-make is building for the wrong target (the platform has musl in its name), or if that platform never ran run-make-fulldeps (because of my network constraints, the test didn't get that far for me), or if run-make-fulldeps is configured differently, and building musl-g++ (if so, I need to apply that configuration different to run-make as well).

@hyd-dev Since you are able to reproduce, did all tests for dist-i586-gnu-i586-i686-musl finish successfully (without this PR)?

And if so, does it run run-make-fulldeps (and in particular run-make-fulldeps/coverage-* tests)?

ghost · 2021-05-08T16:46:50Z

without this PR

I guess you could just check the log of another (merged) PR: https://github.com/rust-lang-ci/rust/runs/2534368555 (looks like run-make-fulldeps was not run on CI).
I could also try locally, but I guess that would take some time (I didn't try to run it without this PR).

richkadel · 2021-05-08T17:05:06Z

You're right. Thank you. No need to check locally. That explains the difference at least.

richkadel · 2021-05-08T17:32:16Z

@jyn514 @petrochenkov - IIRC both of you recommended moving coverage tests from run-make-fulldeps to run-make.

I'm considering abandoning this PR and leaving the coverage tests in run-make-fulldeps since run-make may introduce multiple incompatibilities.

Please let me know if you have a better recommendation.

As recently revealed (see prior comment), apparently run-make runs on more targets than run-make-fulldeps, so the recent CI failures with this PR are because the coverage tests are attempted on platforms they've never targeted before.

I could try # ignore-musl but there's no guarantee that MUSL is the only platform that would fail.

I don't know if it's worth trying to identify ALL of the run-make targets that don't already run run-make-fulldeps, but I don't like the idea of trying and failing CI many times, and then generating a [set of directives that looks like this (7 ignores)](https://github.com/richkadel/rust/blob/1fa48cf181f1fe7e1aba133be199804c652fe55c/src/test/run-make/issue-36710/Makefile.

I don't see a clear, well understood path to porting tests from run-make-fulldeps to run-make.

Thanks in advance for your thoughts and recommendations!

richkadel · 2021-05-08T18:55:19Z

src/bootstrap/test.rs

@@ -1399,7 +1399,7 @@ note: if you're sure you want to do this, please open an issue as to why. In the
            // requirement, but the `-L` library path is not propagated across
            // separate compilations. We can add LLVM's library path to the
            // platform-specific environment variable as a workaround.
-            if !builder.config.dry_run && suite.ends_with("fulldeps") {
+            if !builder.config.dry_run && (suite == "ui-fulldeps" || mode == "run-make") {


This change should be removed. We now know this did not fix the CI issues in the MUSL target.

jyn514 · 2021-05-09T02:59:14Z

I don't understand run-make very well at all (I've also had trouble in #83775) and also I don't have time to review this.

bors · 2021-05-12T01:34:39Z

☔ The latest upstream changes (presumably #85199) made this pull request unmergeable. Please resolve the merge conflicts.

richkadel · 2021-05-12T01:37:23Z

No known way to resolve this, so I'm closing this PR.

rust-highfive assigned tmandry May 6, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 6, 2021

richkadel force-pushed the coverage-tests-to-run-make branch from cc894d1 to 8939a19 Compare May 6, 2021 20:31

richkadel force-pushed the coverage-tests-to-run-make branch from 8939a19 to 9e41596 Compare May 7, 2021 19:53

This comment has been minimized.

Sign in to view

richkadel force-pushed the coverage-tests-to-run-make branch from 9e41596 to baa07f7 Compare May 7, 2021 20:04

This comment has been minimized.

Sign in to view

richkadel force-pushed the coverage-tests-to-run-make branch from baa07f7 to 25692e8 Compare May 7, 2021 20:13

This comment has been minimized.

Sign in to view

richkadel force-pushed the coverage-tests-to-run-make branch from 25692e8 to 013aa9a Compare May 7, 2021 20:18

This comment has been minimized.

Sign in to view

richkadel added 2 commits May 7, 2021 13:31

richkadel force-pushed the coverage-tests-to-run-make branch from 013aa9a to 4881fc7 Compare May 7, 2021 20:32

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 7, 2021

bors removed the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label May 8, 2021

bors added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 8, 2021

richkadel commented May 8, 2021

View reviewed changes

richkadel closed this May 12, 2021

richkadel mentioned this pull request May 19, 2021

Move coverage tests from run-make-fulldeps to run-make #83830

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move coverage tests from run-make-fulldeps to run-make #85007

Move coverage tests from run-make-fulldeps to run-make #85007

richkadel commented May 6, 2021

Mark-Simulacrum commented May 6, 2021

richkadel commented May 6, 2021

cuviper commented May 6, 2021

richkadel commented May 6, 2021

richkadel commented May 6, 2021 •

edited

richkadel commented May 6, 2021

bors commented May 7, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tmandry commented May 7, 2021

bors commented May 7, 2021

bors commented May 8, 2021

rust-log-analyzer commented May 8, 2021

bors commented May 8, 2021

richkadel commented May 8, 2021

ghost commented May 8, 2021

ghost commented May 8, 2021

richkadel commented May 8, 2021

ghost commented May 8, 2021 •

edited by ghost

richkadel commented May 8, 2021

richkadel commented May 8, 2021

richkadel May 8, 2021

jyn514 commented May 9, 2021

bors commented May 12, 2021

richkadel commented May 12, 2021

Move coverage tests from run-make-fulldeps to run-make #85007

Move coverage tests from run-make-fulldeps to run-make #85007

Conversation

richkadel commented May 6, 2021

Mark-Simulacrum commented May 6, 2021

richkadel commented May 6, 2021

cuviper commented May 6, 2021

richkadel commented May 6, 2021

richkadel commented May 6, 2021 • edited

richkadel commented May 6, 2021

bors commented May 7, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tmandry commented May 7, 2021

bors commented May 7, 2021

bors commented May 8, 2021

rust-log-analyzer commented May 8, 2021

bors commented May 8, 2021

richkadel commented May 8, 2021

ghost commented May 8, 2021

ghost commented May 8, 2021

richkadel commented May 8, 2021

ghost commented May 8, 2021 • edited by ghost

richkadel commented May 8, 2021

richkadel commented May 8, 2021

richkadel May 8, 2021

Choose a reason for hiding this comment

jyn514 commented May 9, 2021

bors commented May 12, 2021

richkadel commented May 12, 2021

richkadel commented May 6, 2021 •

edited

ghost commented May 8, 2021 •

edited by ghost