New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

Open
wants to merge 6 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@michaelwoerister
Copy link
Contributor

michaelwoerister commented Jan 11, 2019

Some cross-language run-make tests need a Clang compiler that matches the LLVM version of rustc. Since such a compiler usually isn't available these tests (marked with the needs-matching-clang
directive) are ignored by default.

For some CI jobs we do need these tests to run unconditionally though. In order to support this a --force-clang-based-tests flag is added to compiletest. If this flag is specified, compiletest will fail if it can't detect an appropriate version of Clang.

@rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them?

cc #57438

r? @alexcrichton

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Jan 11, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:0a6a7b73:start=1547211744077826380,finish=1547211813523019759,duration=69445193379
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---

[00:03:20] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:20] tidy error: /checkout/src/bootstrap/test.rs:1109: line longer than 100 chars
[00:03:22] some tidy checks failed
[00:03:22] 
[00:03:22] 
[00:03:22] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:22] 
[00:03:22] 
[00:03:22] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:22] Build completed unsuccessfully in 0:00:46
[00:03:22] Build completed unsuccessfully in 0:00:46
[00:03:22] make: *** [tidy] Error 1
[00:03:22] Makefile:69: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:17e14d0e
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Fri Jan 11 13:07:04 UTC 2019
---
travis_time:end:11d9d38a:start=1547212024644878298,finish=1547212024649309522,duration=4431224
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:11d00126
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2eccbd58
travis_time:start:2eccbd58
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:07f89efa
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:xlto-tests branch from 6491bc8 to 8fc4d91 Jan 11, 2019


if let Some(var) = env::var_os("RUSTBUILD_FORCE_CLANG_BASED_TESTS") {
match &var.to_string_lossy()[..] {
"1" | "yes" | "on" => {

This comment has been minimized.

@alexcrichton

alexcrichton Jan 11, 2019

Member

Heh since this is an internal env var I think matching on only "yes" or "1" is probably fine, we tend to not really need that much error handling and support for internal features like this

This comment has been minimized.

@michaelwoerister

michaelwoerister Jan 14, 2019

Contributor

I wanted to try and make it less likely to have a typo causing things to be silently ignored. What would be really needed though would be fuzzy matching on env var names...

This comment has been minimized.

@alexcrichton

alexcrichton Jan 14, 2019

Member

True yeah, although we could also just check for existence here instead of checking for value perhaps?

This comment has been minimized.

@michaelwoerister

michaelwoerister Jan 15, 2019

Contributor

That's what wanted to do at first but then I remembered the CARGO_INCREMENTAL=0 confusion. Since it's not really complicated to support a few different options here, I just did that.

Show resolved Hide resolved src/tools/compiletest/src/main.rs Outdated

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:xlto-tests branch 2 times, most recently from fc95038 to 4ad6205 Jan 15, 2019

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

michaelwoerister commented Jan 15, 2019

I've simplified how Clang-based tests are set up. There's just a --run-clang-based-tests-with option for compiletest. If the option is supplied, clang-tests are run with the given clang executable. If it is not supplied, the tests are ignored.

Which builders should we enable the tests on?

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 15, 2019

Currently I believe we only enable LLDB on the dist builders for OSX, but we'll probably want to pick a Linux builder which isn't taking too long today and run it there. I don't think it particularly matters which one we choose!

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

michaelwoerister commented Jan 16, 2019

Oh no, one run-make-fulldeps test ignored, but it doesn't say which one :(

@@ -16,9 +16,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh

ENV RUSTBUILD_FORCE_CLANG_BASED_TESTS yes

This comment has been minimized.

@alexcrichton

alexcrichton Jan 17, 2019

Member

Checking a recent test run this builder is already running too long at 2h11m, perhaps we can pick a faster builder like x86_64-gnu-debug to run these tests in?

This comment has been minimized.

@michaelwoerister

michaelwoerister Jan 17, 2019

Contributor

But that doesn't seem to run tests?

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 17, 2019

Was that when testing locally? Or when testing this on CI? (the Travis run for this PR doesn't include the builder activated here)

michaelwoerister added some commits Jan 11, 2019

compiletest: Support opt-in Clang-based run-make tests.
Some cross-language run-make tests need a Clang compiler that
matches the LLVM version of rustc. Since such a compiler usually
isn't available these tests (marked with the "needs-matching-clang"
directive) are ignored by default.

For some CI jobs we do need these tests to run unconditionally
though. In order to support this a --force-clang-based-tests flag
is added to compiletest. If this flag is specified, compiletest
will fail if it can't detect an appropriate version of Clang.
@michaelwoerister

This comment has been minimized.

Copy link
Contributor

michaelwoerister commented Jan 17, 2019

Was that when testing locally? Or when testing this on CI? (the Travis run for this PR doesn't include the builder activated here)

Oh right, the pre-check is using the LLVM-6.0 image.

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:xlto-tests branch from f4ef08a to 6e0ffb3 Jan 17, 2019

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

michaelwoerister commented Jan 17, 2019

I switched to the x86_64-debug builder and made it run the clang-based tests (and only those):
6e0ffb3#diff-f06eb47b50daaf84c58fe00653913115R30

Is that what you meant, @alexcrichton?

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 18, 2019

Ah true that wasn't running any tests... That looks fine, I really don't have an opinion about where this should go other than that this is quite a large ask for CI to build all of clang and run more tests. We need to be careful to not regress cycle time which is already really really bad. If possible can this run through @bors: try to get ballpark estimates on timing?

@michaelwoerister michaelwoerister force-pushed the michaelwoerister:xlto-tests branch from aec74bf to 821e715 Jan 22, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Jan 22, 2019

The job x86_64-gnu-debug of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:05b93990:start=1548167229922750386,finish=1548167301305231579,duration=71382481193
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-debug
---
[00:01:10] Step 7/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --enable-debug       --enable-lld       --enable-lldb       --enable-optimize
[00:01:10]  ---> Running in 8f73d3d91389
[00:01:10] Removing intermediate container 8f73d3d91389
[00:01:10]  ---> cc2ddca93b9d
[00:01:10] Step 8/8 : ENV SCRIPT   python2.7 ../x.py build &&   python2.7 ../x.py test src/test/run-make-fulldeps --test-args clang
[00:01:10] Removing intermediate container a9c725490877
[00:01:10]  ---> bf67eb77a819
[00:01:10] Successfully built bf67eb77a819
[00:01:10] Successfully tagged rust-ci:latest
[00:01:10] Successfully tagged rust-ci:latest
[00:01:10] Built container sha256:bf67eb77a819908ce3d199751d776becb041807bbc6198b9c2ec5351c31c7937
[00:01:10] Uploading finished image to s3://rust-lang-ci-sccache2/docker/443850f43276f1b1220ffd0cf910f2f5a59531a55b07fe397c725d708ad26207600e68981d9be3f02cc20d364dbf6404db559718f4ee2b09acee812129e0468e
[00:01:42] upload failed: - to s3://rust-lang-ci-sccache2/docker/443850f43276f1b1220ffd0cf910f2f5a59531a55b07fe397c725d708ad26207600e68981d9be3f02cc20d364dbf6404db559718f4ee2b09acee812129e0468e Unable to locate credentials

[00:01:42] travis_time:end:07ac3814:start=1548167321659331995,finish=1548167412685253398,duration=91025921403
[CI_JOB_NAME=x86_64-gnu-debug]
[00:01:43] [CI_JOB_NAME=x86_64-gnu-debug]
---
[00:34:40] -- Looking for sys/resource.h - found
[00:34:40] -- Clang version: 8.0.0
[00:34:40] -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG
[00:34:40] -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG - Failed
[00:34:40] -- Could NOT find Z3 (missing:  Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is exact version "4.7.1")
[00:34:41] CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
[00:34:41]   Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
[00:34:41] Call Stack (most recent call first):
[00:34:41]   /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
[00:34:41]   /usr/share/cmake-3.5/Modules/FindPythonLibs.cmake:265 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
[00:34:41]   /checkout/src/tools/lldb/cmake/modules/LLDBConfig.cmake:181 (find_package)
[00:34:41] 
[00:34:41] 
[00:34:41] -- Configuring incomplete, errors occurred!
[00:34:41] -- Configuring incomplete, errors occurred!
[00:34:41] See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeOutput.log".
[00:34:41] See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeError.log".
[00:34:41] command did not execute successfully, got: exit code: 1
[00:34:41] 
[00:34:41] 
[00:34:41] build script failed, must exit now', /cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.33/src/lib.rs:773:5
[00:34:41]  finished in 10.831
[00:34:41] travis_fold:end:llvm

[00:34:41] travis_time:end:llvm:start=1548169380711736177,finish=1548169391543474609,duration=10831738432
---
travis_time:end:04b08978:start=1548169392846867960,finish=1548169392853672676,duration=6804716
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0c07d7d8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0bfb976c
travis_time:start:0bfb976c
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:1af8b019
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

michaelwoerister commented Jan 22, 2019

Huh, don't know what that is about. Cmake can't find python?

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 22, 2019

Oh the containers all have varying amounts of software on them, you may need to update one aspect here or there to install more things like python. Also FWIW PR builds won't be great for timing because they don't have access to sccache credentials which means they can't actually write any contents to the cache, only pull them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment