Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow checking of run-pass execution output in compiletest #63825

Merged
merged 7 commits into from Sep 4, 2019

Conversation

@nathanwhit
Copy link
Contributor

commented Aug 23, 2019

Closes #63751
Adds a check-run-results flag to compiletest headers, which if enabled checks the output of the execution of a run-pass test's binary against expected output.

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 23, 2019

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

@nathanwhit

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

@Mark-Simulacrum
Copy link
Member

left a comment

Thanks! A few comments, but this looks good overall.

@nathanwhit nathanwhit force-pushed the nathanwhit:check-run-results branch from 457bfce to 3dc3cf3 Aug 25, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 25, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-25T23:26:58.9031804Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-25T23:26:58.9238611Z ##[command]git config gc.auto 0
2019-08-25T23:26:58.9324471Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-25T23:26:58.9375356Z ##[command]git config --get-all http.proxy
2019-08-25T23:26:58.9518002Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63825/merge:refs/remotes/pull/63825/merge
---
2019-08-25T23:27:34.0738372Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-25T23:27:34.0738404Z 
2019-08-25T23:27:34.0738605Z   git checkout -b <new-branch-name>
2019-08-25T23:27:34.0738635Z 
2019-08-25T23:27:34.0738705Z HEAD is now at 1132f1b32 Merge 3dc3cf30a45108c45df0492900224515d03b79ea into 521d78407471cb78e9bbf47160f6aa23047ac499
2019-08-25T23:27:34.0910922Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-25T23:27:34.0914541Z ==============================================================================
2019-08-25T23:27:34.0914601Z Task         : Bash
2019-08-25T23:27:34.0914650Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-25T23:33:54.7659866Z    Compiling serde_json v1.0.40
2019-08-25T23:33:56.5941106Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2019-08-25T23:34:07.6290144Z     Finished release [optimized] target(s) in 1m 32s
2019-08-25T23:34:07.6364208Z tidy check
2019-08-25T23:34:08.1676169Z tidy error: /checkout/src/tools/compiletest/src/runtest.rs:4: line longer than 100 chars
2019-08-25T23:34:08.1677953Z tidy error: /checkout/src/tools/compiletest/src/runtest.rs:2942: line longer than 100 chars
2019-08-25T23:34:09.6212119Z some tidy checks failed
2019-08-25T23:34:09.6213546Z 
2019-08-25T23:34:09.6213546Z 
2019-08-25T23:34:09.6214853Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2019-08-25T23:34:09.6215467Z 
2019-08-25T23:34:09.6215653Z 
2019-08-25T23:34:09.6220405Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2019-08-25T23:34:09.6220891Z Build completed unsuccessfully in 0:01:35
2019-08-25T23:34:09.6220891Z Build completed unsuccessfully in 0:01:35
2019-08-25T23:34:09.6269335Z == clock drift check ==
2019-08-25T23:34:09.6283445Z   local time: Sun Aug 25 23:34:09 UTC 2019
2019-08-25T23:34:09.7768637Z   network time: Sun, 25 Aug 2019 23:34:09 GMT
2019-08-25T23:34:09.7776388Z == end clock drift check ==
2019-08-25T23:34:11.1112099Z ##[error]Bash exited with code '1'.
2019-08-25T23:34:11.1145239Z ##[section]Starting: Checkout
2019-08-25T23:34:11.1146981Z ==============================================================================
2019-08-25T23:34:11.1147033Z Task         : Get sources
2019-08-25T23:34:11.1147094Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@nathanwhit nathanwhit force-pushed the nathanwhit:check-run-results branch from 3dc3cf3 to 1c6b986 Aug 26, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 26, 2019

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-26T00:03:26.6088551Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-26T00:03:26.6292091Z ##[command]git config gc.auto 0
2019-08-26T00:03:26.6355363Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-26T00:03:26.6407770Z ##[command]git config --get-all http.proxy
2019-08-26T00:03:26.6533914Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63825/merge:refs/remotes/pull/63825/merge
---
2019-08-26T00:04:01.6262190Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-26T00:04:01.6262489Z 
2019-08-26T00:04:01.6262951Z   git checkout -b <new-branch-name>
2019-08-26T00:04:01.6263261Z 
2019-08-26T00:04:01.6263521Z HEAD is now at ce9430cbf Merge 1c6b98647b9e7cf44d2bbea5497e7ec1bae329f9 into 521d78407471cb78e9bbf47160f6aa23047ac499
2019-08-26T00:04:01.6429693Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-26T00:04:01.6432562Z ==============================================================================
2019-08-26T00:04:01.6432617Z Task         : Bash
2019-08-26T00:04:01.6432658Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-26T00:13:38.1510567Z configure: build.locked-deps    := True
2019-08-26T00:13:38.1510635Z configure: llvm.ccache          := sccache
2019-08-26T00:13:38.1510893Z configure: build.cargo-native-static := True
2019-08-26T00:13:38.1511129Z configure: dist.missing-tools   := True
2019-08-26T00:13:38.1511409Z configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
2019-08-26T00:13:38.1511534Z configure: writing `config.toml` in current directory
2019-08-26T00:13:38.1511582Z configure: 
2019-08-26T00:13:38.1511820Z configure: run `python /checkout/x.py --help`
2019-08-26T00:13:38.1511888Z configure: 
---
2019-08-26T00:22:15.1772440Z    Compiling serde v1.0.99
2019-08-26T00:22:24.1612047Z    Compiling serde_json v1.0.40
2019-08-26T00:22:25.9333716Z    Compiling rustfix v0.4.6
2019-08-26T00:22:29.6875956Z    Compiling compiletest v0.0.0 (/checkout/src/tools/compiletest)
2019-08-26T00:22:29.7226465Z error: expected `;`, found keyword `use`
2019-08-26T00:22:29.7227087Z   |
2019-08-26T00:22:29.7227087Z   |
2019-08-26T00:22:29.7227372Z 4 | use crate::common::{expected_output_path, UI_EXTENSIONS, UI_FIXED, UI_STDERR, UI_STDOUT}
2019-08-26T00:22:29.7227728Z   |                                                                                         - expected `;`
2019-08-26T00:22:29.7227990Z 5 | use crate::{UI_RUN_STDERR, UI_RUN_STDOUT};
2019-08-26T00:22:29.7228333Z 
2019-08-26T00:22:29.7229184Z error: aborting due to previous error
2019-08-26T00:22:29.7229229Z 
2019-08-26T00:22:29.7289733Z error: Could not compile `compiletest`.
---
2019-08-26T00:22:29.7365784Z == clock drift check ==
2019-08-26T00:22:29.7381807Z   local time: Mon Aug 26 00:22:29 UTC 2019
2019-08-26T00:22:30.0165878Z   network time: Mon, 26 Aug 2019 00:22:30 GMT
2019-08-26T00:22:30.0170539Z == end clock drift check ==
2019-08-26T00:22:31.4914643Z ##[error]Bash exited with code '1'.
2019-08-26T00:22:31.4957212Z ##[section]Starting: Checkout
2019-08-26T00:22:31.4958893Z ==============================================================================
2019-08-26T00:22:31.4958943Z Task         : Get sources
2019-08-26T00:22:31.4958986Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@nathanwhit nathanwhit force-pushed the nathanwhit:check-run-results branch from 1c6b986 to 29cc305 Aug 26, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 26, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-26T01:11:42.2304245Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-26T01:11:42.2485061Z ##[command]git config gc.auto 0
2019-08-26T01:11:42.2532536Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-26T01:11:42.2582171Z ##[command]git config --get-all http.proxy
2019-08-26T01:11:42.2707133Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63825/merge:refs/remotes/pull/63825/merge
---
2019-08-26T01:12:16.2442000Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-26T01:12:16.2442023Z 
2019-08-26T01:12:16.2442184Z   git checkout -b <new-branch-name>
2019-08-26T01:12:16.2442207Z 
2019-08-26T01:12:16.2442261Z HEAD is now at 258f84984 Merge 29cc305e936d9637f12f37e0fb6e38e04009d610 into 521d78407471cb78e9bbf47160f6aa23047ac499
2019-08-26T01:12:16.2581600Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-26T01:12:16.2584017Z ==============================================================================
2019-08-26T01:12:16.2584060Z Task         : Bash
2019-08-26T01:12:16.2584110Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-26T02:09:48.6788006Z .................................................................................................... 1500/8953
2019-08-26T02:09:53.9076357Z .................................................................................................... 1600/8953
2019-08-26T02:10:05.5501478Z .............................................i...............i...................................... 1700/8953
2019-08-26T02:10:13.0023819Z .................................................................................................... 1800/8953
2019-08-26T02:10:25.9442778Z .....................................iiiii.......................................................... 1900/8953
2019-08-26T02:10:35.4629534Z .................................................................................................... 2100/8953
2019-08-26T02:10:37.7014429Z .................................................................................................... 2200/8953
2019-08-26T02:10:41.5578349Z .................................................................................................... 2300/8953
2019-08-26T02:10:48.1215673Z .................................................................................................... 2400/8953
---
2019-08-26T02:13:29.9595274Z .........................i...............i.......................................................... 4700/8953
2019-08-26T02:13:40.6811563Z .................................................................................................... 4800/8953
2019-08-26T02:13:46.6843206Z .................................................................................................... 4900/8953
2019-08-26T02:13:56.3073241Z .................................................................................................... 5000/8953
2019-08-26T02:14:01.3981268Z ......ii.ii......................................................................................... 5100/8953
2019-08-26T02:14:14.4263835Z .................................................................................................... 5300/8953
2019-08-26T02:14:20.8232614Z ..............................................................i..................................... 5400/8953
2019-08-26T02:14:27.2372657Z .................................................................................................... 5500/8953
2019-08-26T02:14:34.3069123Z .................................................................................................... 5600/8953
2019-08-26T02:14:34.3069123Z .................................................................................................... 5600/8953
2019-08-26T02:14:44.2388744Z ........................................................ii...i..ii...........i...................... 5700/8953
2019-08-26T02:15:05.3834533Z .................................................................................................... 5900/8953
2019-08-26T02:15:09.8345842Z .................................................................................................... 6000/8953
2019-08-26T02:15:09.8345842Z .................................................................................................... 6000/8953
2019-08-26T02:15:16.2958225Z .........................................................i..ii...................................... 6100/8953
2019-08-26T02:15:41.1398436Z .................................................................................................... 6300/8953
2019-08-26T02:15:43.0168886Z ...i................................................................................................ 6400/8953
2019-08-26T02:15:44.9037814Z ...........................................................................i........................ 6500/8953
2019-08-26T02:15:47.2749594Z .................................................................................................... 6600/8953
---
2019-08-26T02:19:20.9718377Z 
2019-08-26T02:19:20.9719844Z ---- [ui] ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs stdout ----
2019-08-26T02:19:20.9720100Z diff of run.stderr:
2019-08-26T02:19:20.9720255Z 
2019-08-26T02:19:20.9720724Z - [$DIR/dbg-macro-expected-behavior.rs:20] Unit = Unit
2019-08-26T02:19:20.9721232Z - [$DIR/dbg-macro-expected-behavior.rs:21] a = Unit
2019-08-26T02:19:20.9721691Z - [$DIR/dbg-macro-expected-behavior.rs:27] Point{x: 42, y: 24,} = Point {
2019-08-26T02:19:20.9722464Z + [$DIR/dbg-macro-expected-behavior.rs:23] Unit = Unit
2019-08-26T02:19:20.9723086Z + [$DIR/dbg-macro-expected-behavior.rs:24] a = Unit
2019-08-26T02:19:20.9723678Z + [$DIR/dbg-macro-expected-behavior.rs:30] Point{x: 42, y: 24,} = Point {
2019-08-26T02:19:20.9723868Z 4     x: 42,
2019-08-26T02:19:20.9724181Z 5     y: 24,
2019-08-26T02:19:20.9724625Z 
2019-08-26T02:19:20.9724625Z 
2019-08-26T02:19:20.9725153Z - [$DIR/dbg-macro-expected-behavior.rs:28] b = Point {
2019-08-26T02:19:20.9725708Z + [$DIR/dbg-macro-expected-behavior.rs:31] b = Point {
2019-08-26T02:19:20.9761533Z 8     x: 42,
2019-08-26T02:19:20.9813326Z 9     y: 24,
2019-08-26T02:19:20.9813823Z 
2019-08-26T02:19:20.9813823Z 
2019-08-26T02:19:20.9814230Z - [$DIR/dbg-macro-expected-behavior.rs:36]
2019-08-26T02:19:20.9814450Z - [$DIR/dbg-macro-expected-behavior.rs:40] &a = NoCopy(
2019-08-26T02:19:20.9814828Z + [$DIR/dbg-macro-expected-behavior.rs:39]
2019-08-26T02:19:20.9815166Z + [$DIR/dbg-macro-expected-behavior.rs:43] &a = NoCopy(
2019-08-26T02:19:20.9815474Z 14 )
2019-08-26T02:19:20.9815474Z 14 )
2019-08-26T02:19:20.9815717Z - [$DIR/dbg-macro-expected-behavior.rs:40] dbg!(& a) = NoCopy(
2019-08-26T02:19:20.9816092Z + [$DIR/dbg-macro-expected-behavior.rs:43] dbg!(& a) = NoCopy(
2019-08-26T02:19:20.9816397Z 17 )
2019-08-26T02:19:20.9816397Z 17 )
2019-08-26T02:19:20.9816657Z - [$DIR/dbg-macro-expected-behavior.rs:45] f(&42) = 42
2019-08-26T02:19:20.9816999Z + [$DIR/dbg-macro-expected-behavior.rs:48] f(&42) = 42
2019-08-26T02:19:20.9817151Z 19 before
2019-08-26T02:19:20.9817508Z - [$DIR/dbg-macro-expected-behavior.rs:50] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-08-26T02:19:20.9817704Z - [$DIR/dbg-macro-expected-behavior.rs:58] ("Yeah",) = (
2019-08-26T02:19:20.9818089Z + [$DIR/dbg-macro-expected-behavior.rs:53] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-08-26T02:19:20.9818433Z + [$DIR/dbg-macro-expected-behavior.rs:61] ("Yeah",) = (
2019-08-26T02:19:20.9818584Z 22     "Yeah",
2019-08-26T02:19:20.9818723Z 23 )
2019-08-26T02:19:20.9819378Z - [$DIR/dbg-macro-expected-behavior.rs:61] 1 = 1
2019-08-26T02:19:20.9819840Z - [$DIR/dbg-macro-expected-behavior.rs:61] 2 = 2
2019-08-26T02:19:20.9821410Z - [$DIR/dbg-macro-expected-behavior.rs:65] 1u8 = 1
2019-08-26T02:19:20.9821878Z - [$DIR/dbg-macro-expected-behavior.rs:65] 2u32 = 2
2019-08-26T02:19:20.9824258Z - [$DIR/dbg-macro-expected-behavior.rs:65] "Yeah" = "Yeah"
2019-08-26T02:19:20.9825482Z + [$DIR/dbg-macro-expected-behavior.rs:64] 1 = 1
2019-08-26T02:19:20.9825867Z + [$DIR/dbg-macro-expected-behavior.rs:64] 2 = 2
2019-08-26T02:19:20.9826234Z + [$DIR/dbg-macro-expected-behavior.rs:68] 1u8 = 1
2019-08-26T02:19:20.9826594Z + [$DIR/dbg-macro-expected-behavior.rs:68] 2u32 = 2
2019-08-26T02:19:20.9826953Z + [$DIR/dbg-macro-expected-behavior.rs:68] "Yeah" = "Yeah"
2019-08-26T02:19:20.9827390Z 
2019-08-26T02:19:20.9827414Z 
2019-08-26T02:19:20.9827479Z The actual run.stderr differed from the expected run.stderr.
2019-08-26T02:19:20.9827479Z The actual run.stderr differed from the expected run.stderr.
2019-08-26T02:19:20.9828019Z Actual run.stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/dbg-macro-expected-behavior.run.stderr
2019-08-26T02:19:20.9875378Z error: 1 errors occured comparing run output.
2019-08-26T02:19:20.9875516Z status: exit code: 0
2019-08-26T02:19:20.9875994Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a"
2019-08-26T02:19:20.9876046Z stdout:
2019-08-26T02:19:20.9876046Z stdout:
2019-08-26T02:19:20.9876262Z ------------------------------------------
2019-08-26T02:19:20.9876909Z 
2019-08-26T02:19:20.9877176Z ------------------------------------------
2019-08-26T02:19:20.9877237Z stderr:
2019-08-26T02:19:20.9877422Z ------------------------------------------
2019-08-26T02:19:20.9877646Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:23] Unit = Unit
2019-08-26T02:19:20.9877882Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:24] a = Unit
2019-08-26T02:19:20.9878298Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:30] Point{x: 42, y: 24,} = Point {
2019-08-26T02:19:20.9878346Z     x: 42,
2019-08-26T02:19:20.9878392Z     y: 24,
2019-08-26T02:19:20.9878427Z }
2019-08-26T02:19:20.9878686Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:31] b = Point {
2019-08-26T02:19:20.9878742Z     x: 42,
2019-08-26T02:19:20.9878777Z     y: 24,
2019-08-26T02:19:20.9880478Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:39]
2019-08-26T02:19:20.9880478Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:39]
2019-08-26T02:19:20.9881330Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:43] &a = NoCopy(
2019-08-26T02:19:20.9881434Z )
2019-08-26T02:19:20.9881434Z )
2019-08-26T02:19:20.9881744Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:43] dbg!(& a) = NoCopy(
2019-08-26T02:19:20.9882206Z )
2019-08-26T02:19:20.9882206Z )
2019-08-26T02:19:20.9882743Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:48] f(&42) = 42
2019-08-26T02:19:20.9882787Z before
2019-08-26T02:19:20.9883468Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:53] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-08-26T02:19:20.9884034Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] ("Yeah",) = (
2019-08-26T02:19:20.9884090Z     "Yeah",
2019-08-26T02:19:20.9884123Z )
2019-08-26T02:19:20.9884373Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:64] 1 = 1
2019-08-26T02:19:20.9884605Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:64] 2 = 2
2019-08-26T02:19:20.9884833Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] 1u8 = 1
2019-08-26T02:19:20.9885073Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] 2u32 = 2
2019-08-26T02:19:20.9885306Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] "Yeah" = "Yeah"
2019-08-26T02:19:20.9885532Z ------------------------------------------
2019-08-26T02:19:20.9885570Z 
2019-08-26T02:19:20.9885592Z 
2019-08-26T02:19:20.9885613Z 
---
2019-08-26T02:19:20.9886715Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-08-26T02:19:20.9886763Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-08-26T02:19:20.9886789Z 
2019-08-26T02:19:20.9886809Z 
2019-08-26T02:19:20.9888002Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-08-26T02:19:20.9888281Z 
2019-08-26T02:19:20.9888306Z 
2019-08-26T02:19:20.9888359Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-08-26T02:19:20.9888406Z Build completed unsuccessfully in 1:00:54
2019-08-26T02:19:20.9888406Z Build completed unsuccessfully in 1:00:54
2019-08-26T02:19:20.9888490Z == clock drift check ==
2019-08-26T02:19:20.9898371Z   local time: Mon Aug 26 02:19:20 UTC 2019
2019-08-26T02:19:21.1387764Z   network time: Mon, 26 Aug 2019 02:19:21 GMT
2019-08-26T02:19:21.1387926Z == end clock drift check ==
2019-08-26T02:19:22.0396707Z ##[error]Bash exited with code '1'.
2019-08-26T02:19:22.0436057Z ##[section]Starting: Checkout
2019-08-26T02:19:22.0437718Z ==============================================================================
2019-08-26T02:19:22.0437777Z Task         : Get sources
2019-08-26T02:19:22.0437833Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@nathanwhit nathanwhit force-pushed the nathanwhit:check-run-results branch from 29cc305 to cdf3beb Aug 26, 2019

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member

commented Aug 26, 2019

I've filed #63924 to follow up on my comment about the ignores, but this looks good to go. Thanks so much for getting it done!

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 26, 2019

📌 Commit cdf3beb has been approved by Mark-Simulacrum

@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2019

⌛️ Testing commit cdf3beb with merge f970d3d...

bors added a commit that referenced this pull request Aug 28, 2019
Auto merge of #63825 - nathanwhit:check-run-results, r=Mark-Simulacrum
Allow checking of run-pass execution output in compiletest

Closes #63751
Adds a `check-run-results` flag to compiletest headers, which if enabled checks the output of the execution of a run-pass test's binary against expected output.
@bors

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2019

💔 Test failed - checks-azure

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Aug 28, 2019

The job armhf-gnu of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-28T11:08:19.1766441Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-28T11:08:19.1766503Z 
2019-08-28T11:08:19.1767303Z   git checkout -b <new-branch-name>
2019-08-28T11:08:19.1767394Z 
2019-08-28T11:08:19.1767803Z HEAD is now at f970d3d04 Auto merge of #63825 - nathanwhit:check-run-results, r=Mark-Simulacrum
2019-08-28T11:08:19.1990385Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-28T11:08:19.1994216Z ==============================================================================
2019-08-28T11:08:19.1994342Z Task         : Bash
2019-08-28T11:08:19.1994423Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-28T13:40:02.0775712Z 
2019-08-28T13:40:02.0775896Z failures:
2019-08-28T13:40:02.0823701Z 
2019-08-28T13:40:02.0824877Z ---- [ui] ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs stdout ----
2019-08-28T13:40:02.0825319Z normalized run.stdout:
2019-08-28T13:40:02.0829161Z uploaded "$TEST_BUILD_DIR/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-08-28T13:40:02.0829597Z 
2019-08-28T13:40:02.0829662Z 
2019-08-28T13:40:02.0829773Z The actual run.stdout differed from the expected run.stdout.
2019-08-28T13:40:02.0829773Z The actual run.stdout differed from the expected run.stdout.
2019-08-28T13:40:02.0830543Z Actual run.stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/dbg-macro-expected-behavior.run.stdout
2019-08-28T13:40:02.0830819Z error: 1 errors occured comparing run output.
2019-08-28T13:40:02.0830920Z status: exit code: 0
2019-08-28T13:40:02.0830920Z status: exit code: 0
2019-08-28T13:40:02.0831434Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a"
2019-08-28T13:40:02.0831885Z ------------------------------------------
2019-08-28T13:40:02.0831885Z ------------------------------------------
2019-08-28T13:40:02.0832317Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-08-28T13:40:02.0832709Z ------------------------------------------
2019-08-28T13:40:02.0832791Z stderr:
2019-08-28T13:40:02.0833081Z ------------------------------------------
2019-08-28T13:40:02.0833081Z ------------------------------------------
2019-08-28T13:40:02.0833429Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:23] Unit = Unit
2019-08-28T13:40:02.0833806Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:24] a = Unit
2019-08-28T13:40:02.0834195Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:30] Point{x: 42, y: 24,} = Point {
2019-08-28T13:40:02.0834317Z     x: 42,
2019-08-28T13:40:02.0834385Z     y: 24,
2019-08-28T13:40:02.0834478Z }
2019-08-28T13:40:02.0834817Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:31] b = Point {
2019-08-28T13:40:02.0834933Z     x: 42,
2019-08-28T13:40:02.0835001Z     y: 24,
2019-08-28T13:40:02.0835672Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:39]
2019-08-28T13:40:02.0835672Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:39]
2019-08-28T13:40:02.0836149Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:43] &a = NoCopy(
2019-08-28T13:40:02.0836329Z )
2019-08-28T13:40:02.0836329Z )
2019-08-28T13:40:02.0836681Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:43] dbg!(& a) = NoCopy(
2019-08-28T13:40:02.0836866Z )
2019-08-28T13:40:02.0836866Z )
2019-08-28T13:40:02.0837579Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:48] f(&42) = 42
2019-08-28T13:40:02.0837683Z before
2019-08-28T13:40:02.0838079Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:53] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-08-28T13:40:02.0838696Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] ("Yeah",) = (
2019-08-28T13:40:02.0838812Z     "Yeah",
2019-08-28T13:40:02.0838877Z )
2019-08-28T13:40:02.0839200Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:64] 1 = 1
2019-08-28T13:40:02.0839533Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:64] 2 = 2
2019-08-28T13:40:02.0839885Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] 1u8 = 1
2019-08-28T13:40:02.0840232Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] 2u32 = 2
2019-08-28T13:40:02.0840569Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:68] "Yeah" = "Yeah"
2019-08-28T13:40:02.0840910Z ------------------------------------------
2019-08-28T13:40:02.0840962Z 
2019-08-28T13:40:02.0840999Z 
2019-08-28T13:40:02.0841055Z 
---
2019-08-28T13:40:02.0891443Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-08-28T13:40:02.0891994Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-08-28T13:40:02.0918096Z 
2019-08-28T13:40:02.0918540Z 
2019-08-28T13:40:02.0921216Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-arm-unknown-linux-gnueabihf" "--mode" "ui" "--target" "arm-unknown-linux-gnueabihf" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "arm-linux-gnueabihf-gcc" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/arm-unknown-linux-gnueabihf/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--llvm-version" "9.0.0-rust-1.39.0-dev\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--remote-test-client" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-08-28T13:40:02.0922057Z 
2019-08-28T13:40:02.0922122Z 
2019-08-28T13:40:02.0932223Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target arm-unknown-linux-gnueabihf
2019-08-28T13:40:02.0932823Z Build completed unsuccessfully in 2:26:45
2019-08-28T13:40:02.0932823Z Build completed unsuccessfully in 2:26:45
2019-08-28T13:40:02.1000476Z == clock drift check ==
2019-08-28T13:40:02.1019075Z   local time: Wed Aug 28 13:40:02 UTC 2019
2019-08-28T13:40:02.1751209Z   network time: Wed, 28 Aug 2019 13:40:02 GMT
2019-08-28T13:40:02.1755768Z == end clock drift check ==
2019-08-28T13:40:02.9521818Z ##[error]Bash exited with code '1'.
2019-08-28T13:40:02.9564433Z ##[section]Starting: Upload CPU usage statistics
2019-08-28T13:40:02.9571751Z ==============================================================================
2019-08-28T13:40:02.9571835Z Task         : Bash
2019-08-28T13:40:02.9571913Z Description  : Run a Bash script on macOS, Linux, or Windows

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Sep 2, 2019

The job arm-android of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-09-02T00:24:47.0255789Z 
2019-09-02T00:24:47.0255879Z failures:
2019-09-02T00:24:47.0289739Z 
2019-09-02T00:24:47.0290473Z ---- [ui] ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs stdout ----
2019-09-02T00:24:47.0290591Z normalized run.stdout:
2019-09-02T00:24:47.0291077Z uploaded "$TEST_BUILD_DIR/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-09-02T00:24:47.0292371Z 
2019-09-02T00:24:47.0292405Z 
2019-09-02T00:24:47.0292503Z The actual run.stdout differed from the expected run.stdout.
2019-09-02T00:24:47.0292503Z The actual run.stdout differed from the expected run.stdout.
2019-09-02T00:24:47.0293042Z Actual run.stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/dbg-macro-expected-behavior.run.stdout
2019-09-02T00:24:47.0293230Z error: 1 errors occured comparing run output.
2019-09-02T00:24:47.0293387Z status: exit code: 0
2019-09-02T00:24:47.0293387Z status: exit code: 0
2019-09-02T00:24:47.0293868Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a"
2019-09-02T00:24:47.0294435Z ------------------------------------------
2019-09-02T00:24:47.0294435Z ------------------------------------------
2019-09-02T00:24:47.0294866Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-09-02T00:24:47.0295493Z ------------------------------------------
2019-09-02T00:24:47.0295556Z stderr:
2019-09-02T00:24:47.0295783Z ------------------------------------------
2019-09-02T00:24:47.0295783Z ------------------------------------------
2019-09-02T00:24:47.0296048Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:20] Unit = Unit
2019-09-02T00:24:47.0296343Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:21] a = Unit
2019-09-02T00:24:47.0296829Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:27] Point{x: 42, y: 24,} = Point {
2019-09-02T00:24:47.0296910Z     x: 42,
2019-09-02T00:24:47.0296981Z     y: 24,
2019-09-02T00:24:47.0297035Z }
2019-09-02T00:24:47.0297327Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:28] b = Point {
2019-09-02T00:24:47.0297401Z     x: 42,
2019-09-02T00:24:47.0297472Z     y: 24,
2019-09-02T00:24:47.0297795Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:36]
2019-09-02T00:24:47.0297795Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:36]
2019-09-02T00:24:47.0298086Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:40] &a = NoCopy(
2019-09-02T00:24:47.0298230Z )
2019-09-02T00:24:47.0298230Z )
2019-09-02T00:24:47.0298524Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:40] dbg!(& a) = NoCopy(
2019-09-02T00:24:47.0298671Z )
2019-09-02T00:24:47.0298671Z )
2019-09-02T00:24:47.0298936Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:45] f(&42) = 42
2019-09-02T00:24:47.0299027Z before
2019-09-02T00:24:47.0299333Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:50] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-09-02T00:24:47.0299657Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:58] ("Yeah",) = (
2019-09-02T00:24:47.0299731Z     "Yeah",
2019-09-02T00:24:47.0299800Z )
2019-09-02T00:24:47.0300061Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] 1 = 1
2019-09-02T00:24:47.0300355Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] 2 = 2
2019-09-02T00:24:47.0300641Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] 1u8 = 1
2019-09-02T00:24:47.0300940Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] 2u32 = 2
2019-09-02T00:24:47.0301855Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] "Yeah" = "Yeah"
2019-09-02T00:24:47.0302204Z ------------------------------------------
2019-09-02T00:24:47.0302271Z 
2019-09-02T00:24:47.0302305Z 
2019-09-02T00:24:47.0302338Z 
---
2019-09-02T00:24:47.0344865Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-09-02T00:24:47.0345162Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-09-02T00:24:47.0363412Z 
2019-09-02T00:24:47.0363552Z 
2019-09-02T00:24:47.0366255Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/arm-linux-androideabi/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-arm-linux-androideabi" "--mode" "ui" "--target" "arm-linux-androideabi" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "/android/ndk/arm-14/bin/arm-linux-androideabi-clang" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/arm-linux-androideabi/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--llvm-version" "9.0.0-rust-1.39.0-dev\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--remote-test-client" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "/android/ndk/arm-14" "--color" "always"
2019-09-02T00:24:47.0367393Z 
2019-09-02T00:24:47.0367530Z 
2019-09-02T00:24:47.0375124Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target arm-linux-androideabi
2019-09-02T00:24:47.0375958Z Build completed unsuccessfully in 2:09:12
2019-09-02T00:24:47.0375958Z Build completed unsuccessfully in 2:09:12
2019-09-02T00:24:47.0434074Z == clock drift check ==
2019-09-02T00:24:47.0455085Z   local time: Mon Sep  2 00:24:47 UTC 2019
2019-09-02T00:24:47.3176369Z   network time: Mon, 02 Sep 2019 00:24:47 GMT
2019-09-02T00:24:47.3182244Z == end clock drift check ==
2019-09-02T00:24:48.4924561Z ##[error]Bash exited with code '1'.
2019-09-02T00:24:48.4959129Z ##[section]Starting: Upload CPU usage statistics
2019-09-02T00:24:48.4963693Z ==============================================================================
2019-09-02T00:24:48.4963807Z Task         : Bash
2019-09-02T00:24:48.4963879Z Description  : Run a Bash script on macOS, Linux, or Windows

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2019

💔 Test failed - checks-azure

@RalfJung

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

Hm, this makes them "pass" on that platform though, instead of being marked as "ignored" as it probably should.
@Mark-Simulacrum should know if that is acceptable.

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

Yeah, I'd rather we didn't mark them as having passed.

@nathanwhit

This comment has been minimized.

Copy link
Contributor Author

commented Sep 2, 2019

Okay, I'll go ahead and make that change.
My thinking behind running instead of ignoring was that even though we can't check the output of execution for that target, we can still check the compiler output and pass/fail based on that, but I can see how that could be undesirable.

Ignore check-run-results tests for wasm32-bare
Ignores run-pass tests with the `check-run-results` flag enabled for the
wasm32-bare ("wasm32-unknown-unknown") target, as it does not support
printing to stdout/stderr.

@nathanwhit nathanwhit force-pushed the nathanwhit:check-run-results branch from e86d112 to bf44f12 Sep 2, 2019

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

Yeah, I think it's a bit ambiguous as to what the correct behavior there is but overall I'd prefer to ignore the test entirely I think. We can of course change that at a later point.

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2019

📌 Commit bf44f12 has been approved by Mark-Simulacrum

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 3, 2019

⌛️ Testing commit bf44f12 with merge c11b9bb...

bors added a commit that referenced this pull request Sep 3, 2019
Auto merge of #63825 - nathanwhit:check-run-results, r=Mark-Simulacrum
Allow checking of run-pass execution output in compiletest

Closes #63751
Adds a `check-run-results` flag to compiletest headers, which if enabled checks the output of the execution of a run-pass test's binary against expected output.
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented Sep 3, 2019

The job armhf-gnu of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-09-03T04:31:13.4366953Z 
2019-09-03T04:31:13.4367018Z failures:
2019-09-03T04:31:13.4403774Z 
2019-09-03T04:31:13.4404771Z ---- [ui] ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs stdout ----
2019-09-03T04:31:13.4405092Z normalized run.stdout:
2019-09-03T04:31:13.4405523Z uploaded "$TEST_BUILD_DIR/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-09-03T04:31:13.4405831Z 
2019-09-03T04:31:13.4406167Z 
2019-09-03T04:31:13.4406429Z The actual run.stdout differed from the expected run.stdout.
2019-09-03T04:31:13.4406429Z The actual run.stdout differed from the expected run.stdout.
2019-09-03T04:31:13.4406973Z Actual run.stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/dbg-macro-expected-behavior.run.stdout
2019-09-03T04:31:13.4407372Z error: 1 errors occured comparing run output.
2019-09-03T04:31:13.4407486Z status: exit code: 0
2019-09-03T04:31:13.4407486Z status: exit code: 0
2019-09-03T04:31:13.4408035Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a"
2019-09-03T04:31:13.4408639Z ------------------------------------------
2019-09-03T04:31:13.4408639Z ------------------------------------------
2019-09-03T04:31:13.4409191Z uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior/a", waiting for result
2019-09-03T04:31:13.4409773Z ------------------------------------------
2019-09-03T04:31:13.4410011Z stderr:
2019-09-03T04:31:13.4410668Z ------------------------------------------
2019-09-03T04:31:13.4410668Z ------------------------------------------
2019-09-03T04:31:13.4411283Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:20] Unit = Unit
2019-09-03T04:31:13.4411855Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:21] a = Unit
2019-09-03T04:31:13.4412408Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:27] Point{x: 42, y: 24,} = Point {
2019-09-03T04:31:13.4412672Z     x: 42,
2019-09-03T04:31:13.4412766Z     y: 24,
2019-09-03T04:31:13.4412862Z }
2019-09-03T04:31:13.4413225Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:28] b = Point {
2019-09-03T04:31:13.4413489Z     x: 42,
2019-09-03T04:31:13.4413597Z     y: 24,
2019-09-03T04:31:13.4414031Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:36]
2019-09-03T04:31:13.4414031Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:36]
2019-09-03T04:31:13.4414544Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:40] &a = NoCopy(
2019-09-03T04:31:13.4414925Z )
2019-09-03T04:31:13.4414925Z )
2019-09-03T04:31:13.4415297Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:40] dbg!(& a) = NoCopy(
2019-09-03T04:31:13.4415466Z )
2019-09-03T04:31:13.4415466Z )
2019-09-03T04:31:13.4415809Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:45] f(&42) = 42
2019-09-03T04:31:13.4415915Z before
2019-09-03T04:31:13.4416279Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:50] { foo += 1; eprintln!("before"); 7331 } = 7331
2019-09-03T04:31:13.4416901Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:58] ("Yeah",) = (
2019-09-03T04:31:13.4417174Z     "Yeah",
2019-09-03T04:31:13.4417269Z )
2019-09-03T04:31:13.4417692Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] 1 = 1
2019-09-03T04:31:13.4418022Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:61] 2 = 2
2019-09-03T04:31:13.4418384Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] 1u8 = 1
2019-09-03T04:31:13.4418714Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] 2u32 = 2
2019-09-03T04:31:13.4419071Z [/checkout/src/test/ui/rfc-2361-dbg-macro/dbg-macro-expected-behavior.rs:65] "Yeah" = "Yeah"
2019-09-03T04:31:13.4419660Z ------------------------------------------
2019-09-03T04:31:13.4419710Z 
2019-09-03T04:31:13.4419747Z 
2019-09-03T04:31:13.4419800Z 
---
2019-09-03T04:31:13.4460057Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-09-03T04:31:13.4460994Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-09-03T04:31:13.4483321Z 
2019-09-03T04:31:13.4483438Z 
2019-09-03T04:31:13.4485696Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-arm-unknown-linux-gnueabihf" "--mode" "ui" "--target" "arm-unknown-linux-gnueabihf" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "arm-linux-gnueabihf-gcc" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/arm-unknown-linux-gnueabihf/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--llvm-version" "9.0.0-rust-1.39.0-dev\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--remote-test-client" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-09-03T04:31:13.4486400Z 
2019-09-03T04:31:13.4486459Z 
2019-09-03T04:31:13.4494185Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target arm-unknown-linux-gnueabihf
2019-09-03T04:31:13.4494329Z Build completed unsuccessfully in 2:24:24
2019-09-03T04:31:13.4494329Z Build completed unsuccessfully in 2:24:24
2019-09-03T04:31:13.4553509Z == clock drift check ==
2019-09-03T04:31:13.4573275Z   local time: Tue Sep  3 04:31:13 UTC 2019
2019-09-03T04:31:13.5301592Z   network time: Tue, 03 Sep 2019 04:31:13 GMT
2019-09-03T04:31:13.5301737Z == end clock drift check ==
2019-09-03T04:31:14.2526513Z ##[error]Bash exited with code '1'.
2019-09-03T04:31:14.2565956Z ##[section]Starting: Upload CPU usage statistics
2019-09-03T04:31:14.2572640Z ==============================================================================
2019-09-03T04:31:14.2572767Z Task         : Bash
2019-09-03T04:31:14.2572845Z Description  : Run a Bash script on macOS, Linux, or Windows

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 3, 2019

💔 Test failed - checks-azure

Strip remote-test-client output from run stdout
The remote-test-client outputs a message of the form "uploaded
"<build_dir>/<executable_path>", waiting for result" onto stdout when
executing a test, which is then captured in the process result. This needs to be removed when
comparing the results of the run-pass test execution.
@nathanwhit

This comment has been minimized.

Copy link
Contributor Author

commented Sep 3, 2019

The most recent commit should take care of the failing test, and allow for the PR to merge.

@Mark-Simulacrum

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 3, 2019

📌 Commit 12adc39 has been approved by Mark-Simulacrum

@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2019

⌛️ Testing commit 12adc39 with merge 6c18a3d...

bors added a commit that referenced this pull request Sep 4, 2019
Auto merge of #63825 - nathanwhit:check-run-results, r=Mark-Simulacrum
Allow checking of run-pass execution output in compiletest

Closes #63751
Adds a `check-run-results` flag to compiletest headers, which if enabled checks the output of the execution of a run-pass test's binary against expected output.
@bors

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2019

☀️ Test successful - checks-azure
Approved by: Mark-Simulacrum
Pushing 6c18a3d to master...

@bors bors added the merged-by-bors label Sep 4, 2019

@bors bors merged commit 12adc39 into rust-lang:master Sep 4, 2019

5 checks passed

homu Test successful
Details
pr Build #20190903.36 succeeded
Details
pr (Linux mingw-check) Linux mingw-check succeeded
Details
pr (Linux x86_64-gnu-llvm-6.0) Linux x86_64-gnu-llvm-6.0 succeeded
Details
pr (LinuxTools) LinuxTools succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.