New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: Add option to persist doc test executables #56189

Open
wants to merge 4 commits into
base: master
from

Conversation

@rep-nop
Contributor

rep-nop commented Nov 23, 2018

Fixes #37048.

This is the initial version of the code so the doctest executables can be used for stuff like code coverage (specifically xd009642/tarpaulin#13) the folders it goes into were just a first idea, so any better ones are welcome.

Right now it creates a directory structure like:

  given_path/
          |_____ <filename>_rs_<linenum>/
          |_____ ...
          |_____ <filename>_rs_<linenum>/
                 |_____ rust_out

I couldn't figure out where it actually outputs the file w/ the name, I suspect its somewhere deeper in the compiler.

It also adds the unstable --persist-doctests flag to rustdoc that enables this behavior.

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 23, 2018

r? @steveklabnik

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 23, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:03fed96f:start=1543014799290895690,finish=1543014858642986612,duration=59352090922
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---

[00:03:16] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:270: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:277: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:308: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:388: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:621: line longer than 100 chars
[00:03:18] some tidy checks failed
[00:03:18] 
[00:03:18] 
[00:03:18] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:18] 
[00:03:18] 
[00:03:18] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:18] Build completed unsuccessfully in 0:00:59
[00:03:18] Build completed unsuccessfully in 0:00:59
[00:03:18] Makefile:79: recipe for target 'tidy' failed
[00:03:18] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0bc38e20
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Fri Nov 23 23:17:46 UTC 2018
---
travis_time:end:18fd18bc:start=1543015066518747159,finish=1543015066524464485,duration=5717326
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:21fe0823
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2e55ff67
travis_time:start:2e55ff67
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:22778229
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 24, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:021b8011:start=1543017798489266461,finish=1543017859998675257,duration=61509408796
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---

[00:03:15] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:271: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:278: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:309: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:389: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:622: line longer than 100 chars
[00:03:17] some tidy checks failed
[00:03:17] 
[00:03:17] 
[00:03:17] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:17] 
[00:03:17] 
[00:03:17] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:17] Build completed unsuccessfully in 0:00:57
[00:03:17] Build completed unsuccessfully in 0:00:57
[00:03:17] Makefile:79: recipe for target 'tidy' failed
[00:03:17] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0097cf20
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Nov 24 00:07:46 UTC 2018
---
travis_time:end:03998c14:start=1543018067118755215,finish=1543018067124316553,duration=5561338
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0dbec2e8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:02c2b270
travis_time:start:02c2b270
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:03fa6259
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Nov 24, 2018

My local tidy isn't working correctly, it keeps spitting out stuff in jemalloc, any idea why that would be? 🤔

Edit: figured it out I think, was just using it wrong lol

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Nov 24, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:01f5bc37:start=1543018741490622165,finish=1543018798753373196,duration=57262751031
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:50:20] 
[00:50:20] running 5050 tests
[00:50:23] .................................................................................................... 100/5050
[00:50:26] .................................................................................................... 200/5050
[00:50:29] .............................ii............................................ii...................ii.. 300/5050
[00:50:32] ...............................................................................................iii.. 400/5050
[00:50:35] .....iiiiiiii.iii............................iii...........................................i........ 500/5050
[00:50:42] .................................................................................................... 700/5050
[00:50:48] ...................................................................................i...........i.... 800/5050
[00:50:52] .................................................................................................... 900/5050
[00:50:55] ..iiiii..................ii.iiii.................................................................... 1000/5050
---
[00:51:32] .................................................................................................... 2200/5050
[00:51:36] .................................................................................................... 2300/5050
[00:51:40] .................................................................................................... 2400/5050
[00:51:44] .................................................................................................... 2500/5050
[00:51:47] ...........................................................................................iiiiiiiii 2600/5050
[00:51:54] .........................................................ii......................................... 2800/5050
[00:51:57] .................................................................................................... 2900/5050
[00:52:01] .................................................................................................... 3000/5050
[00:52:05] ......................................................i............................................. 3100/5050
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:06:16] 
[01:06:16] running 117 tests
[01:06:19] i..ii...iii..iiii.....i...i.........i..iii...........i.....i.....ii...i..i.ii..............i...ii..i 100/117
[01:06:20] i.i.....iiii.....
[01:06:20] 
[01:06:20]  finished in 3.731
[01:06:20] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:06:35] 
[01:06:35] running 118 tests
[01:06:59] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[01:07:03] ......iii.i.....ii
[01:07:03] 
[01:07:03]  finished in 27.737
[01:07:03] travis_fold:end:test_debuginfo

---
[01:42:10] 
[01:42:10] 12 3 | no
[01:42:10] 13   | ^^ not found in this scope
[01:42:10] 14 
[01:42:10] - thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:323:13
[01:42:10] + thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:368:13
[01:42:10] 17 
[01:42:10] 17 
[01:42:10] 18 ---- $DIR/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:42:10] 
[01:42:10] 21 thread 'main' panicked at 'oh no', $DIR/failed-doctest-output.rs:3:1
[01:42:10] 23 
[01:42:10] - ', src/librustdoc/test.rs:358:17
[01:42:10] + ', src/librustdoc/test.rs:414:17
[01:42:10] 25 
[01:42:10] 25 
[01:42:10] 26 
[01:42:10] 27 failures:
[01:42:10] 
[01:42:10] 
[01:42:10] The actual stdout differed from the expected stdout.
[01:42:10] Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/failed-doctest-output.stdout
[01:42:10] To update references, rerun the tests and pass the `--bless` flag
[01:42:10] To only update this specific test, also pass `--test-args failed-doctest-output.rs`
[01:42:10] error: 1 errors occurred comparing output.
[01:42:10] status: exit code: 101
[01:42:10] status: exit code: 101
[01:42:10] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/src/test/rustdoc-ui/failed-doctest-output.rs" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/a" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/auxiliary"
[01:42:10] ------------------------------------------
[01:42:10] 
[01:42:10] running 2 tests
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) ... FAILED
[01:42:10] failures:
[01:42:10] 
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:42:10] error[E0425]: cannot find value `no` in this scope
[01:42:10]   |
[01:42:10] 3 | no
[01:42:10]   | ^^ not found in this scope
[01:42:10] 
[01:42:10] 
[01:42:10] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:368:13
[01:42:10] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:42:10] 
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:42:10] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)' panicked at 'test executable failed:
[01:42:10] 
[01:42:10] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:42:10] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:42:10] 
[01:42:10] ', src/librustdoc/test.rs:414:17
[01:42:10] 
[01:42:10] 
[01:42:10] 
[01:42:10] failures:
[01:42:10]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)
[01:42:10]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)
[01:42:10] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:42:10] 
[01:42:10] 
[01:42:10] ------------------------------------------
---
[01:42:10] test result: FAILED. 9 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:42:10] 
[01:42:10] 
[01:42:10] 
[01:42:10] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/rustdoc-ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-5.0/bin/FileCheck" "--host-rustcflags" "-Zunstable-options " "--target-rustcflags" "-Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "5.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:42:10] 
[01:42:10] 
[01:42:10] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:42:10] Build completed unsuccessfully in 0:55:50
[01:42:10] Build completed unsuccessfully in 0:55:50
[01:42:10] make: *** [check] Error 1
[01:42:10] Makefile:58: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:007541b8
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Nov 24 02:02:18 UTC 2018
---
travis_time:end:14eb2f58:start=1543024941449845534,finish=1543024941454854194,duration=5008660
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:12cfc3af
$ ln -s . checkout && for CORE in obj/cores/core.*; 

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Nov 26, 2018

Hm, I didn't noticed that somehow the jemalloc submodule got pulled in. Should I remove that?

@Dylan-DPC

This comment has been minimized.

Member

Dylan-DPC commented Dec 3, 2018

Pinging from triage, looking for a reviewer from @rust-lang/docs (sorry for the ping) @rust-lang/rustdoc

@GuillaumeGomez

This comment has been minimized.

Member

GuillaumeGomez commented Dec 3, 2018

This looks nice and I think it's good to go. However, since it touches rustdoc options and @QuietMisdreavus reworked it recently, I'll let them review this PR.

@Dylan-DPC

This comment has been minimized.

Member

Dylan-DPC commented Dec 3, 2018

let mut target_path: PathBuf = env::var("OUT_DIR")
.expect("Rustdoc must be run with Cargo to persist doctest executables.").into();
while !target_path.ends_with("target") {

This comment has been minimized.

@ehuss

ehuss Dec 3, 2018

Contributor

This doesn't seem safe. The target directory is not necessarily named "target". Also, cargo has a bit of an expectation of managing the directory layout. Normally tools use something like -o to control their output.

}
let target_path = if persist_executable {
let mut target_path: PathBuf = env::var("OUT_DIR")

This comment has been minimized.

@ehuss

ehuss Dec 3, 2018

Contributor

OUT_DIR is only set for build scripts. How is this supposed to be used?

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

Is it? I don't believe the project I tested it on had one, but I could be remembering wrong. Regardless, I'll be taking @QuietMisdreavus's advice and taking a path in the cmd args :)

@QuietMisdreavus

Thanks so much for this! Improvements like this are something i've wanted to work on for doctests, or at least see happen, so it's exciting to see! I've left a few suggestions, to make the feature fit better with the way rustdoc works in general, and one or two personal style preferences.

let target_path = if persist_executable {
let mut target_path: PathBuf = env::var("OUT_DIR")
.expect("Rustdoc must be run with Cargo to persist doctest executables.").into();

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 3, 2018

Member

I feel like Rustdoc shouldn't assume the presence of Cargo. If it's being run bare, we can probably get away with either just creating/using a rustdoctest subdirectory of the current working directory, or (my preferred choice) taking the directory as a CLI flag.

In fact, i think it would be better if the CLI flag that unlocked this functionality just took the directory itself, rather than trying to infer it like this. This way, tooling that wants to use this functionality can set the directory itself instead of working backwards from the target directory. Plus, if Cargo adds a flag to unlock this later on without forcing users to use RUSTDOCFLAGS, it can pass target/rustdoctest by itself.

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

Huh, I never considered taking in the path, sounds like a much better idea! 👌

@@ -346,6 +391,17 @@ fn run_test(test: &str, cratename: &str, filename: &FileName, line: usize,
};
cmd.env(var, &newpath);
if persist_executable {
let mut target_path = target_path.unwrap();

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 3, 2018

Member

This could probably be better written as if let Some(target_path) = target_path, to avoid the unwrap.

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

The reason I didn't was because it needs cloned, but that's solvable with if let too and I like that better also.

std::fs::write(
target_path,
newpath.as_os_str().to_str().unwrap().as_bytes(),

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 3, 2018

Member

This seems a little awkward, because it's not always the same environment variable that's being set here. A better option would be to make this an environment file, and make the file output be the string "{var}={newpath}", to at least show that this should be setting something other than PATH (on non-Windows platforms, at least). (With these file contents, i'd probably also name it something like test.env, to make it more obvious what it's there for.)

One situation where this breaks down is if the doctests are being built for a different target than the host. It looks like the chosen environment variable will always be stuck to what the host expects, but if you're expecting to run the doctests on another target (say, some embedded platform), then that may be different. However, in this situation it's also possible that the dylib path extension may not be necessary. Still, i think the above suggestion should be good enough.

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

Ah, okay, yeah fair enough :)

This comment has been minimized.

@ollie27

ollie27 Dec 3, 2018

Contributor

I'm pretty sure we don't need any of this newpath stuff now that we're not compiling doctests with prefer-dynamic since #54939. I don't think we need a path.txt or test.env file at all.

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

That would be fantastic! However I wasn't able to execute them without the paths generated in that file, so I'm not sure what the issue would be there.

This comment has been minimized.

@ollie27

ollie27 Dec 4, 2018

Contributor

I'm guessing you may have been using a version of rustdoc that didn't have #54939 applied yet. It'll be a good idea to rebase this PR and try again. Anyway I've submitted #56514 to see if anything breaks.

@@ -338,6 +338,11 @@ fn opts() -> Vec<RustcOptGroup> {
"enable-index-page",
"To enable generation of the index page")
}),
unstable("persist-executable", |o| {

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 3, 2018

Member

Bikeshed alert: I feel like --persist-doctests is a better name, since the executable here feels a little ambiguous if there's a binary crate somewhere in the user's mind that could get confused with this.

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

Haha, no idea how I missed that. Definitely agree with you there.

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Dec 3, 2018

Thanks for the comments @ehuss and @QuietMisdreavus! I'll get the changes in soon™️ (maybe even tonight or tomorrow night!)

std::fs::write(
target_path,
newpath.as_os_str().to_str().unwrap().as_bytes(),

This comment has been minimized.

@frewsxcv

frewsxcv Dec 3, 2018

Member

Path has a to_str method, you souldn't need to go through OsStr

https://doc.rust-lang.org/nightly/std/path/struct.Path.html#method.to_str

This comment has been minimized.

@rep-nop

rep-nop Dec 3, 2018

Contributor

newpath is an OsString

This comment has been minimized.

@frewsxcv

frewsxcv Dec 3, 2018

Member

Ah, well OsString coerces to an OsStr, so you should be able to call to_str() directly on the OsString, but not a big deal

@bors

This comment has been minimized.

Contributor

bors commented Dec 6, 2018

☔️ The latest upstream changes (presumably #54517) made this pull request unmergeable. Please resolve the merge conflicts.

@rep-nop rep-nop closed this Dec 8, 2018

@rep-nop rep-nop force-pushed the rep-nop:keep_doc_test_executable branch from ab904f1 to 0c999ed Dec 8, 2018

@rep-nop rep-nop reopened this Dec 8, 2018

@rep-nop rep-nop force-pushed the rep-nop:keep_doc_test_executable branch from 09434f5 to f6a5fd6 Dec 8, 2018

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Dec 8, 2018

Does anyone have any other suggestions for where to put the doctest executables themselves? The filename_rs_<line> seems to work but feels somewhat hacky.

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Dec 8, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:2ae0cfc0:start=1544297028912724404,finish=1544297031029122736,duration=2116398332
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:53:24] 
[00:53:24] running 120 tests
[00:53:27] i..ii...iii..iiii.....i...i..........i..iii.............i.....i.....ii...i..i.ii..............i...ii 100/120
[00:53:27] ..ii.i.....iiii.....
[00:53:27] 
[00:53:27]  finished in 3.364
[00:53:27] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:53:42] 
[00:53:42] running 118 tests
[00:54:05] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[00:54:08] ......iii.i.....ii
[00:54:08] 
[00:54:08]  finished in 27.050
[00:54:08] travis_fold:end:test_debuginfo

---
[01:23:48] 
[01:23:48] 12 3 | no
[01:23:48] 13   | ^^ not found in this scope
[01:23:48] 14 
[01:23:48] - thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:326:13
[01:23:48] + thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:360:13
[01:23:48] 17 
[01:23:48] 17 
[01:23:48] 18 ---- $DIR/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:23:48] 
[01:23:48] 21 thread 'main' panicked at 'oh no', $DIR/failed-doctest-output.rs:3:1
[01:23:48] 23 
[01:23:48] - ', src/librustdoc/test.rs:361:17
[01:23:48] + ', src/librustdoc/test.rs:395:17
[01:23:48] 25 
[01:23:48] 25 
[01:23:48] 26 
[01:23:48] 27 failures:
[01:23:48] 
[01:23:48] 
[01:23:48] The actual stdout differed from the expected stdout.
[01:23:48] Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/failed-doctest-output.stdout
[01:23:48] To update references, rerun the tests and pass the `--bless` flag
[01:23:48] To only update this specific test, also pass `--test-args failed-doctest-output.rs`
[01:23:48] error: 1 errors occurred comparing output.
[01:23:48] error: 1 errors occurred comparing output.
[01:23:48] statusoc-ui/failed-doctest-output.rs - SomeStruct (line 21)' panicked at 'test executable failed:
[01:23:48] 
[01:23:48] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:23:48] 
[01:23:48] ', src/librustdoc/test.rs:395:17
[01:23:48] 
[01:23:48] 
[01:23:48] 
[01:23:48] failures:
[01:23:48]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)
[01:23:48]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)
[01:23:48] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:23:48] 
[01:23:48] 
[01:23:48] ------------------------------------------
---
[01:23:48] 
[01:23:48] 
[01:23:48] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-liafter_failure.4
travis_time:start:0decf013
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:046775b8
travis_time:start:046775b8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:055a487a
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Dec 8, 2018

@QuietMisdreavus unless you or anyone else has final thoughts on where its outputting the test files, looks like we're good to go whenever 👍

@QuietMisdreavus

This comment has been minimized.

Member

QuietMisdreavus commented Dec 10, 2018

@QuietMisdreavus unless you or anyone else has final thoughts on where its outputting the test files, looks like we're good to go whenever 👍

One option is to use something like the generate_name function that we use to name the test itself for libtest. This also tracks things like headings in markdown to make the test easier to locate:

rust/src/librustdoc/test.rs

Lines 634 to 636 in e3b7222

fn generate_name(&self, line: usize, filename: &FileName) -> String {
format!("{} - {} (line {})", filename, self.names.join("::"), line)
}

You could make another function like this (to make something that was easier to use as a directory name) and hand that to run_test to use as the folder name.

@QuietMisdreavus

Just a couple comments, but otherwise this looks good to me!

unstable("persist-doctests", |o| {
o.optopt("",
"persist-doctests",
"Persists the rustdoc test executables",

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 10, 2018

Member
Suggested change Beta
"Persists the rustdoc test executables",
"Directory to persist doctest executables into",

Makes it obvious what the path is for.

.rsplit('/')
.next()
.unwrap()
.replace(".", "_"), line)

This comment has been minimized.

@QuietMisdreavus

QuietMisdreavus Dec 10, 2018

Member

The , line hanging on the end of this call looks awkward. Can you break it onto its own line?

@bors

This comment has been minimized.

Contributor

bors commented Dec 14, 2018

☔️ The latest upstream changes (presumably #56818) made this pull request unmergeable. Please resolve the merge conflicts.

@cramertj

This comment has been minimized.

Member

cramertj commented Dec 14, 2018

Does anyone have any other suggestions for where to put the doctest executables themselves? The filename_rs_ seems to work but feels somewhat hacky.

Would it be possible to combine the doctests into a single binary? That'd make it much easier (slash possible) for external tools and build systems to consume the resulting files, since they could be pulled in via a single pre-determined name.

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Dec 15, 2018

@cramertj that was my original hope, but I'm not experienced enough with the code base to actually implement that behavior without some additional guidance. After speaking with @QuietMisdreavus it seems that change would probably be quite large in scope -- certainly warranting its own PR I'd imagine. The question then I suppose is that do we consider that possible change in behavior down the line breaking? I would think not myself, but would like to hear input from others about that as well.

@QuietMisdreavus

This comment has been minimized.

Member

QuietMisdreavus commented Dec 15, 2018

@cramertj Sadly, i think that's out of scope for this PR. The current model of rustdoc is to compile everything separately as its own standalone binary, so mirroring the compiler's model for unit tests (where the unit test functions are rolled together and there's a libtest entry point that calls them all) would need a bit of an overhaul to make sure that everything can still work.

@rep-nop

This comment has been minimized.

Contributor

rep-nop commented Dec 16, 2018

whelp, something went horribly wrong there.

@rep-nop rep-nop force-pushed the rep-nop:keep_doc_test_executable branch from 2469c0b to 769c757 Dec 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment