[WIP] split libtest types and formatting into a libtest-utils crate

[gnzlbg]

Do not merge: I just want to get initial feedback about the general direction. I'll close this afterwards and submit a more finished PR.

cc @djrenren @alexcrichton we'll better chat about this, but this is basically a "small" step towards allowing custom test frameworks to re-use some of the libtest facilities.

[rust-highfive]

r? @Kimundi

(rust_highfive has picked a reviewer for you, use r? to override)

[rust-highfive]

⚠️ Warning ⚠️

• These commits modify submodules.

[rust-highfive]

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0ce0eb40:start=1545511078985212849,finish=1545511133064664832,duration=54079451983
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---
#########################                                                 35.7%
######################################################################## 100.0%
[00:01:16] extracting /checkout/obj/build/cache/2018-12-09/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:01:16]     Updating crates.io index
[00:01:24] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:24] Build completed unsuccessfully in 0:00:19
[00:01:24] Makefile:81: recipe for target 'prepare' failed
[00:01:24] make: *** [prepare] Error 1
[00:01:25] Command failed. Attempt 2/5:
[00:01:25] Command failed. Attempt 2/5:
[00:01:25] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:25] Build completed unsuccessfully in 0:00:00
[00:01:25] make: *** [prepare] Error 1
[00:01:25] Makefile:81: recipe for target 'prepare' failed
[00:01:27] Command failed. Attempt 3/5:
[00:01:27] Command failed. Attempt 3/5:
[00:01:28] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:28] Build completed unsuccessfully in 0:00:00
[00:01:28] make: *** [prepare] Error 1
[00:01:28] Makefile:81: recipe for target 'prepare' failed
[00:01:31] Command failed. Attempt 4/5:
[00:01:31] Command failed. Attempt 4/5:
[00:01:31] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:31] Build completed unsuccessfully in 0:00:00
[00:01:31] make: *** [prepare] Error 1
[00:01:31] Makefile:81: recipe for target 'prepare' failed
[00:01:35] Command failed. Attempt 5/5:
[00:01:35] Command failed. Attempt 5/5:
[00:01:35] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:35] Build completed unsuccessfully in 0:00:00
[00:01:35] make: *** [prepare] Error 1
[00:01:35] Makefile:81: recipe for target 'prepare' failed
[00:01:35] The command has failed after 5 attempts.
---
travis_time:end:063952c8:start=1545511237902351042,finish=1545511237907228779,duration=4877737
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:1d97b95a
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:290f2740
travis_time:start:290f2740
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:223099d2
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[djrenren]

Hmmm the *Desc structs still seem to be tied to the libtest types. Currently a Desc contains a functional test but it seems like this effort would benefit from restructuring the Desc to be orthogonal to the underlying test. Otherwise it'll be hard to conveniently reuse the formatters with other test formats.

[gnzlbg]

@djrenren

Hmmm the *Desc structs still seem to be tied to the libtest types. Currently a Desc contains a functional test

So not really. Like you can see here, the Desc struct only contains the test name, whether it is ignored, whether it should panic, or is it allowed to fail.

FWIW I think we might want a better API for this than a struct, e.g., some kind of trait, but I think this is not too bad of a start, and if we put this on a crate on crates.io, we can iterate on this using normal crate.io releases, instead of tying people to particular versions of rustc.

[djrenren]

Ah missed that in my skim. Sorry! Will give it a more thorough look after the holidays.

…

On Dec 22, 2018, at 17:12, gnzlbg ***@***.***> wrote: @gnzlbg commented on this pull request. In src/libtest/libtest-utils/src/test.rs: > @@ -0,0 +1,85 @@ +//! Tests utilities + +use crate::{format::Padding, ShouldPanic}; +use std::{borrow::Cow, fmt}; + +/// Test description. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub struct Desc { @djrenren Hmmm the *Desc structs still seem to be tied to the libtest types. Currently a Desc contains a functional test So not really. Like you can see here, the Desc struct only contains the test name, whether it is ignored, whether it should panic, or is it allowed to fail. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[bors]

☔️ The latest upstream changes (presumably #57079) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

Thanks for this @gnzlbg! I'm not entirely certain what to do about this though. I feel like this is a good direction to go in, and libtest is one of those pieces that seems like it would hugely benefit from living outside this repository and being developed externally. So, for example, let me paint a picture of where we might take this PR to the limit.

• Let's assume we create a repository like rust-lang/libtest
• This repository would all be 100% stable Rust and publishable on crates.io, organized however it would like
• The rust-lang/rust repository would continue to have the "one and only" libtest, and libtest would just depend on these crates.io crates published from rust-lang/libtest
• The libtest in rust-lang/rust would be a very thin shim which defines the actual interface that rustc expects. This way all the unstable code would live in this repository, and then the rust-lang/rust libtest would delegate to rust-lang/libtest when it's called.

That way, long term, we'd have a dedicated project to libtest and all the rust-lang/rust repository would contain is thin shims that translate between the stable crates.io interface and the unstable rustc interface.

How's that sound?

[gnzlbg]

That's pretty much what I had in mind.

[alexcrichton]

@gnzlbg and I discussed a bit on IRC and the conclusion was that @gnzlbg would open up an internal discussion to see if others have thoughts on this, and assuming others agree I think we can just delete most of the crate in this repository, move it externally, publish it, reintegrate here, and we should be good to go!

[gnzlbg]

I've opened the discussion here: https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139

I think we can just delete most of the crate in this repository, move it externally, publish it, reintegrate here, and we should be good to go!

I'm closing this because I prefer to open a PR to do just that first.