Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Activating features for tests/benchmarks #2911

Open
ctz opened this issue Jul 24, 2016 · 32 comments
Open

Activating features for tests/benchmarks #2911

ctz opened this issue Jul 24, 2016 · 32 comments
Labels
A-features

Comments

@ctz
Copy link

@ctz ctz commented Jul 24, 2016

Given a library crate, I want to have optional features which are available to integration-level tests and examples so that 'cargo test' tests them.

Here's what I have:

[package]
name = "rustls"
version = "0.1.0"
(etc)

[features]
default = []
clientauth = []

[dependencies]
untrusted = "0.2.0"
(etc)

[dev-dependencies]
env_logger = "0.3.3"
(etc)

Here's what I've tried so far. Attempt one: add the crate into dev-dependencies with the feature, ie:

[dev-dependencies.rustls]
path = "."
version = "0.1.0"
features = ["clientauth"]

I kind of expected that to work, but it panics:

jbp@debian:~/rustls$ RUST_BACKTRACE=1 cargo test
thread '<main>' panicked at 'assertion failed: my_dependencies.insert(dep.clone())', src/cargo/util/dependency_queue.rs:86
stack backtrace:
   1:     0x558d17c3a2a0 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
   2:     0x558d17c4383b - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
   3:     0x558d17c434c3 - std::panicking::default_hook::hc2c969e7453d080c
   4:     0x558d17c28048 - std::panicking::rust_panic_with_hook::hfe203e3083c2b544
   5:     0x558d17701fef - std::panicking::begin_panic::h4ebf9fe884b2415f
   6:     0x558d17802d27 - cargo::ops::cargo_rustc::job_queue::JobQueue::enqueue::h98b20a33b8744ebe
   7:     0x558d177f4faf - cargo::ops::cargo_rustc::compile::h93336dd21aba29d0
   8:     0x558d1777a513 - cargo::ops::cargo_rustc::compile_targets::h7223dffae01d6b9d
   9:     0x558d1776eec9 - cargo::ops::cargo_compile::compile_pkg::h48656ffa9b1541f3
  10:     0x558d1776bd09 - cargo::ops::cargo_compile::compile::h1b43b20047c53d10
  11:     0x558d1785fb0d - cargo::ops::cargo_test::compile_tests::h028a22d3131e0d65
  12:     0x558d1785f379 - cargo::ops::cargo_test::run_tests::h0ad98c54a6f40c9d
  13:     0x558d176de256 - cargo::call_main_without_stdin::hd484f08b17c419d1
  14:     0x558d1768afe7 - cargo::execute::hab7accd6bf9b5c64
  15:     0x558d17683f8b - cargo::call_main_without_stdin::hf099bd36acb849a3
  16:     0x558d17680da1 - cargo::main::h2b219a79f378c1ef
  17:     0x558d17c430d8 - std::panicking::try::call::hc5e1f5b484ec7f0e
  18:     0x558d17c4e0eb - __rust_try
  19:     0x558d17c4e08e - __rust_maybe_catch_panic
  20:     0x558d17c42b03 - std::rt::lang_start::h61f4934e780b4dfc
  21:     0x7f79ea9bf86f - __libc_start_main
  22:     0x558d17680728 - <unknown>

(version: "cargo 0.11.0-nightly (259324c 2016-05-20)")

Attempt two: see if features.* works like profile.*:

[features.test]
default = ["clientauth"]

alas

$ cargo test
error: failed to parse manifest at `/home/jbp/rustls/Cargo.toml`

Caused by:
  expected a value of type `array`, but found a value of type `table` for the key `features.test`

Is this possible?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 25, 2016

Unfortunately, no, it's not possible for cargo test to automatically enable features currently. The panic though definitely seems bad!

@alexcrichton alexcrichton added the A-errors label Jul 25, 2016
@jjpe
Copy link

@jjpe jjpe commented Dec 18, 2016

I just ran into this issue because I'm currently using a feature to decide whether or not to do some rather verbose tracing. I'd like to be able to set this during testing so that when tests fail I can look at the trace.
The reason I made it a feature is that the code itself is in a library, but at the same time it seems more appropriate to have the consuming crate decide whether or not to perform the trace.

To be clear, an actual consuming crate has no problem setting the feature and getting corresponding results. It is just for testing the library crate itself that it doesn't seem possible at the moment.

Update: There is a workaround I just found:
If the feature is called foo, then this will test with the feature activated:

cargo test --features "foo"

The only thing is that this is activated manually, and being able to specify it in the library's Cargo.toml would activate the feature by default, which is desirable in my use case.

@sanmai-NL
Copy link

@sanmai-NL sanmai-NL commented May 4, 2017

@alexcrichton:

As of yet, the Cargo panic is gone.

  cargo --version
cargo 0.19.0-nightly (fa7584c14 2017-04-26)

Excerpt from Cargo.toml

[features]
default = [] # "clippy"

[profile.test]
default = ["clippy"]

The output is now as follows:

cargo test --color=always --package my_app --bin my_app test_1 -- --nocapture
warning: unused manifest key: profile.test.default
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/structure_web_server-3dc5f86e9b4fb259

running 1 test
test tests::test_1 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

You wrote this isn't possible. It appears there are no immediate objections to adding such a feature. Could you explain how fundamental the limitation is? And, if possible, sketch a solution? Just so a PR becomes more likely to happen.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented May 4, 2017

Ah I believe this was fixed in #3921 as #3902 was a dupe of this issue. I guess I was wrong before!

@sanmai-NL
Copy link

@sanmai-NL sanmai-NL commented May 4, 2017

@alexcrichton: the crashes have been resolved yes, but the original feature request still stands AFAIK. Please reopen.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented May 4, 2017

Oh sorry, missed that aspect.

@sunjay
Copy link
Member

@sunjay sunjay commented Oct 27, 2017

Is there any way to add features per profile now? That is, if I want to enable certain features during tests, can I do that yet?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Oct 28, 2017

@sunjay unfortunately no, you'll still need to pass --features on the command line

@sunjay
Copy link
Member

@sunjay sunjay commented Oct 28, 2017

Is this change something that would require an RFC or could it just be implemented? It sounds like being able to specify features in the profile configuration is something that could benefit a lot of people.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Oct 30, 2017

@sunjay I think due to the subtle possible ramifications of auto-enabling features I'd personally want to see an RFC, but it may be worth canvassing others from @rust-lang/cargo first as well

@sunjay
Copy link
Member

@sunjay sunjay commented Oct 30, 2017

I would love to work on an RFC for this. Very interested to hear what the cargo team thinks.

Right now, I'm having to constantly remember to pass features in when I run cargo test. I would really prefer to not have to do that. It's an extra point of friction for anyone who uses my code.

Though it would be great to even just be able to configure features in tests, I would really like to see cargo support a features/default features option in each of the [profile.*] sections. This would make it so you could configure default features for your release, debug and benchmarking profiles too. My RFC would be proposing something like that.

@matklad
Copy link
Member

@matklad matklad commented Oct 31, 2017

I think we have an RFC by @withoutboats for this already: rust-lang/rfcs#1956, which is postponed because of profiles.

@sunjay
Copy link
Member

@sunjay sunjay commented Oct 31, 2017

@matklad Ah yup that's exactly what I would have proposed. Thanks for looking that up! I read the discussion and it seems like that's pending on some work with how multiple profiles are applied together (e.g. test --release). It seems that the current behavior is more complicated than what it looks like at first glance.

Is there any way to do this in a backwards compatible way? It would be great if there was some way to just add another configuration option that is in effect the same as the command line argument while keeping everything the same as it is now. Would that make it too hard to change to something better (like profiles) later?

The main issue I'd like to solve is that right now, introducing features into your codebase makes it so that you have to pass features in as command line arguments every time you run a certain commands. If these aren't default features that you need for all builds, it's not possible to simply configure the ones you want in Cargo.toml. This makes it harder for people new to your codebase or to cargo set things up. It is very repetitive and easy to forget. Is there an easier way to just address that issue?

@brson
Copy link
Contributor

@brson brson commented May 21, 2018

I've also run into the need to have features automatically enabled in test builds. In my case I want the python bindings "extension-module" feature on for production builds, but off for test builds so that I can link directly to python during testing.

@Object905
Copy link

@Object905 Object905 commented May 23, 2018

Also it's very useful for #![no_std] ergonomics. Since you need to test with libstd linked, otherwise you get
error: no #[default_lib_allocator] found but one is required; is libstd not linked?. So you constantly running cargo test --features std

@ebfull
Copy link

@ebfull ebfull commented Jul 2, 2018

Note that you can do this with required-features if you specify your test targets explicitly, aka:

[[test]]
name = "whatever"
path = "tests/whatever.rs"
required-features = ["mycoolfeature"]

@RReverser
Copy link
Contributor

@RReverser RReverser commented Jul 19, 2018

@ebfull That only skips building/running tests if feature is not enabled instead of force-enabling the feature during build.

@synek317
Copy link

@synek317 synek317 commented Aug 25, 2018

Some ugly workaround, at least for some cases, is to extend all your #[cfg(feature = ...)] with test (e.g. #[cfg(any(test, feature = "..."))] and copy all optional dependencies in [dev-dependencies] and remove optional = true.

It works nice for me but for sure there are cases where it is not enough.

@nixpulvis
Copy link

@nixpulvis nixpulvis commented Oct 14, 2018

Yea I feel like an RFC here would make sense if one doesn't already exist. I'm using rust again, and finding myself really wanting something like cargo test --feature-matrix for example.

@ehuss ehuss added A-features and removed A-errors labels Nov 18, 2018
@Owez
Copy link

@Owez Owez commented Apr 10, 2021

There is currently a work-around for this. (Source)

[dev-dependencies]
self-package-name= { path = ".", features = ["desired_feature"] }

where self-package-name is the name of the package in the Cargo.toml file.

Sample repo for the same

Unfortunately this doesn't seem to work on an edge case in a no_std environment where your crate allows no_std and has a feature which enables std. Here's an example of where the issue can occur:

error[E0152]: found duplicate lang item `panic_impl`
  --> package/src/main.rs:15:1
   |
15 | fn panic(_info: &PanicInfo) -> ! {
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: the lang item is first defined in crate `std` (which `sprawl_format` depends on)
   = note: first definition in `std` loaded from /home/owen/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-3d933644112ba794.so, /home/owen/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-3d933644112ba794.rlib
   = note: second definition in the local crate (`sprawl_package`)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0152`.
error: could not compile `sprawl-package`

To learn more, run the command again with --verbose.

I assume this is because its importing std for the developer dependencies section, and this function is trying to overwrite a now-included stdlib function.

@Owez
Copy link

@Owez Owez commented Apr 10, 2021

The workaround for this workaround seems to be

#[cfg(not(test))]
// std overriding code here

In cases where std is acceptable if you still want to test

EDIT: This may actually be a bug with workspacing? if I delete my workspace file and use them separately this works fine. The other option is that I'm doing something wrong 😄

@daxpedda
Copy link
Contributor

@daxpedda daxpedda commented Aug 13, 2021

This seems to activate desired_feature even during non-dev build, for some reason.

Unfortunately this doesn't seem to work on an edge case in a no_std environment where your crate allows no_std and has a feature which enables std.

I assume this is because its importing std for the developer dependencies section, and this function is trying to overwrite a now-included stdlib function.

EDIT: This may actually be a bug with workspacing? if I delete my workspace file and use them separately this works fine. The other option is that I'm doing something wrong 😄

Indeed the issue is simply because of the old bug of features from dependencies and dev-dependencies being unified, see #4866. This is fixed by using the new resolver.

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Dec 20, 2021
Most of our current testing is via an integration test,
which I designed after reading
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html

However, I then wanted to use our internal OCI build tooling from the
integration test, which is blocked (by design) from seeing non-`pub`
things.

I initially tried doing:
```
pub fn do_oci_stuff(...)
```

but that didn't work.  (Perhaps it should?)

After some searching I came across this gem:
rust-lang/cargo#2911 (comment)
Which, looks an amazing hack.  But it works.

The *big* downside of this is that now in order to properly test
building without the integration test feature, one must do so
via a fully distinct crate.
cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Dec 20, 2021
Most of our current testing is via an integration test,
which I designed after reading
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html

However, I then wanted to use our internal OCI build tooling from the
integration test, which is blocked (by design) from seeing non-`pub`
things.

I initially tried doing:
```
#[cfg(test)]
pub fn do_oci_stuff(...)
```

but that didn't work.  (Perhaps it should?)

After some searching I came across this gem:
rust-lang/cargo#2911 (comment)
Which, looks an amazing hack.  But it works.

The *big* downside of this is that now in order to properly test
building without the integration test feature, one must do so
via a fully distinct crate.
ltratt added a commit to ltratt/yk that referenced this issue Feb 13, 2022
…old.

This commit introduces a new test which showed up more than one race
condition in the location transition function.

The first problem was that I was, even by my standards, incredibly
stupid: I created a dummy Location and locked that, forgetting that did
not lock the original Location. Not only did we have a race condition,
but because we didn't lock things properly, we were in undefined
behaviour. The fix is to resuscitate an old idea: the Compiling state is
now:

  Compiling(Arc<Mutex<Option<Box<CompiledTrace>>>>)

Thus the compiling thread and any "normal" thread can communicate
whether compilation has finished safely. This does mean that while a
location is in the Compiling state, one has to perform two locks. Since
we use parking_lot's `Mutex` type, in the (common) uncontended case this
is fairly cheap.

As a happy coincidence, moving (back) to this scheme also got rid of
some code that turned pointers into usize's: it might well have been
correct, but it was very fragile.

The second problem is that there was a subtle race condition when
deciding whether it was safe to start tracing or not. This race
condition did not lead to undefined behaviour, but meant that two
threads could start trying to start tracing on the same location. The
fix for this is be less clever when starting tracing: we don't go around
in a loop when starting tracing. If we fail to start tracing, we give
up, on the basis that if tracing is the right thing to do, there'll be
another opportunity soon.

Notice that the `Compiled::non_null` function, needed by `ykrt` for
testing purposes requires the cunning "only enable a feature when
testing" Cargo hack from:

  rust-lang/cargo#2911 (comment)
ltratt added a commit to ltratt/yk that referenced this issue Feb 13, 2022
…old.

This commit introduces a new test which showed up more than one race
condition in the location transition function.

The first problem was that I was, even by my standards, incredibly
stupid: I created a dummy Location and locked that, forgetting that did
not lock the original Location. Not only did we have a race condition,
but because we didn't lock things properly, we were in undefined
behaviour. The fix is to resuscitate an old idea: the Compiling state is
now:

  Compiling(Arc<Mutex<Option<Box<CompiledTrace>>>>)

Thus the compiling thread and any "normal" thread can communicate
whether compilation has finished safely. This does mean that while a
location is in the Compiling state, one has to perform two locks. Since
we use parking_lot's `Mutex` type, in the (common) uncontended case this
is fairly cheap.

As a happy coincidence, moving (back) to this scheme also got rid of
some code that turned pointers into usize's: it might well have been
correct, but it was very fragile.

The second problem is that there was a subtle race condition when
deciding whether it was safe to start tracing or not. This race
condition did not lead to undefined behaviour, but meant that two
threads could start trying to start tracing on the same location. The
fix for this is be less clever when starting tracing: we don't go around
in a loop when starting tracing. If we fail to start tracing, we give
up, on the basis that if tracing is the right thing to do, there'll be
another opportunity soon.

Notice that the `Compiled::non_null` function, needed by `ykrt` for
testing purposes requires the cunning "only enable a feature when
testing" Cargo hack from:

  rust-lang/cargo#2911 (comment)
ltratt added a commit to ltratt/yk that referenced this issue Feb 13, 2022
…old.

This commit introduces a new test which showed up more than one race
condition in the location transition function.

The first problem was that I was, even by my standards, incredibly
stupid: I created a dummy Location and locked that, forgetting that did
not lock the original Location. Not only did we have a race condition,
but because we didn't lock things properly, we were in undefined
behaviour. The fix is to resuscitate an old idea: the Compiling state is
now:

  Compiling(Arc<Mutex<Option<Box<CompiledTrace>>>>)

Thus the compiling thread and any "normal" thread can communicate
whether compilation has finished safely. This does mean that while a
location is in the Compiling state, one has to perform two locks. Since
we use parking_lot's `Mutex` type, in the (common) uncontended case this
is fairly cheap.

As a happy coincidence, moving (back) to this scheme also got rid of
some code that turned pointers into usize's: it might well have been
correct, but it was very fragile.

The second problem is that there was a subtle race condition when
deciding whether it was safe to start tracing or not. This race
condition did not lead to undefined behaviour, but meant that two
threads could start trying to start tracing on the same location. The
fix for this is be less clever when starting tracing: we don't go around
in a loop when starting tracing. If we fail to start tracing, we give
up, on the basis that if tracing is the right thing to do, there'll be
another opportunity soon.

Notice that the `Compiled::non_null` function, needed by `ykrt` for
testing purposes requires the cunning "only enable a feature when
testing" Cargo hack from:

  rust-lang/cargo#2911 (comment)
ltratt added a commit to ltratt/yk that referenced this issue Feb 21, 2022
…old.

This commit introduces a new test which showed up more than one race
condition in the location transition function.

The first problem was that I was, even by my standards, incredibly
stupid: I created a dummy Location and locked that, forgetting that did
not lock the original Location. Not only did we have a race condition,
but because we didn't lock things properly, we were in undefined
behaviour. The fix is to resuscitate an old idea: the Compiling state is
now:

  Compiling(Arc<Mutex<Option<Box<CompiledTrace>>>>)

Thus the compiling thread and any "normal" thread can communicate
whether compilation has finished safely. This does mean that while a
location is in the Compiling state, one has to perform two locks. Since
we use parking_lot's `Mutex` type, in the (common) uncontended case this
is fairly cheap.

As a happy coincidence, moving (back) to this scheme also got rid of
some code that turned pointers into usize's: it might well have been
correct, but it was very fragile.

The second problem is that there was a subtle race condition when
deciding whether it was safe to start tracing or not. This race
condition did not lead to undefined behaviour, but meant that two
threads could start trying to start tracing on the same location. The
fix for this is be less clever when starting tracing: we don't go around
in a loop when starting tracing. If we fail to start tracing, we give
up, on the basis that if tracing is the right thing to do, there'll be
another opportunity soon.

Notice that the `Compiled::non_null` function, needed by `ykrt` for
testing purposes requires the cunning "only enable a feature when
testing" Cargo hack from:

  rust-lang/cargo#2911 (comment)
ltratt added a commit to ltratt/yk that referenced this issue Feb 21, 2022
…old.

This commit introduces a new test which showed up more than one race
condition in the location transition function.

The first problem was that I was, even by my standards, incredibly
stupid: I created a dummy Location and locked that, forgetting that did
not lock the original Location. Not only did we have a race condition,
but because we didn't lock things properly, we were in undefined
behaviour. The fix is to resuscitate an old idea: the Compiling state is
now:

  Compiling(Arc<Mutex<Option<Box<CompiledTrace>>>>)

Thus the compiling thread and any "normal" thread can communicate
whether compilation has finished safely. This does mean that while a
location is in the Compiling state, one has to perform two locks. Since
we use parking_lot's `Mutex` type, in the (common) uncontended case this
is fairly cheap.

As a happy coincidence, moving (back) to this scheme also got rid of
some code that turned pointers into usize's: it might well have been
correct, but it was very fragile.

The second problem is that there was a subtle race condition when
deciding whether it was safe to start tracing or not. This race
condition did not lead to undefined behaviour, but meant that two
threads could start trying to start tracing on the same location. The
fix for this is be less clever when starting tracing: we don't go around
in a loop when starting tracing. If we fail to start tracing, we give
up, on the basis that if tracing is the right thing to do, there'll be
another opportunity soon.

Notice that the `Compiled::non_null` function, needed by `ykrt` for
testing purposes requires the cunning "only enable a feature when
testing" Cargo hack from:

  rust-lang/cargo#2911 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-features
Projects
None yet
Development

No branches or pull requests