Test/build all combinations of features

[vorner]

I have already seen several bugs related to non-default selection of features (eg. it doesn't compile if --no-default-features is passed, something else fails to compile if a dependency gets additional feature through some other path in the dep graph).

I'm wondering if there's a way to help preventing these kinds of things. Some random ideas are:

• A flag to cargo test that would find all allowed combinations of features (or maybe even features of the dependencies) and iterate through that?
• Require this to pass on cargo publish?

But this is just an idea. I'm not sure this is the right place to start discussion, so if it's not, I'd be happy to be pointed to the right one.

[lukaslueg]

I've been bitten by this once or twice and I think it's a useful feature to have in Cargo.

I'm wondering about how to provide the UI for this, though. Consider a crate that provides features musl, libc, foo and bar. None, one or both of foo and bar and exactly one of musl or libc can be specified; the crate is expected to and will fail to compile with none or both of musl and libc given. The valid combinations therefor are (musl, ), (musl, foo), (musl, foo, bar), (libc, ), (libc, foo) and (libc, foo, bar).
The point here is how to steer Cargo away from (rare) invalid combinations without having to explicitly state (plentiful, hard to get right) valid ones; we don't want a set-intersection-DSL on the commandline.

One way to solve this are two new flags --feature-combinations and --all-feature-combinations for cargo test. Given in conjunction with --features and --default-features, Cargo iterates over all unique combinations and builds+executes the test suite. The features given by --features are kept fixed, the ones given by --feature-combinations or discovered by --all-feature-combinations are mixed in.

To drive the example above home, the user would have to use two invocations, cargo test --features libc --feature-combinations foo, bar; cargo test --features musl --feature-combinations foo, bar.

A crate with no poisonous feature combinations can be tested by cargo test --all-feature-combinations, a subset of the default features (including the default features themselves) by cargo test --default-features --all-feature-combinations.

[luser]

The problem here is that you get combinatorial explosion in the number of features. There are 2ⁿ build configurations for n features. We've seen this countless times with the Firefox build system, where if we provide a --disable-feature-x option but don't have builds for it in CI it inevitably gets broken.

[vorner]

Yes, it's likely problematic with large code bases, because it takes long to compile just one combination, not saying much about 2²⁰.

But with a small crate with just 3 or 4 features, it should be possible to do as part of a single build (eg. not defining 16 different build jobs, but one that runs them linearly).

[jD91mZM2]

While we're waiting, a variation of this script could be used for small and simple crates. Would really be awesome to have this as a built-in command though :)

[TotalKrill]

Yeah, I have seen this happen a few times to many as well. Its really annoying since in small crates this is usually just an oversight of the developer, forgetting to add a #[cfg(feature = "bla")] flag to a use statement of something similar.

Would like to have a default option of just testing all flags in different combos, and then specifying valid combos or ignored combos in the Cargo.toml, similar to how features can specify which dependencies they require. Features can already I think specify other features they require, and thus adding tests like this would help debug the Cargo.toml a bit as well.

[frewsxcv]

To those who may be interested, I just wrote build-all-features and test-all-features Cargo subcommands that build/test all feature combinations for a crate. You can check it out here: https://github.com/frewsxcv/cargo-all-features

[frewsxcv]

It also appears to be implemented in cargo-hack's --feature-powerset flag:

https://github.com/taiki-e/cargo-hack#usage