skip cases based on feature flag?

[flokli]

I use rstest to run a bunch of "acceptance tests" across different implementations all exposing the same trait.

I use rstest_reuse's #[template] and #[apply(…)] to write a template containing all implementations I want to test with (as test cases), and use apply for each individual test:

#[template]
#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
pub fn directory_services(#[case] directory_service: impl DirectoryService) {}

#[apply(directory_services)]
#[tokio::test]
async fn test_non_exist(directory_service: impl DirectoryService) {
    let resp = directory_service.get(&DIRECTORY_A.digest()).await;
    assert!(resp.unwrap().is_none())
}

#[apply(directory_services)]
#[tokio::test]
async fn put_get(directory_service: impl DirectoryService) {
    // ...
}

This keeps the amount of boilerplate for each test case to a minimum, thanks a lot :-)

Now I want to add an additional store implementation, but its code is behind a feature flag.

It seems it's not possible to have a single #[case:…] with a #[cfg(feature = "cloud")] - it doesn't apply to the macro(s), but entirely enables/disables the whole pub fn directory_services template function:

#[template]
#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
#[cfg(feature = "cloud")]
#[case::bigtable(make_bigtable().await)]
pub fn directory_services(#[case] directory_service: impl DirectoryService) {}

Currently my only way out seems to be having two implementations of pub fn directory_services, one without, and one with the feature flag:

#[cfg(not(feature = "cloud"))]
#[template]
#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
pub fn directory_services(#[case] directory_service: impl DirectoryService) {}

#[cfg(feature = "cloud")]
#[template]
#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
#[case::bigtable(make_bigtable().await)]
pub fn directory_services(#[case] directory_service: impl DirectoryService) {}

This obviously won't scale when feature flags get more (granular). Any thoughts on how to do this? Maybe adding an optional argument, allowing to express a cfg!() expression for when a case should get included?

[la10736]

Can you just check if it works if you don't use template? I guess it doesn't 😢

Maybe the compiler process cfg attrs before the rstest's procedural macros. I guess that I can introduce a #[rstest::cfg(...)] macro that it's expanded in #[cfg(...)], but that's my last resource. I need to understand better what's happening.

[flokli]

Without #[template], the #[cfg(feature= "…] statement also applies to the entire block:

#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
#[cfg(feature = "cloud")]
#[case::bigtable(make_bigtable().await)]
#[tokio::test]
async fn pingpong(#[case] directory_service: impl DirectoryService) {
    let resp = directory_service.get(&DIRECTORY_A.digest()).await;
    assert!(resp.unwrap().is_none())
}

This either runs all tests if the "cloud" feature flag is set, or none (if it's not set).

Priorities of procedural macros is something I didn't expose myself with yet, so sorry for not being a lot of help 🙈

[la10736]

Ok, you should use cfg_attr instead:

#[rstest]
#[case::grpc(make_grpc_directory_service_client().await)]
#[case::memory(directoryservice::from_addr("memory://").await.unwrap())]
#[case::sled(directoryservice::from_addr("sled://").await.unwrap())]
#[cfg_attr(feature = "cloud", case::bigtable(make_bigtable().await))]
#[tokio::test]
async fn pingpong(#[case] directory_service: impl DirectoryService) {
    let resp = directory_service.get(&DIRECTORY_A.digest()).await;
    assert!(resp.unwrap().is_none())
}

That should work also with #[template]/#[apply(...)]

[flokli]

Thanks! Indeed this works, also for template/apply!

Let me check if there's a good place to document this (maybe somewhere close to test cases).

[flokli]

I opened #236, PTAL!