New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Result::Err is not interpreted as a failure #50
Comments
Hmm, we definitely need to look into that. For now you can work around it with Most of tests that I wrote for such scenarios included explicit unwrap without returning Result from test function, but being compatible with rust test is probably way to go. |
I just ran into this on accident, I wasn't expecting |
This really should be tracked as a Bug not an Enhancement: moving from This has lead to a large number of tests passing when they should have failed for us. We're now looking for an alternative to |
Yeah, you have a point that it should be a bug. Unfortunately I don't know when I'll have time to implement a fix. Maintaining this lib almost alone with work and studies in parallel ain't easiest thing to do & I don't see many people lining up trying to work on open issues. |
I recommend forking the repository. It is the essence of open source, after all. And while we made this library open-source and available for everyone, we (usually) are still driven by what we need in our personal and professional projects. From my experience, I didn't fix this bug simply because it had no severity. So while we value the input from other users, we have a limited amount of time for OSS projects. That's why I didn't make any promises about this fix. If someone else thinks this is a critical issue, I'm more than happy to guide how to fix it and review and merge the PR. |
I appreciate the responses on this. And yes, I do understand the nature of OSS and that everyone's time is limited for these sorts of things 🙂 Also I'd like to apologise for my original comment being as confrontational as it is: hitting this bug added to an already stressful day and I shouldn't have taken that out here. As some additional context for why I'm finding this behaviour surprising: my main issue with this particular bug is that Given that all we want/need is parameterised tests, I've done as you suggest and written a minimal crate that just handles that side of things. |
One short-term option might be to cause
This causes this error for a
Thoughts on this? If this approach seems OK I could put together a PR. |
That unfortunately would be a major breaking change.
As you can see the return type is supported in that case and it is asserted with However, I think we could do such a detection with extra condition: |
Because it is a bug, we would have to patch it in the 1.x branch. |
Thanks @frondeus. Here's an alternate version targeting the
Now given these test cases: #[cfg(test)]
mod tests {
use test_case::test_case;
#[test_case(2, 2; "Wrong")]
#[test_case(2, 3; "Right")]
fn return_no_expect(a: i32, b: i32) -> Result<i32, String> {
let result = a + b;
return if result == 5 {
Ok(result)
} else {
Err("Not equal 5".to_owned())
}
}
#[test_case(2, 2 => Err("Not equal 5".to_owned()); "Wrong")]
#[test_case(2, 3 => Ok(5); "Right")]
fn return_with_expect(a: i32, b: i32) -> Result<i32, String> {
let result = a + b;
return if result == 5 {
Ok(result)
} else {
Err("Not equal 5".to_owned())
}
}
} You get this:
|
That looks like an awesome fix. Thank you for your contribution :)! |
No worries, I'll put it together of the next few days. |
@frondeus There doesn't actually appear to be a v1 support branch to target; the |
1.0.x support is old branch with some incompatible changes. @frondeus we probably should create similar branch for 1.2.1 from 1.2.1 tag since master is already at 2.0.0-rc. I guess we need to come up with some proper branching strat. |
I will prepare proper target branch today |
Ok, had some free time, I created branch I see now that it'd be better if I worked on 2.0 release on some different branch than master. Well, will remember to do so in the future. |
As @bspradling noticed this code: #[test_case("ISBN:1839485743"; "isbn10")]
#[test_case("ISBN:1839485743123"; "isbn13")]
#[tokio::test]
async fn test_bibliography_key_serde(expected: &str) -> Result<(), Box<dyn Error>> {
let key: BibliographyKey = serde_json::from_str(expected)?;
let actual = serde_json::to_string(&key)?;
assert_eq!(expected, actual);
Ok(())
} cannot be simply fixed by adding Because In that case the author doesn't want to check the error message, he just want to make sure the function does not return any errors at all. But Rust doesn't know that. The only solution I see for now is to do something like this: #[test_case("ISBN:1839485743"; "isbn10")]
#[test_case("ISBN:1839485743123"; "isbn13")]
#[tokio::test]
async fn test_bibliography_key_serde(expected: &str) {
async fn inner(expected: &str) -> Result<(), Box<dyn Error>> {
let key: BibliographyKey = serde_json::from_str(expected)?;
let actual = serde_json::to_string(&key)?;
assert_eq!(expected, actual);
Ok(())
}
inner(expected).await.unwrap();
} But it requires a lot of boilerplate and it's just... ugly. Therefore I think this case proves we should increase the priority of this issue. It can be really inconvenient and the workaround is not easy as I wish it would be. Also keep in mind that I believe matching the Err matching could be possible but the Rust does not support it either. #[test]
#[should_panic(expected = "123")]
fn my_test() -> Result<(), Box<dyn Error>> { ... } but perhaps |
About the last paragraph: I would imagine something like: #[test_case(1, 2 => panics "fail")]
fn test_me(a: usize, b: usize) -> Result<usize, Box<dyn Error>> {
....
} To work just fine. |
Extra: #86 (comment) |
In Rust 2018 test can return a
Result
type, withResult::Err
being interpreted as a failure. However with parameterised tests returned errors are treated as passing. While this can be fixed by specifying an expected value, it is unintuitive.Example:
Expected: Both
tests::with_params::failing()
andis_5()
fail.Actual: Only
is_5()
fails.The text was updated successfully, but these errors were encountered: