Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug with tolerances and other undesired behavior with approx using dict #8703

Open
danilomendesdias opened this issue May 26, 2021 · 1 comment
Labels
topic: approx Related to pytest.approx function

Comments

@danilomendesdias
Copy link

danilomendesdias commented May 26, 2021

Hi there,

Since I was introduced (yesterday) to the awesome feature of using approx with dictionaries I started to investigate its usage and I found some interesting things that I would like to share with you. I'm using pytest 6.2.2.

Let's start by showing what is perfectly working, so we have a baseline to the other cases:

 [1, 2, 3 + 1e-6] == pytest.approx([1, 2, 3], abs=1e-6)
>> False 

[1, 2, 3 + 1e-6]  == pytest.approx([1, 2, 3], abs=1e-5)
>> True

{'a': [1, 2, 3 + 1e-6]} == pytest.approx({'a': [1, 2, 3]}, abs=1e-6)
>> False

But here, the symmetry starts to break:

{'a': [1, 2, 3 + 1e-6]} == pytest.approx({'a': [1, 2, 3]}, abs=1e-5)
>> False

For brevity, I'm not showing here, but you can reproduce analogous issues with default tolerances.
I think this is a bug, but maybe you already have an opinion about this. I would like to know if I can help or if I should move on with another approach.

Beyond that, I also found some weird behavior with Numpy arrays. I know, this is even more tricky, but I would like to report some findings. Here we go:

{'a': np.array([1, 2, 3])} == pytest.approx({'a': [1, 2, 3]})
>> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

So, the numpy array (LHS) is being converted to a list and, well, maybe this could be considered a wrong usage, as the feature is very complex by itself. But there is also this case:

{'a': np.array([1, 2, 3])} == pytest.approx({'a': np.array([1, 2, 3])})
>> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Now I have two numpy arrays and both are converted to lists and are being compared directly with == operator, which gives us this error.

Any thoughts? If there's something I can help, I would love to. I just ask for some tips and I can prepare a PR.

Thank you!

@danilomendesdias danilomendesdias changed the title Bug with tolerances and other undesired behaviors with approx using dict Bug with tolerances and other undesired behavior with approx using dict May 26, 2021
@Zac-HD Zac-HD added the topic: approx Related to pytest.approx function label May 27, 2021
@bluetech
Copy link
Member

I believe the two issues you describe both derive from the fact that approx is not recursive - that is, comparing two sequences or mappings just compares the lengths and then compares the values as scalars. If the values are themselves sequences/mappings, it doesn't try to compare them as sequences/mappings.

To me it does seem intuitive that it should be recursive, although I almost never use approx myself so I'm probably not the right person to judge this. I also can't exactly say how difficult it would be to implement -- with the right refactoring it should be doable cleanly, however the assert repr (the nice formatted message shown on failure) would be more complex, unless the existing pytest recursive assert repr infrastructure could somehow be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: approx Related to pytest.approx function
Projects
None yet
Development

No branches or pull requests

3 participants