New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest -v does not include "Differing Items" with long-enough dicts #1512

Closed
quodlibetor opened this Issue Apr 13, 2016 · 6 comments

Comments

Projects
None yet
5 participants
@quodlibetor
Contributor

quodlibetor commented Apr 13, 2016

in the environment (I also get the error in pytest 2.8.7)

$ uname -v
Darwin Kernel Version 13.4.0
$ pip freeze
py==1.4.31
pytest==2.9.1

given ex.py:

def test_long_assertion():
    long = 'a' * 50
    a = {i: long for i in range(8)}
    b = dict(**a)
    b[7] = 'short'
    assert a == b

Invoking py.test gives a very helpful error message:

$ py.test ex.py
========= test session starts =========
platform darwin -- Python 2.7.5, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1
collected 1 items

ex.py F

=================== FAILURES ===================
____________________ test_long_assertion ____________________
    def test_long_assertion():
        long = 'a' * 50
        a = {i: long for i in range(8)}
        b = dict(**a)
        b[7] = 'short'
>       assert a == b
E       assert {0: 'aaaaaaaa...aaaaaaa', ...} == {0: 'aaaaaaaaa...aaaaaaa', ...}
E         Omitting 7 identical items, use -v to show
E         Differing items:
E         {7: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'} != {7: 'short'}
E         Use -v to get the full diff

ex.py:6: AssertionError

However, running with -v breaks the diff (note the lack of "differing items" at the end):

$ py.test -v ex.py
================================== test session starts ===================================
platform darwin -- Python 2.7.5, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /Users/bwm/findable/virtualenvs/consumer/bin/python
cachedir: .cache
rootdir: /private/tmp, inifile:
plugins: cov-2.2.1
collected 1 items

ex.py::test_long_assertion FAILED

======================================== FAILURES ========================================
__________________________________ test_long_assertion ___________________________________

    def test_long_assertion():
        long = 'a' * 50
        a = {i: long for i in range(8)}
        b = dict(**a)
        b[7] = 'short'
>       assert a == b
E       assert {0: 'aaaaaaaa...aaaaaaa', ...} == {0: 'aaaaaaaaa...aaaaaaa', ...}
E         Common items:
E         {0: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          1: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          2: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          3: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          4: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          5: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
E          6: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}
E         Differing items:
E         Detailed information truncated (11 more lines), use "-vv" to show

ex.py:6: AssertionError
================================ 1 failed in 0.01 seconds ================================

-vv does include everything (including that diff that is excluded), but it's very verbose when I have dozens of items in my dicts.

This is not an issue when the dict is short -- maybe because we get the full diff instead of "Differing Items":

$ cat ex2.py
def test_long_assertion():
    long = 'a' * 50
    a = {i: i for i in range(8)}
    b = dict(**a)
    b[7] = 'short'
    assert a == b
$ py.test ex2.py -v
================================= test session starts ==================================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /Users/bwm/findable/virtualenvs/tmp-5f9f221206eaa0d8/bin/python2.7
cachedir: .cache
rootdir: /private/tmp, inifile:
collected 1 items

ex2.py::test_long_assertion FAILED

======================================= FAILURES =======================================
_________________________________ test_long_assertion __________________________________

    def test_long_assertion():
        long = 'a' * 50
        a = {i: i for i in range(8)}
        b = dict(**a)
        b[7] = 'short'
>       assert a == b
E       assert {0: 0, 1: 1, 2: 2, 3: 3, ...} == {0: 0, 1: 1, 2: 2, 3: 3, ...}
E         Common items:
E         {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6}
E         Differing items:
E         {7: 7} != {7: 'short'}
E         Full diff:
E         - {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7}
E         ?                                               ^
E         + {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 'short'}
E         ?
@mattduck

This comment has been minimized.

Contributor

mattduck commented Sep 19, 2016

Hey @flub, this is caused because all output that has verbosity < 2 is truncated at 10 lines:

show_max = 10
.

The py.test ex.py example above only shows the differing lines because the assertion explanation is short. With py.test -v ex.py, the explanation is more verbose so gets truncated. Pytest isn't explicitly choosing to remove the Differing lines: items, it just looks like it is because Differing lines: is the last line before truncation.

Not sure the best approach here. Do you think it's more useful to just remove the common items output when verbosity=1? The differing items are probably more useful to the user.

We could also change the truncation formatting to make it clearer that all output is truncated.

@flub

This comment has been minimized.

Member

flub commented Sep 19, 2016

sounds good to both :)

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

Don't display dict common items if verbosity=1
Part one of pytest-dev#1512.

If verbosity=1, assertion explanations are truncated at 10 lines. In this
situation, it's more important to tell the user which dictionary items are
different than which are the same.

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

Tidy formatting of assertion truncation
Part two of pytest-dev#1512. Update the format
of the truncation message to help make it clear that pytest truncates the
entire assertion output when verbosity < 2.

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

Don't display dict common items if verbosity=1
Part one of pytest-dev#1512.

If verbosity=1, assertion explanations are truncated at 10 lines. In this
situation, it's more important to tell the user which dictionary items are
different than which are the same.

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

Tidy formatting of assertion truncation
Part two of pytest-dev#1512. Update the format
of the truncation message to help make it clear that pytest truncates the
entire assertion output when verbosity < 2.

mattduck added a commit to mattduck/pytest that referenced this issue Sep 19, 2016

@merwok

This comment has been minimized.

merwok commented Jan 28, 2017

Not sure the best approach here. Do you think it's more useful to just remove the common items
output when verbosity=1? The differing items are probably more useful to the user.

This behaviour would be great.

@mattduck

This comment has been minimized.

Contributor

mattduck commented Jan 30, 2017

hey @merwok, this change has been merged into the features branch, so it should be included in the next minor release (3.1.0).

@merwok

This comment has been minimized.

merwok commented Jan 30, 2017

Good news!

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Jan 30, 2017

Fixed by #1951

This was referenced Mar 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment