Better diff for asserts on dicts

[SuperDoxin]

Currently in some edge cases py.test returns utterly useless diffs for asserts on dicts, it doesn't provide the context needed to find which parts are differing in deeply nested dicts.

A better, and simpler, solution would be to use a json serializer to serialize the dicts and then use difflibs unified diff to generate the final diff. this solution will produce a diff in an easy to read format that people are familiar with and provides enough context to find the right place in the dict where changes need to be made.

any objects not json serializable in the dict can be adequately dealt with by using repr as serializer.

this whole function could be replaced with ~5 lines of code.

[RonnyPfannschmidt]

would you like to turn that into a pull request?

also note, that for "proper" diff of nested structures one needs a path to the object being diffed as well

[SuperDoxin]

I'd love to, but I can't seem to figure out the correct way to run pytest from source.

[The-Compiler]

I usually do something like:

$ virtualenv .venv
$ ./.venv/bin/pip install -e .
$ ./.venv/bin/python -m pytest ...

And to run the tests/linters, simply run tox.

[TimDaub]

I'd also be interested in this :)

[SuperDoxin]

I've had a good look at this recently, and it turns out to be a way bigger project than I had initially estimated. I'm afraid I can't quite find the time to get this done. sorry.

[nicoddemus]

No problem, thanks for the heads up.

[sabrown89]

I am interested in tackling this. Just now starting to look into it.

[four43]

Just to add to this old (but important to me!) issue - it seems if you call assert {} == {} from within a function that is called from a test, pytest doesn't really help you. If I move my assertions to within the main test body they work great, but if they are called via a function it doesn't help.

[The-Compiler]

@four43 That is unrelated to this issue and intended - pytest's assertion introspection/rewriting is only active for test_*.py files (and a few more cases) by default, to avoid changing the behavior of your code under test.

See the docs for details:

• Assertion introspection details
• Assertion Rewriting

[four43]

Hey okay that's pretty slick. Thanks for the tip, I bet that solves my problem. I appreciate it.

[m-aciek]

Just for the record (and inspiration) there are two pytest plugins that prints better dict comparisons:

• pytest-icdiff
• pytest-dictsdiff

Source: https://eshlox.net/2020/04/07/better-diffs-in-pytest

[nicoddemus]

Thanks @m-aciek,

I had a few minutes so I tried pytest-dictsdiff, here's the output using the example on the README:

E               Failed:  => CHANGE: at key 'dob.age' values are different | Left: 44 | Right: 34
E                => CHANGE: at key 'dob.date' values are different | Left: '1983-11-04T01:21:14Z' | Right: '1953-11-04T01:21:04Z'
E                => CHANGE: at key 'location.city' values are different | Left: 'tholen' | Right: 'Tholen'
E                => CHANGE: at key 'location.postcode' values are different | Left: 64504 | Right: 64509
E                => CHANGE: at key 'name.first' values are different | Left: 'zeyneb' | Right: 'Zeyneb'
E                => CHANGE: at key 'name.last' values are different | Left: 'elfring' | Right: 'Elfring'

Style aside, the diff is readable to me.

While at it decided to take a stab at @SuperDoxin's idea of using json+difflib (using the same data sample from pytest-dictsdiff):

def test_compare_dicts():
    import json
    left = json.dumps(RESULT, indent=2, sort_keys=True)
    right = json.dumps(EXPECTED_DATA, indent=2, sort_keys=True)

    import difflib
    diff = difflib.unified_diff(left.splitlines(True), right.splitlines(True), fromfile="left", tofile="right")
    assert 0, "\n" + "".join(diff)

Here's the output:

E       AssertionError:
E         --- left
E         +++ right
E         @@ -1,8 +1,8 @@
E          {
E            "cell": "(056)-022-8631",
E            "dob": {
E         -    "age": 44,
E         -    "date": "1983-11-04T01:21:14Z"
E         +    "age": 34,
E         +    "date": "1953-11-04T01:21:04Z"
E            },
E            "email": "zeyneb.elfring@example.com",
E            "gender": "female",
E         @@ -11,12 +11,12 @@
E              "value": "36180866"
E            },
E            "location": {
E         -    "city": "tholen",
E         +    "city": "Tholen",
E              "coordinates": {
E                "latitude": "46.8823",
E                "longitude": "175.8856"
E              },
E         -    "postcode": 64504,
E         +    "postcode": 64509,
E              "state": "groningen",
E              "street": "2074 adriaen van ostadelaan",
E              "timezone": {
E         @@ -34,8 +34,8 @@
E              "uuid": "37e30c59-bc79-4172-aac6-e2c640e165fa"
E            },
E            "name": {
E         -    "first": "zeyneb",
E         -    "last": "elfring",
E         +    "first": "Zeyneb",
E         +    "last": "Elfring",
E              "title": "mrs"
E            },
E            "nat": "NL",
E
E       assert 0

I like it too, but this has the problematic downside that json doesn't handle custom objects, and AFAIK there's no easy way to generalize it.

[BenjaminSchubert]

@nicoddemus I believe the reason to use json dumping here is that the json output will format the diff with consistent { on their own lines and all of those small tweaks?

If so, I believe it should be doable to extend/override the pretty printer used by pytest to have the same behavior. It might be quite some code due to how the PrettyPrinter is implemented, but would continue to work as today for everything else.

I'd be happy to attempt a solution with prettyprinter if no-one is actively working on this and pytest would be fine with that added code

[nicoddemus]

@nicoddemus I believe the reason to use json dumping here is that the json output will format the diff with consistent { on their own lines and all of those small tweaks?

Exactly. 👍

If so, I believe it should be doable to extend/override the pretty printer used by pytest to have the same behavior. It might be quite some code due to how the PrettyPrinter is implemented, but would continue to work as today for everything else.

That would be amazing!

I'd be happy to attempt a solution with prettyprinter if no-one is actively working on this and pytest would be fine with that added code

Oh yes, please go ahead, thanks a lot.

[BenjaminSchubert]

#11537 would be my proposed solution