Can't properly compare dictionaries if specific key used

[slyapustin]

Hi!

I'm trying to compare xml documents converted to dict and here issue:

    from deepdiff import DeepDiff

    d1 = {
        'item': [
            {'title': 1, 'http://purl.org/rss/1.0/modules/content/:encoded': '1'},
            {'title': 2, 'http://purl.org/rss/1.0/modules/content/:encoded': '2'}
        ]
    }

    d2 = {
        'item': [
            {'http://purl.org/rss/1.0/modules/content/:encoded': '1', 'title': 1},
            {'http://purl.org/rss/1.0/modules/content/:encoded': '2', 'title': 2}
        ]
    }

    DeepDiff(d1, d2, ignore_order=True)  # Returns non empty dictionary
    DeepDiff(d1, d2, ignore_order=False)  # Returns {}

I expected result should be the same - empty dictionaries.

[seperman]

@inoks Thanks for reporting this. This is caused by pickle caring about dictionary order which I was not aware of:

>>> import pickle
>>> a={'title': 1, 'http://purl.org/rss/1.0/modules/content/:encoded': '1'}
>>> b={'http://purl.org/rss/1.0/modules/content/:encoded': '1', 'title': 1}
>>> a==b
True
>>> pickle.dumps(a)==pickle.dumps(b)
False
>>> pickle.dumps(a)
"(dp0\nS'http://purl.org/rss/1.0/modules/content/:encoded'\np1\nS'1'\np2\nsS'title'\np3\nI1\ns."
>>> pickle.dumps(b)
"(dp0\nS'title'\np1\nI1\nsS'http://purl.org/rss/1.0/modules/content/:encoded'\np2\nS'1'\np3\ns."

I will fix it as soon as I can.

[seperman]

Testing further, it works correctly 90% of times in python3. I'm gonna be refactoring some code to make this work.

[ThriceGood]

Hey,

I'm wondering if this is related to the issue I'm having:

d1 = {
    'key1': 'val1',
    'key2': [
        {
            'key3': 'val3',
            'key4': 'val4',
        },
        {
            'key5': 'val5',
            'key6': 'val6',
        },
    ],
}
d2 = {
    'key1': 'val1',
    'key2': [
        {
            'key3': 'val3',
            'key4': 'val4',
        },
        {
            'key5': 'CHANGE',
            'key6': 'val6',
        },
    ],
}
>>> diff = DeepDiff(d1, d2)
{'values_changed': {"root['key2'][1]['key5']": {'newvalue': 'CHANGE', 'oldvalue': 'val5'}}}
"""
    works as expected. What seems to cause a problem it when
    I re-order the list and have a changed value in one of the lists dicts. 
    Using with ignore_order=True.
"""
d1 = {
    'key1': 'val1',
    'key2': [
        {
            'key3': 'val3',
            'key4': 'val4',
        },
        {
            'key5': 'val5',
            'key6': 'val6',
        },
    ],
}
d2 = {
    'key1': 'val1',
    'key2': [
        {
            'key5': 'CHANGE',
            'key6': 'val6',
        },
        {
            'key3': 'val3',
            'key4': 'val4',
        },
    ],
}
>>> diff = DeepDiff(d1, d2, ignore_order=True)
{'iterable_item_removed': {"root['key2'][1]": {'key6': 'val6', 'key5': 'val5'}}, 'iterable_item_added': {"root['key2'][0]": {'key6': 'val6', 'key5': 'CHANGE'}}} 
"""
    it looks like it sees it as a new dict instead of a changed value in a current dict
"""

[seperman]

@ThriceGood Yep, I need to fork Pickle and make it sort data while serializing. I will do that this weekend.

[seperman]

@ThriceGood I looked at your notes again and you are correct, that is a whole different issue and it is currently by design. I will open a ticket and add you there so we can discuss.

[seperman]

This issue is fixed in 1.5.0 release. I will put it on pypi today.

[slyapustin]

Unfortunately that update does not helps with my case, I will put more complex example later.

[seperman]

@inoks Yes, please post an example that fails and reopen the ticket then. Thanks!