`iterable_compare_func` does not seem to work on nested lists

[jvacek]

Please checkout the F.A.Q page before creating a bug ticket to make sure it is not already addressed.

Describe the bug
I cannot get the compare function to work properly on lists that are not 1st-level.

To Reproduce
Define the two following objects. They are identical except that:

• the version attribute went 0.0.0->0.0.1
• in the second object has a number field removed in the numberfield_set list, and a new one added. UUID4s are used to track which one should be correlated with which, much like ids. in the example in the docs.

self_json= b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"332e5cfe-886d-41cc-b4d9-0fc1296b3ea0","name":"field_2","address":5,"documentation":"","first_bit_offset":5,"size_in_bits":6,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.0","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'
other_json=b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"056429c5-812b-4f49-9aae-1b52bd40aacd","name":"field_3","address":7,"documentation":"","first_bit_offset":8,"size_in_bits":9,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.1","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'

make the iterable compare func

def field_diff_function(x, y, level=None):
    try:
        return x["uuid"] == y["uuid"]

    except Exception:
        raise CannotCompare() from None

run the compare:

        diff_native = DeepDiff(
            json.loads(self_json),
            json.loads(other_json),
            iterable_compare_func=field_diff_function,
            ignore_order=True,
        )

the following diff is the result:

{
  "values_changed": {
    "root['numberfield_set'][1]['uuid']": {
      "new_value": "056429c5-812b-4f49-9aae-1b52bd40aacd",
      "old_value": "332e5cfe-886d-41cc-b4d9-0fc1296b3ea0"
    },
    "root['numberfield_set'][1]['name']": {
      "new_value": "field_3",
      "old_value": "field_2"
    },
    "root['numberfield_set'][1]['address']": {
      "new_value": 7,
      "old_value": 5
    },
    "root['numberfield_set'][1]['first_bit_offset']": {
      "new_value": 8,
      "old_value": 5
    },
    "root['numberfield_set'][1]['size_in_bits']": {
      "new_value": 9,
      "old_value": 6
    },
    "root['version']": {
      "new_value": "0.0.1",
      "old_value": "0.0.0"
    }
  }
}

Expected behavior
As the fields should be matched using the uuid attribute, it should show that one has been added, and the other has been removed, and not that they changed.

OS, DeepDiff version and Python version (please complete the following information):

• Python Version: 3.9.10
• deepdiff version: 5.8.0

Additional context
I'm definitely not ruling out there's a problem somewhere between the chair and the keyboard

[jvacek]

Upon further inspection, slapping a print() statement inside the compare function shows that the function is actually never run.

[LizardBlizzard]

The iterable_compare_func function is not called when you set ignore_order parameter to True from what I know

[havardthom]

The iterable_compare_func function is not called when you set ignore_order parameter to True from what I know

Is this a bug or intended behaviour? It is not mentioned in the docs atleast.

I'm doing diffs on nested dicts with list of dicts which have id keys where iterable_compare_func helps comparing correct dicts to eachother, however the performance is not very good. Setting ignore_order=True helps performance alot but wrong dicts are compared sometimes since iterable_compare_func is not used.

[Omar-Abdul-Azeez]

I'm having the same issue and was about to create an issue...
I also done some more testing and debugging...

I think there are actually 2 bugs in this:

1. This is exclusive to ignore_order=True. There's a len > 1 check that prevents the fuction from being called.
   only if there are two or more different items in both lists does the function actually get called. ie. If there's 1 addition and 1 removal it doesn't call the function to check whether they are pairs or not.
   I'm almost certain this is incorrect behaviour. I edited it locally and got this fixed (you can mimic by just introducing another element in both lists with differences). Instead of getting key 'id' changed from a to b, I'm now getting dict {'id': a} changed to dict {'id': b} ie. instead of considering k, v changes, the entire dicts are considered changes.
   https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L1094

2. Irrelevent of ignore_order. The result of the custom compare function is ignored for nested dicts
   I've tested this on a list of dicts, inside each of these dicts is a key whose value is also a list of dicts. each dict has an id key. The example below sets ignore_order=True but it works for both cases, just make sure the order is correct.

[
{'id': '1010', 'g': [ {'id': '2020'} ] },
{'id': '73', 'g': [ {'id': '101'}, {'id': '6790'} ] }
]

[
{'id': '73', 'g': [ {'id': '202'},  {'id': '15294'} ] },
{'id': '1012', 'g': [ {'id': '2020'} ] }
]

I use this as compare function. Except the prints, taken straight from the docs.

def compare_func(x, y, level=None):
    print(x)
    print(y)
    print('in')
    if not isinstance(x, dict) or not isinstance(y, dict):
        print('not dict')
        raise CannotCompare
    if x['id'] == y['id']:
        print('match')
        return True
    print('not match')
    return False

I found out that it gets called 8 times. 4 times comparing the 2 top level dicts to each other. 4 times comparing the 2 nested dicts to each other (the ones in the dict with the same id)
the prints were as expected for both the top level and nested dicts. but the results are different... 'id': '1010' and 'id': '1012' are not pairs and appeared in iterable_item_added and iterable_item_removed accordingly. but nested dicts were paired with eachother and entered in values_changed DESPITE the function returning False for all pairing checks between them...

[Omar-Abdul-Azeez]

I did some more debugging and I found out that the dicts are reported as iterable_item_removed and not as values_changed

which led me to think the problem is in the this line
https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L309
up to this point all differences found are under removed or added but once this function gets called it changes to changed for some reason... taking a quick look inside that function there's a suspicious mutual_add_removes_to_become_value_changes() call that makes me believe it doesn't respect the custom comparing function... I took a quick peek inside it and it seems to do exactly that. I think I need a break...
https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L1551

[seperman]

Hi @jvacek @Omar-Abdul-Azeez @havardthom @LizardBlizzard
Somehow this ticket was lost among other tickets I was paying attention to.
@Omar-Abdul-Azeez Thank you for diving into it already. I am going to take a look soon.

[pascalstindt]

I think I'm running into the same issue. I'm using a custom iterable_compare_func function to match on id.
All the results BUT ONE are correctly reported either as iterable_item_added or iterable_item_removed. The single incorrect one is reported as values_changed with clearly a non matching id.

@seperman and @Omar-Abdul-Azeez have you been able to fix this? Thanks a lot.

[amyasnikov]

Running into the same issue even on the latest (7.0.1) version.
@seperman it would be very nice if you could get back to this issue

[cdiproperzio1]

Having the same issue. Thanks!

[bongi23]

Similar issue here. I am comparing dicts with a field that contains list of dicts. Dicts in these lists have an "_id" field that I would like to use to compare items.

Example

obj_1 = {
    "name": "Andrea",
    "addresses": [
        "_id": 1
       "AddressName": "name1"
    ]
}

obj_2 = {
    "name": "Andrea",
    "addresses": [
        "_id": 1
       "AddressName": "name2"
    ]
}

I would like to get a value_change event for root[addresses][0][AddressName] but the comparator that returns True for a match on the _id filed does not get called.

I have both ignore_order and ignore_repetition set to True

[dtorres-sf]

This issue can now be closed. This is fixed in deepdiff 8+, I believe with this PR:

#473

Tested the following:

        self_json = b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"332e5cfe-886d-41cc-b4d9-0fc1296b3ea0","name":"field_2","address":5,"documentation":"","first_bit_offset":5,"size_in_bits":6,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.0","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'
        other_json = b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"056429c5-812b-4f49-9aae-1b52bd40aacd","name":"field_3","address":7,"documentation":"","first_bit_offset":8,"size_in_bits":9,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.1","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'

        def field_diff_function(x, y, level=None):
            try:
                return x["uuid"] == y["uuid"]

            except Exception:
                raise CannotCompare() from None
        diff_native = DeepDiff(
            json.loads(self_json),
            json.loads(other_json),
            iterable_compare_func=field_diff_function,
            report_repetition=True,
            ignore_order=True,
        )
        print(json.dumps(diff_native, indent=2))

Got this result:

{
  "values_changed": {
    "root['version']": {
      "new_value": "0.0.1",
      "old_value": "0.0.0"
    }
  },
  "iterable_item_added": {
    "root['numberfield_set'][1]": {
      "uuid": "056429c5-812b-4f49-9aae-1b52bd40aacd",
      "name": "field_3",
      "address": 7,
      "documentation": "",
      "first_bit_offset": 8,
      "size_in_bits": 9,
      "is_signed": false,
      "is_lsb_left": false,
      "value_multiply_by": 1.0,
      "value_divide_by": 1.0,
      "value_increase_by": 0.0,
      "value_unit": ""
    }
  },
  "iterable_item_removed": {
    "root['numberfield_set'][1]": {
      "uuid": "332e5cfe-886d-41cc-b4d9-0fc1296b3ea0",
      "name": "field_2",
      "address": 5,
      "documentation": "",
      "first_bit_offset": 5,
      "size_in_bits": 6,
      "is_signed": false,
      "is_lsb_left": false,
      "value_multiply_by": 1.0,
      "value_divide_by": 1.0,
      "value_increase_by": 0.0,
      "value_unit": ""
    }
  }
}

[Omar-Abdul-Azeez]

while the example in the original post may be fixed (I haven't tested), the one in my comment isn't. reposting for ease.

Irrelevent of ignore_order. The result of the custom compare function is ignored for nested dicts
I've tested this on a list of dicts, inside each of these dicts is a key whose value is also a list of dicts. each dict has an id key. The example below sets ignore_order=True but it works for both cases, just make sure the order is correct.

[
{'id': '1010', 'g': [ {'id': '2020'} ] },
{'id': '73', 'g': [ {'id': '101'}, {'id': '6790'} ] }
]
[
{'id': '73', 'g': [ {'id': '202'},  {'id': '15294'} ] },
{'id': '1012', 'g': [ {'id': '2020'} ] }
]

I use this as compare function. Except the prints, taken straight from the docs.

def compare_func(x, y, level=None):
    print(x)
    print(y)
    print('in')
    if not isinstance(x, dict) or not isinstance(y, dict):
        print('not dict')
        raise CannotCompare
    if x['id'] == y['id']:
        print('match')
        return True
    print('not match')
    return False

actually checking this example now yields no prints(???) and the result being value changes for root[0] (id:1010 with id:73) and for root[1] (id:73 with id:1012) which is clearly wrong.

and it is still due to this function not respecting the custom comparison.
https://github.com/seperman/deepdiff/blob/ecc823a3fdb9c094b7392ca40f859eab8cc8572a/deepdiff/diff.py#L1754

before the function is called there are additions for id: 73, 1012 and removals for id: 1010, and 73. I'm not sure if 73 should be paired beforehand or this function pairs them and finds out that the change is in a nested dict inside rather than the top level so I can't tell if it will even find out the difference correctly or not.