[Dualtor-AA] Bug: unexpected syslog errors caused by the test behavior in test_crm_nexthop_group

[congh-nvidia]

Is it platform specific

generic

Importance or Severity

Medium

Description of the bug

Syslog errors are observed in test case crm.test_crm.test_crm_nexthop_group on dualtor-aa testbed:

'2025 Aug 29 12:06:28.831329 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.3.132/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
'2025 Aug 29 12:06:28.831555 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.3.120/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
'2025 Aug 29 12:06:28.831632 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.3.141/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
...
'2025 Aug 29 12:06:28.853451 r-4700-72 ERR syncd#SDK: [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.'
'2025 Aug 29 12:06:28.929416 r-4700-72 ERR syncd#SDK: message repeated 9 times: [ [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.]'
'2025 Aug 29 12:06:28.929447 r-4700-72 ERR syncd#SDK: :- sendApiResponse: api SAI_COMMON_API_BULK_REMOVE failed in syncd mode: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:28.932570 r-4700-72 ERR swss#orchagent: :- flush_removing_entries: EntityBulker.flush remove entries failed
number of entries to remove: 336
status: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:28.979623 r-4700-72 ERR swss#orchagent: :- removeRoutePost: Failed to remove route prefix:2.0.3.117/32'
'2025 Aug 29 12:06:28.979852 r-4700-72 ERR swss#orchagent: :- removeRoutePost: Failed to remove route prefix:2.0.3.119/32'
...
'2025 Aug 29 12:06:30.894467 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.5.77/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
'2025 Aug 29 12:06:30.894467 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.5.61/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
'2025 Aug 29 12:06:30.894467 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.5.59/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
...
'2025 Aug 29 12:06:30.909964 r-4700-72 ERR syncd#SDK: [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.'
'2025 Aug 29 12:06:31.044167 r-4700-72 ERR syncd#SDK: message repeated 106 times: [ [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.]'
'2025 Aug 29 12:06:31.044167 r-4700-72 ERR syncd#SDK: :- sendApiResponse: api SAI_COMMON_API_BULK_REMOVE failed in syncd mode: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:31.048911 r-4700-72 ERR swss#orchagent: :- flush_removing_entries: EntityBulker.flush remove entries failed
number of entries to remove: 516
status: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:31.070236 r-4700-72 ERR swss#orchagent: :- removeRoutePost: Failed to remove route prefix:2.0.3.172/32'
'2025 Aug 29 12:06:31.070582 r-4700-72 ERR swss#orchagent: :- removeRoutePost: Failed to remove route prefix:2.0.3.173/32'
...
'2025 Aug 29 12:06:33.265114 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.5.207/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
'2025 Aug 29 12:06:33.265123 r-4700-72 ERR swss#orchagent: :- meta_generic_validation_remove: object key SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"2.0.5.204/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"} doesn\'t exist'
...
'2025 Aug 29 12:06:33.338084 r-4700-72 ERR syncd#SDK: [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.'
'2025 Aug 29 12:06:33.363171 r-4700-72 ERR syncd#SDK: message repeated 8 times: [ [SAI_ROUTE.ERR] ./src/mlnx_sai_route.c[1113]- mlnx_get_route: Failed to get 1 route entries Entry Not Found.]'
'2025 Aug 29 12:06:33.363171 r-4700-72 ERR syncd#SDK: :- sendApiResponse: api SAI_COMMON_API_BULK_REMOVE failed in syncd mode: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:33.366070 r-4700-72 ERR swss#orchagent: :- flush_removing_entries: EntityBulker.flush remove entries failed
number of entries to remove: 412
status: SAI_STATUS_FAILURE'
'2025 Aug 29 12:06:33.370268 r-4700-72 ERR swss#orchagent: :- removeRoutePost: Failed to remove route prefix:2.0.5.174/32'
'...

From @Ndancejic 's analysis, the root casue is some of these route entries are removed in the cleanup step when the corresponding neighbor entries are removed. The way this test currently works is that it creates a route prefix ex: 10.0.0.3/32 which points to a next hop group {10.0.0.1, 10.0.0.3}. This works fine for testing CRM but it’s not an expected use-case for Sonic. In dualtor, when a neighbor entry is not resolved, we also create a tunnel route w/prefix 10.0.0.3/32. What’s happening in this test case is that some of these routes are being created and the test is finishing before the corresponding neighbor entries are created. Then during the cleanup step when the neighbor entries are being removed, dualtor logic is removing the prefix (which happens to correspond to the route entry as well). So when route cleanup is happening, we attempt to remove that prefix but it does not exist.

In the future, likely 202511, we’ll be switching to no_host_route handling of dualtor neighbors: sonic-net/sonic-swss#3722. After this, the errors should not be observed anymore.

Steps to Reproduce

Run test crm.test_crm.test_crm_nexthop_group on a SN4700 dualtor-aa-64-breakout testbed.

Actual Behavior and Expected Behavior

No errors.

Relevant log output

Output of show version

Attach files (if any)

No response

[lolyu]

Hi @congh-nvidia, orchagent/syncd will exit in such case, right? If so, this will leave the testbed in an unhealthy state, so all subsequent cases crm/test_crm.py::test_acl_entry will fail in such case:

crm/test_crm.py::test_crm_nexthop_group[str3-msn4700-12-None-False-2.2.2.0/24] ERROR [ 63%]
crm/test_crm.py::test_crm_nexthop_group[str3-msn4700-12-None-True-2.2.2.0/24] FAILED [ 72%]
crm/test_crm.py::test_crm_nexthop_group[str3-msn4700-12-None-True-2.2.2.0/24] ERROR [ 72%]
crm/test_crm.py::test_acl_entry[str3-msn4700-12-None] FAILED             [ 81%]
crm/test_crm.py::test_acl_entry[str3-msn4700-12-None] ERROR              [ 81%]

[congh-nvidia]

Hi @lolyu ,
From our test, xfail is enough, the failure doesn't affect the following test cases.
But currently we have another test issue in test_crm_nexthop, the failure of this one will cause other test cases to fail. Do you have failure on this test case?

This is our result in 202505, test_crm_nexthop is skipped:

INFO     : RunPytest  - All cases: [ 
    { 

        "name": "crm/test_crm.py::test_crm_route[r-4700-72-None-ipv4]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_route[r-4700-72-None-ipv6]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_nexthop[r-4700-72-None-4-2.2.2.2]", 
        "result": "skipped" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_nexthop[r-4700-72-None-6-2001::1]", 
        "result": "skipped" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_neighbor[r-4700-72-None-4-2.2.2.2-2.2.2.1/8]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_neighbor[r-4700-72-None-6-2001::1-2001::2/64]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_nexthop_group[r-4700-72-None-False-2.2.2.0/24]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_nexthop_group[r-4700-72-None-True-2.2.2.0/24]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_acl_entry[r-4700-72-None]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_acl_counter[r-4700-72-None]", 
        "result": "passed" 
    }, 

    { 

        "name": "crm/test_crm.py::test_crm_fdb_entry[r-4700-72-None]", 
        "result": "skipped" 
    } 

] 

[yxieca]

@lolyu @congh-nvidia the image fix is in place, my understanding is that the crash is no longer there but we have to deal with error logs? If so, can we drop the skip/xfail PR and add the whitelist instead?

[congh-nvidia]

Hi @lolyu , could you please help clarify how we should handle the errors? Xfail the test as we previously discussed or ignore the errors in the common_ignore file or ignore in the test?