[Route bulk] Fix bugs in case a SET operation follows a DEL operation in the same bulk

[stephenxs]

What I did

Fix 2 issues in route bulk operation in the scenario where a SET operation follows a DEL operation on the same prefix.

1. The route entry SET operation can be ignored if it follows a DEL operation with the same prefix and next-hop.
2. DEL and then SET default route results in orchagent aborted

Added test case to cover the scenarios

Signed-off-by: Stephen Sun stephens@nvidia.com

Why I did it

How I verified it

Manually test
Unit test

Details if related
The route entry SET operation can be ignored if it follows a DEL operation with the same prefix and next-hop.
For example, the following operations

2021-12-18.06:17:54.793234|ROUTE_TABLE:1.1.1.0/24|SET|nexthop:10.0.0.33|ifname:Ethernet64
2021-12-18.06:19:53.187294|ROUTE_TABLE:1.1.1.0/24|DEL
2021-12-18.06:20:10.279850|ROUTE_TABLE:1.1.1.0/24|SET|nexthop:10.0.0.33|ifname:Ethernet64

result in the next sai operations

2021-12-18.06:19:31.203092|C|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"1.1.1.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000010"}|SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID=oid:0x4000000000aae
2021-12-18.06:20:23.813475|R|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"1.1.1.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000010"}

and the last SET is lost.

This is because:
Both DEL and the 2nd SET operations were bundled into bulk operations.
There is a logic that is not to program ASIC if the next-hop information in the route entry update isn't changed,
which is done by comparing the next-hop information between route entry update and m_syncdRoutes.
However, in bulk mode, a route entry that will be removed will not be removed from m_syncdRoutes until the bulk is flushed,
The bulk has not been flushed when the 2nd SET is handled,
which means the comparison returns "not changed" and the 2nd SET operation is ignored.

The fix is to check whether the prefix is in gRouteBulker.removing_entries and treat it as non-exist if yes.

DEL and then SET default route results in orchagent aborted

This is because:

1. packet forward action is initialized when route entry is created
2. the default route will be set to DROP if the application asks for removing it
3. if the default route is readded later in the same bulk session,
   it doesn't update packet forward action but provides a valid next-hop
4. SAI complains if both DROP and a valid next hop are provided

Solution:
Always set packet action for the default route

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[qiluo-msft]

@shi-su Could you help review?

[stephenxs]

@shi-su @qiluo-msft can we merge this pr if no other concern? We need it to fix azure pipeline checker failures. Thanks

[qiluo-msft]

Do we need to backport this PR to old branches?

[stephenxs]

Do we need to backport this PR to old branches?

I suggest doing so as it is a production issue. We need #2071 first