[DPB] Fix a potential command failure when break out a port that is a member of portchannel.

[sandyw777]

What I did

Fix issue: sonic-net/sonic-buildimage#10005 where DPB may not be able to break out ports when these ports are port-channel member. Upon analyzing the output obtained via “redis-cli monitor” command, we found the Consumer cannot process DEL port request in time before executing HSET admin_status and mtu operation from teammgr removeLagMember function.
This sequence mismatch led to the failure in port deletion, causing Orchagent to miss the DELETE message.

How I did it

The _deletePorts function was split into two parts:
Part1: Deletion of dependencies
Part2: Deletion of port itself
Additionally an 1 second delay was added between Part1 and Part2 to ensure the serialization of ConfigDB update.

Example:

• Original deleted config:
  ConfigMgmt: Write in DB: {'PORT': {'Ethernet0': None}, 'PORTCHANNEL_MEMBER': None}

• Separate the deleted config into two parts (solution):

part1). 'delete dependencies'(delConfigDepToLoad):
ConfigMgmt: Write in DB: {'PORTCHANNEL_MEMBER': None}

sleep 1 sec...

part2). 'delete port'(delConfigToLoad):
ConfigMgmt: Write in DB: {'PORT': {'Ethernet0': None}}

How to verify it

Step1. Create portchannel with Ethernet0

Step2. Dynamic port breakout:
admin@sonic:~$ sudo config interface breakout Ethernet0 4x10G[10G,1G] -fy

Step3. Check console:
admin@sonic:~$ sudo config interface breakout Ethernet0 4x10G[10G,1G] -fy

Running Breakout Mode : 1x40G[40G,10G,1G]
Target Breakout Mode : 4x10G[10G,1G]

Ports to be deleted :
{
"Ethernet0": "40000"
}
Ports to be added :
{
"Ethernet0": "10000",
"Ethernet1": "10000",
"Ethernet2": "10000",
"Ethernet3": "10000"
}
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run config save.

Step4. Check interface:

admin@sonic:~$ show interface status
  Interface            Lanes    Speed    MTU    FEC           Alias    Vlan    Oper    Admin    Type    Asym PFC
-----------  ---------------  -------  -----  -----  --------------  ------  ------  -------  ------  ----------
  Ethernet0               25      10G   9100    N/A          etp1-1  routed    down     down     N/A         N/A
  Ethernet1               26      10G   9100    N/A          etp1-2  routed    down     down     N/A         N/A
  Ethernet2               27      10G   9100    N/A          etp1-3  routed    down     down     N/A         N/A
  Ethernet3               28      10G   9100    N/A          etp1-4  routed    down     down     N/A         N/A
  Ethernet4      29,30,31,32      40G   9100    N/A    fortyGigE0/4  routed    down     down     N/A         N/A

Step5. Check syslog:
"Deleting Port Ethernet0" can be found. Orchagent received the DELETE message.

Step6. Check redis-cli monitor:
HSET admin_status/mtu and DEL port by Consumer are Separated and DEL Ethernet0 success.

Producer:
1703584555.434850 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "mtu" "9100"
1703584555.434862 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.434877 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"
1703584555.434889 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"

Write in DB: {'PORTCHANNEL_MEMBER': None}
1703584555.436248 [4 127.0.0.1:33054] "DEL" "PORTCHANNEL_MEMBER|PortChannel01|Ethernet0"

Producer:
1703584555.439558 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "3" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down"
1703584555.439577 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.439584 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"

Producer:
1703584555.471847 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "6875900592cdd1621c6191fe038ec3b29775aa13" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "_PORT_TABLE:Ethernet0" "G" "Ethernet0" "admin_status" "down" "mtu" "9100"
1703584555.471914 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584555.471920 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "admin_status" "down"
1703584555.471937 [0 lua] "HSET" "_PORT_TABLE:Ethernet0" "mtu" "9100"

Consumer:
1703584556.044427 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1703584556.044438 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1703584556.044574 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.044606 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1703584556.044614 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "mtu" "9100"
1703584556.044631 [0 lua] "HSET" "PORT_TABLE:Ethernet0" "admin_status" "down"
1703584556.044645 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"

Write in DB: {'PORT': {'Ethernet0': None}}
1703584556.439136 [4 127.0.0.1:33066] "DEL" "PORT|Ethernet0"

Producer:
1703584556.440402 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88ba6312b8de850b3506966425174d8899aadd93" "4" "PORT_TABLE_CHANNEL@0" "PORT_TABLE_KEY_SET" "_PORT_TABLE:Ethernet0" "PORT_TABLE_DEL_SET" "G" "Ethernet0" "''" "''"
1703584556.440419 [0 lua] "SADD" "PORT_TABLE_KEY_SET" "Ethernet0"
1703584556.440435 [0 lua] "SADD" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.440449 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"
1703584556.440453 [0 lua] "PUBLISH" "PORT_TABLE_CHANNEL@0" "G"

Consumer:
1703584556.869997 [0 unix:/var/run/redis/redis.sock] "EVALSHA" "88270a7c5c90583e56425aca8af8a4b8c39fe757" "3" "PORT_TABLE_KEY_SET" "PORT_TABLE:" "PORT_TABLE_DEL_SET" "1024" "_"
1703584556.870010 [0 lua] "SPOP" "PORT_TABLE_KEY_SET" "1024"
1703584556.870060 [0 lua] "SREM" "PORT_TABLE_DEL_SET" "Ethernet0"
1703584556.870251 [0 lua] "DEL" "PORT_TABLE:Ethernet0"
1703584556.870269 [0 lua] "HGETALL" "_PORT_TABLE:Ethernet0"
1703584556.870274 [0 lua] "DEL" "_PORT_TABLE:Ethernet0"

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

[sandyw777]

@alexrallen @dgsudharsan @praveen-li @zhenggen-xu
Could you help to review the commit? Thank you!

[puffc]

@alexrallen @dgsudharsan @praveen-li @zhenggen-xu Please review and merge this pr. thanks.