Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

orchagent crashed when trying to delete fdb static entry with swssconfig #11046

Closed
ZhaohuiS opened this issue Jun 6, 2022 · 5 comments
Closed

orchagent crashed when trying to delete fdb static entry with swssconfig #11046

ZhaohuiS opened this issue Jun 6, 2022 · 5 comments
Labels
BRCM Triaged this issue has been triaged

Comments

@ZhaohuiS
Copy link
Contributor

ZhaohuiS commented Jun 6, 2022 •
edited

Description

orchagent crashed when trying to delete fdb static entry with swssconfig,
the fdb static entry was added with swssconfig minutes ago.

Steps to reproduce the issue:

  1. Run test script tests/drop_counters/test_configurable_drop_counters.py in sonic-mgmt repo.
  2. In script, it adds a static fdb entry by command docker exec -i swss swssconfig /fdb_set_config.json
  3. clear fdp and arp
    sonic-clear fdb all
    sonic-clear arp
  4. Remove the static fdb entry by the same command but with different json file, docker exec -i swss swssconfig /fdb_del_config.json
  5. orchagent crashed and can't be restarted. Attach core dump file.
  6. It does not always happen; the reproduced rate may be less than 20%.

Describe the results you received:

Error log:

Jun  6 08:09:37.858764 str2-dx010-acs-7 NOTICE swss#orchagent: :- flush: performing log rotate
Jun  6 08:09:49.047751 str2-dx010-acs-7 NOTICE swss#orchagent: :- flushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:09:49.048113 str2-dx010-acs-7 NOTICE swss#orchagent: :- recordFlushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:09:49.562663 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: processing consolidated fdb flush event of type: SAI_FDB_ENTRY_TYPE_DYNAMIC
Jun  6 08:09:49.567363 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: fdb flush took 0.004668 sec
Jun  6 08:09:51.959521 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel101:10.0.0.57 after SET operation
Jun  6 08:09:52.008731 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel102:10.0.0.59 after SET operation
Jun  6 08:09:52.032171 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel103:10.0.0.61 after SET operation
Jun  6 08:09:52.063777 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel104:10.0.0.63 after SET operation
Jun  6 08:09:52.370895 str2-dx010-acs-7 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.1.156, 72:06:00:01:00:41 on Vlan1000
Jun  6 08:09:52.373452 str2-dx010-acs-7 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.1.156 on Vlan1000
Jun  6 08:10:11.467532 str2-dx010-acs-7 NOTICE swss#orchagent: :- addDropReason: Added drop reason L3_EGRESS_LINK_DOWN to drop counter TEST
Jun  6 08:10:11.524798 str2-dx010-acs-7 NOTICE swss#orchagent: :- reconcileFreeDropCounters: Successfully matched drop reasons to counter TEST
Jun  6 08:10:11.524798 str2-dx010-acs-7 NOTICE swss#orchagent: :- installDebugCounter: Successfully created drop counter TEST
Jun  6 08:10:32.604374 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Get port state change notification id:1000000000002 status:2
Jun  6 08:10:32.604886 str2-dx010-acs-7 NOTICE swss#orchagent: :- updatePortOperStatus: Port Ethernet80 oper state set from up to down
Jun  6 08:10:32.610286 str2-dx010-acs-7 NOTICE swss#orchagent: :- setHostIntfsOperStatus: Set operation status DOWN to host interface Ethernet80
Jun  6 08:10:32.611353 str2-dx010-acs-7 NOTICE swss#orchagent: :- flushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 1
Jun  6 08:10:32.611550 str2-dx010-acs-7 NOTICE swss#orchagent: :- recordFlushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 1
Jun  6 08:10:32.650856 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: processing consolidated fdb flush event of type: SAI_FDB_ENTRY_TYPE_DYNAMIC
Jun  6 08:10:32.655723 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: fdb flush took 0.004869 sec
Jun  6 08:10:56.538915 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Get port state change notification id:1000000000002 status:1
Jun  6 08:10:56.538915 str2-dx010-acs-7 NOTICE swss#orchagent: :- updatePortOperStatus: Port Ethernet80 oper state set from down to up
Jun  6 08:10:56.546636 str2-dx010-acs-7 NOTICE swss#orchagent: :- setHostIntfsOperStatus: Set operation status UP to host interface Ethernet80
Jun  6 08:10:56.551325 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Ethernet80 oper speed is 100000
Jun  6 08:10:58.341149 str2-dx010-acs-7 NOTICE swss#orchagent: :- flushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:10:58.341746 str2-dx010-acs-7 NOTICE swss#orchagent: :- recordFlushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:10:58.833820 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: processing consolidated fdb flush event of type: SAI_FDB_ENTRY_TYPE_DYNAMIC
Jun  6 08:10:58.838618 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: fdb flush took 0.004757 sec
Jun  6 08:10:58.852598 str2-dx010-acs-7 ERR swss#orchagent: :- meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"72:06:00:01:00:41","switch_id":"oid:0x21000000000000"} already exists
Jun  6 08:10:58.852968 str2-dx010-acs-7 ERR swss#orchagent: :- update: Failed to create FDB 72:06:00:01:00:41 on Ethernet80, rv:-6
Jun  6 08:10:58.853581 str2-dx010-acs-7 WARNING swss#orchagent: :- meta_sai_on_fdb_event_single: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"DE:AD:BE:EF:12:34","switch_id":"oid:0x21000000000000"} doesn't exist but received AGED event
Jun  6 08:11:00.546160 str2-dx010-acs-7 NOTICE swss#orchagent: :- removeNeighbor: Removed next hop 192.168.1.156 on Vlan1000
Jun  6 08:11:00.549150 str2-dx010-acs-7 NOTICE swss#orchagent: :- removeNeighbor: Removed neighbor 72:06:00:01:00:41 on Vlan1000
Jun  6 08:11:00.961519 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel101:10.0.0.57 after SET operation
Jun  6 08:11:01.008672 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel102:10.0.0.59 after SET operation
Jun  6 08:11:01.033527 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel103:10.0.0.61 after SET operation
Jun  6 08:11:01.047291 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Removed pending neighbor DEL operation for PortChannel104:10.0.0.63 after SET operation
Jun  6 08:11:04.970882 str2-dx010-acs-7 INFO python[863326]: ansible-command Invoked with _raw_params=docker exec -i swss swssconfig /fdb.json warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun  6 08:11:05.163633 str2-dx010-acs-7 NOTICE swss#swssconfig: :- main: Loading config from JSON file:/fdb.json...
Jun  6 08:11:05.213229 str2-dx010-acs-7 ERR swss#orchagent: :- meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"72:06:00:01:00:41","switch_id":"oid:0x21000000000000"} doesn't exist
Jun  6 08:11:05.213504 str2-dx010-acs-7 ERR swss#orchagent: :- removeFdbEntry: FdbOrch RemoveFDBEntry: Failed to remove FDB entry. mac=72:06:00:01:00:41, bv_id=0x260000000005c5
Jun  6 08:11:05.213730 str2-dx010-acs-7 ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_FDB, status: SAI_STATUS_INVALID_PARAMETER
Jun  6 08:11:07.610329 str2-dx010-acs-7 INFO swss#supervisord 2022-06-06 08:11:07,609 INFO exited: orchagent (terminated by SIGABRT (core dumped); not expected)
Jun  6 08:11:08.627098 str2-dx010-acs-7 INFO swss#supervisor-proc-exit-listener: Process 'orchagent' exited unexpectedly. Terminating supervisor 'swss

The output of docker ps:

admin@str2-dx010-acs-7:/var/core$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED        STATUS        PORTS     NAMES
91eeb716cf98   docker-sonic-telemetry:latest        "/usr/local/bin/supe…"   20 hours ago   Up 9 hours              telemetry
b5112f492674   docker-snmp:latest                   "/usr/local/bin/supe…"   20 hours ago   Up 9 hours              snmp
05ddbed1b883   docker-sonic-mgmt-framework:latest   "/usr/local/bin/supe…"   20 hours ago   Up 9 hours              mgmt-framework
336362fc3031   docker-platform-monitor:latest       "/usr/bin/docker_ini…"   21 hours ago   Up 9 hours              pmon
5f3c49dcabe9   docker-lldp:latest                   "/usr/bin/docker-lld…"   21 hours ago   Up 9 hours              lldp
4323ae1f03f9   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   21 hours ago   Up 9 hours              bgp
2b7e5e5f31d3   docker-database:latest               "/usr/local/bin/dock…"   21 hours ago   Up 10 hours             database

Describe the results you expected:

orchagent should not crash.

Output of show version:

admin@str2-dx010-acs-7:/var/core$ show version

SONiC Software Version: SONiC.master.105616-0552d6b17
Distribution: Debian 11.3
Kernel: 5.10.0-12-2-amd64
Build commit: 0552d6b17
Build date: Thu Jun  2 21:35:30 UTC 2022
Built by: AzDevOps@sonic-build-workers-001KKS

Platform: x86_64-cel_seastone-r0
HwSKU: Celestica-DX010-C32
ASIC: broadcom
ASIC Count: 1
Serial Number: N/A
Model Number: N/A
Hardware Revision: N/A
Uptime: 08:49:07 up  9:57,  1 user,  load average: 0.56, 0.43, 0.56
Date: Mon 06 Jun 2022 08:49:07

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-syncd-brcm             latest                    96ed6fea1e65   749MB
docker-syncd-brcm             master.105616-0552d6b17   96ed6fea1e65   749MB
docker-gbsyncd-credo          latest                    53af31f169c1   425MB
docker-gbsyncd-credo          master.105616-0552d6b17   53af31f169c1   425MB
docker-orchagent              latest                    f03ada49688e   471MB
docker-orchagent              master.105616-0552d6b17   f03ada49688e   471MB
docker-macsec                 latest                    099f866cb228   455MB
docker-dhcp-relay             latest                    b3fe92cbbec2   446MB
docker-sonic-telemetry        latest                    241c3c0af082   517MB
docker-sonic-telemetry        master.105616-0552d6b17   241c3c0af082   517MB
docker-platform-monitor       latest                    ea3fd8f00c89   559MB
docker-platform-monitor       master.105616-0552d6b17   ea3fd8f00c89   559MB
docker-router-advertiser      latest                    91799256bd57   437MB
docker-router-advertiser      master.105616-0552d6b17   91799256bd57   437MB
docker-mux                    latest                    7f2d27d123b7   486MB
docker-mux                    master.105616-0552d6b17   7f2d27d123b7   486MB
docker-database               latest                    d605b5000a4a   437MB
docker-database               master.105616-0552d6b17   d605b5000a4a   437MB
docker-teamd                  latest                    39d5f511896c   422MB
docker-teamd                  master.105616-0552d6b17   39d5f511896c   422MB
docker-nat                    latest                    1944d57899b1   425MB
docker-nat                    master.105616-0552d6b17   1944d57899b1   425MB
docker-sflow                  latest                    37ab4a110aeb   423MB
docker-sflow                  master.105616-0552d6b17   37ab4a110aeb   423MB
docker-fpm-frr                latest                    723b0e177c00   451MB
docker-fpm-frr                master.105616-0552d6b17   723b0e177c00   451MB
docker-snmp                   latest                    0d4875f1ccce   451MB
docker-snmp                   master.105616-0552d6b17   0d4875f1ccce   451MB
docker-sonic-mgmt-framework   latest                    58f48d9da899   551MB
docker-sonic-mgmt-framework   master.105616-0552d6b17   58f48d9da899   551MB
docker-lldp                   latest                    bc9f6ce8adff   448MB
docker-lldp                   master.105616-0552d6b17   bc9f6ce8adff   448MB

Output of show techsupport:

show_tech.log

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

orchagent.1654503065.105.core.gz

fdb_set_config.txt

fdb_del_config.txt
test_configurable_drop_counters-0606-dx010-7-1.log
@ZhaohuiS
Copy link
Contributor Author

ZhaohuiS commented Jun 7, 2022

After checking log file test_configurable_drop_counters-0606-dx010-7-1.log, analysis:
Test steps:

Steps:

  1. start arp_responder
  2. sonic-clear fdb all
  3. sonic-clear arp
  4. ping 192.168.1.156 -c 3
  5. config dropcounters install TEST PORT_INGRESS_DROPS L3_EGRESS_LINK_DOWN
  6. docker exec -i swss swssconfig /fdb_set.json
  7. fanout, interface Ethernet21/1,shutdown
  8. show arp 192.168.1.156
  9. ip neigh show 192.168.1.156
  10. show mac | grep 72:06:00:01:00:41 | grep Ethernet80 | wc -l
  11. sonic-clear dropcounters
  12. Sending traffic from ptf on port 11
  13. show dropcounters counts -t PORT_INGRESS_DROPS
  14. fanout, interface Ethernet21/1, no shutdown
  15. sonic-clear fdb all
  16. sonic-clear arp
  17. docker exec -i swss swssconfig /fdb_del.json

Analysis:

  1. docker exec -i swss swssconfig /fdb_set.json FDB add successfully. FDB Add-[Ok]: MAC:72-06-00-01-00-41 vfi:0x3e8
Jun  6 08:10:16.791549 str2-dx010-acs-7 INFO python[861809]: ansible-command Invoked with _raw_params=docker exec -i swss swssconfig /fdb.json warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun  6 08:10:16.972960 str2-dx010-acs-7 NOTICE swss#swssconfig: :- main: Loading config from JSON file:/fdb.json...
Jun  6 08:10:16.988247 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_set_fdb_entry_attribute:1130 FDB MAC:72-06-00-01-00-41 port_tid:0x1 tgid:0 port_type:Port vlan_id:1000 vid:0x3e8
Jun  6 08:10:16.988247 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_set_fdb_entry_attribute:1140 Setting Type SAI_FDB_ENTRY_TYPE_STATIC:
Jun  6 08:10:16.988247 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_bcm_l2_addr_add:154 FDB Add-[Ok]: MAC:72-06-00-01-00-41 vfi:0x3e8
Jun  6 08:10:16.992527 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_set_fdb_entry_attribute:1130 FDB MAC:72-06-00-01-00-41 port_tid:0x1 tgid:0 port_type:Port vlan_id:1000 vid:0x3e8
Jun  6 08:10:16.992818 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_set_fdb_entry_attribute:1251 L2 port: 1
Jun  6 08:10:16.993032 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_bcm_l2_addr_add:154 FDB Add-[Ok]: MAC:72-06-00-01-00-41 vfi:0x3e8
  1. shutdown fanout interface, delete 72-06-00-01-00-41 fdb. fdbEvent: delete (0) for mac 72-06-00-01-00-41 vid:0x3e8
Jun  6 08:10:32.599375 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_PORT:_brcm_sai_link_event_cb:1150 Port 1 link down event cause: LOCAL
Jun  6 08:10:32.604374 str2-dx010-acs-7 NOTICE swss#orchagent: :- doTask: Get port state change notification id:1000000000002 status:2
Jun  6 08:10:32.604886 str2-dx010-acs-7 NOTICE swss#orchagent: :- updatePortOperStatus: Port Ethernet80 oper state set from up to down
Jun  6 08:10:32.610286 str2-dx010-acs-7 NOTICE swss#orchagent: :- setHostIntfsOperStatus: Set operation status DOWN to host interface Ethernet80
Jun  6 08:10:32.611092 str2-dx010-acs-7 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet80 admin:1 oper:0 addr:00:e0:ec:c2:af:7a ifindex:95 master:91
Jun  6 08:10:32.611353 str2-dx010-acs-7 NOTICE swss#orchagent: :- flushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 1
Jun  6 08:10:32.611550 str2-dx010-acs-7 NOTICE swss#orchagent: :- recordFlushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 1
Jun  6 08:10:32.612827 str2-dx010-acs-7 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet80(ok:down) to state db
Jun  6 08:10:32.613094 str2-dx010-acs-7 NOTICE swss#portsyncd: :- onMsg: nlmsg type:16 key:Ethernet80 admin:1 oper:0 addr:00:e0:ec:c2:af:7a ifindex:95 master:91
Jun  6 08:10:32.613379 str2-dx010-acs-7 NOTICE swss#portsyncd: :- onMsg: Publish Ethernet80(ok:down) to state db
Jun  6 08:10:32.613488 str2-dx010-acs-7 INFO kernel: [33556.638716] Bridge: port 20(Ethernet80) entered disabled state
Jun  6 08:10:32.615512 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFdbFlush: fdb flush succeeded, updating redis database
Jun  6 08:10:32.615512 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFlushEvent: received a flush port fdb event, portVid = oid:0x3a0000000005ea, bvId = oid:0x0
Jun  6 08:10:32.615512 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFlushEvent: pattern ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:*, portStr oid:0x3a0000000005ea
Jun  6 08:10:32.617644 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:216 fdbEvent: delete (0) for mac 72-06-00-01-00-41 vid:0x3e8, port:0x1 lagid:0x0 flags:0x10c60 flags2:0x0 lag:false station flags 0x0
  1. Send packet, add 72-06-00-01-00-41 fdb back. SAI_FDB_ENTRY_TYPE_DYNAMIC FDB Add-[Ok]: MAC:72-06-00-01-00-41 vfi:0x3e8
Jun  6 08:10:32.650856 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: processing consolidated fdb flush event of type: SAI_FDB_ENTRY_TYPE_DYNAMIC
Jun  6 08:10:32.655723 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: fdb flush took 0.004869 sec
Jun  6 08:10:32.658985 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_bcm_l2_addr_add:154 FDB Add-[Ok]: MAC:72-06-00-01-00-41 vfi:0x3e8
Jun  6 08:10:32.659099 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:114 Process fdb nbr update for [1000].[72:06:00:01:00:41]. add 0, dir 5
Jun  6 08:10:32.659175 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:114 Process fdb nbr update for [1000].[72:06:00:01:00:41]. add 1, dir 6
Jun  6 08:10:32.659252 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:360 brcm_sai_mptnl_notify_nhop_event success: 0, add 1 dir 6
Jun  6 08:10:32.659326 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:364 Updated egress object: 100036 flags : 0
Jun  6 08:10:32.659400 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:brcm_sai_create_fdb_entry:920 FDB Create: MAC:72-06-00-01-00-41 port:0x1 tid:0x0 port_type:Port vid:0x3e8
  1. Send packet, add DE-AD-BE-EF-12-34 fdb as well. fdbEvent: add (1) for mac DE-AD-BE-EF-12-34 vid:0x3e8
Jun  6 08:10:47.132892 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:216 fdbEvent: add (1) for mac DE-AD-BE-EF-12-34 vid:0x3e8, port:0x3e lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Jun  6 08:10:47.132987 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:105 Mac entry not found - skip nbr processing. add 1, dir 4
  1. sonic-clear fdb, delete fdb for both 72-06-00-01-00-41 and DE-AD-BE-EF-12-34. fdbEvent: delete (0) for mac 72-06-00-01-00-41 fdbEvent: delete (0) for mac DE-AD-BE-EF-12-34. Err reported. meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"72:06:00:01:00:41","switch_id":"oid:0x21000000000000"} already exists
Jun  6 08:10:57.241109 str2-dx010-acs-7 INFO python[862885]: ansible-command Invoked with _raw_params=sonic-clear fdb all warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun  6 08:10:58.341149 str2-dx010-acs-7 NOTICE swss#orchagent: :- flushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:10:58.341746 str2-dx010-acs-7 NOTICE swss#orchagent: :- recordFlushFdbEntries: flush key: SAI_OBJECT_TYPE_FDB_FLUSH:oid:0x21000000000000, fields: 0
Jun  6 08:10:58.347394 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:216 fdbEvent: delete (0) for mac 72-06-00-01-00-41 vid:0x3e8, port:0x1 lagid:0x0 flags:0x10020 flags2:0x0 lag:false station flags 0x0
Jun  6 08:10:58.347679 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:114 Process fdb nbr update for [1000].[72:06:00:01:00:41]. add 0, dir 4
Jun  6 08:10:58.347892 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:360 brcm_sai_mptnl_notify_nhop_event success: 0, add 0 dir 4
Jun  6 08:10:58.348070 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:364 Updated egress object: 100036 flags : 100000
Jun  6 08:10:58.356141 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:216 fdbEvent: delete (0) for mac DE-AD-BE-EF-12-34 vid:0x3e8, port:0x3e lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Jun  6 08:10:58.356386 str2-dx010-acs-7 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:105 Mac entry not found - skip nbr processing. add 0, dir 4
Jun  6 08:10:58.833311 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFdbFlush: fdb flush succeeded, updating redis database
Jun  6 08:10:58.833820 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: processing consolidated fdb flush event of type: SAI_FDB_ENTRY_TYPE_DYNAMIC
Jun  6 08:10:58.838618 str2-dx010-acs-7 NOTICE swss#orchagent: :- meta_sai_on_fdb_flush_event_consolidated: fdb flush took 0.004757 sec
Jun  6 08:10:58.838995 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFlushEvent: received a flush port fdb event, portVid = oid:0x0, bvId = oid:0x0
Jun  6 08:10:58.839254 str2-dx010-acs-7 NOTICE syncd#syncd: :- processFlushEvent: pattern ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:*, portStr 
Jun  6 08:10:58.852598 str2-dx010-acs-7 ERR swss#orchagent: :- meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"72:06:00:01:00:41","switch_id":"oid:0x21000000000000"} already exists
Jun  6 08:10:58.852968 str2-dx010-acs-7 ERR swss#orchagent: :- update: Failed to create FDB 72:06:00:01:00:41 on Ethernet80, rv:-6
Jun  6 08:10:58.853581 str2-dx010-acs-7 WARNING swss#orchagent: :- meta_sai_on_fdb_event_single: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"DE:AD:BE:EF:12:34","switch_id":"oid:0x21000000000000"} doesn't exist but received AGED event
  1. docker exec -i swss swssconfig /fdb_del.json to cause FdbOrch RemoveFDBEntry: Failed to remove FDB entry. mac=72:06:00:01:00:41 and orchagent crashed.
Jun  6 08:11:04.970882 str2-dx010-acs-7 INFO python[863326]: ansible-command Invoked with _raw_params=docker exec -i swss swssconfig /fdb.json warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun  6 08:11:05.163633 str2-dx010-acs-7 NOTICE swss#swssconfig: :- main: Loading config from JSON file:/fdb.json...
Jun  6 08:11:05.213229 str2-dx010-acs-7 ERR swss#orchagent: :- meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000005c5","mac":"72:06:00:01:00:41","switch_id":"oid:0x21000000000000"} doesn't exist
Jun  6 08:11:05.213504 str2-dx010-acs-7 ERR swss#orchagent: :- removeFdbEntry: FdbOrch RemoveFDBEntry: Failed to remove FDB entry. mac=72:06:00:01:00:41, bv_id=0x260000000005c5
Jun  6 08:11:05.213730 str2-dx010-acs-7 ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_FDB, status: SAI_STATUS_INVALID_PARAMETER
Jun  6 08:11:07.610329 str2-dx010-acs-7 INFO swss#supervisord 2022-06-06 08:11:07,609 INFO exited: orchagent (terminated by SIGABRT (core dumped); not expected)
Jun  6 08:11:08.627098 str2-dx010-acs-7 INFO swss#supervisor-proc-exit-listener: Process 'orchagent' exited unexpectedly. Terminating supervisor 'swss'

@zhangyanzhao
Copy link
Collaborator

BRCM will take a look.

@zhangyanzhao zhangyanzhao added Triaged this issue has been triaged BRCM labels Jun 8, 2022
@anilkpan
Copy link

FDB flush is removing static mac as well. Need to change flush call to remove only dynamic mac. Will make the change and create PR for review.

@anilkpan
Copy link

PR created for the fix:
sonic-net/sonic-swss#2332

This was referenced Jun 16, 2022
Draft
Closed
Merged
@yxieca yxieca mentioned this issue Jun 22, 2022
Merged
yxieca added a commit to yxieca/sonic-buildimage that referenced this issue Jun 23, 2022
@yxieca
[202205][swss][utilities][sairedis][pmon] advance submodule head
9988316 
swss:
* 43b9f06 2022-06-23 | [ACL] Support stage particular match fields (sonic-net#2341) (HEAD -> 202205) [bingwang-ms]
* c7f1add 2022-06-22 | Fix qos map test in vs test (sonic-net#2343) [bingwang-ms]
* 0549f3c 2022-06-22 | Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig sonic-net#11046" (sonic-net#2332) [anilkpan]

utilities:
* 785508d 2022-05-25 | [GCU] Handling type1 lists (sonic-net#2171) (HEAD -> 202205) [Mohamed Ghoneim]
* 56c2c6b 2022-06-22 | [config reload] Fixing config reload when timer based services are disabled (sonic-net#2200) (sonic-net#2226) (HEAD -> 202205, github/202205) [Sudharsan Dhamal Gopalarathnam]
* 600979e 2022-06-16 | Fix header for the output table following 'show ipv6 interface' command (sonic-net#2219) [Dror Prital]

sairedis:
* 14fba45 2022-06-13 | [vslib]: Fixbug in cleanup MACsec device (sonic-net#1059) (HEAD -> 202205) [Ze Gan]
* ac6e7ad 2022-04-29 | [vslib]: Update packet number of MACsec SA at runtime (sonic-net#1007) [Ze Gan]

platform-daemons:
* 6ffec22 2022-06-22 | [ycabled] Fix the init values for active-active ports (sonic-net#266) (HEAD -> 202205) [vdahiya12]

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
yxieca added a commit that referenced this issue Jun 23, 2022
@yxieca
[202205][swss][utilities][sairedis][pmon] advance submodule head (#11223
f6f87f1 
)

swss:
* 43b9f06 2022-06-23 | [ACL] Support stage particular match fields (#2341) (HEAD -> 202205) [bingwang-ms]
* c7f1add 2022-06-22 | Fix qos map test in vs test (#2343) [bingwang-ms]
* 0549f3c 2022-06-22 | Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig #11046" (#2332) [anilkpan]

utilities:
* 785508d 2022-05-25 | [GCU] Handling type1 lists (#2171) (HEAD -> 202205) [Mohamed Ghoneim]
* 56c2c6b 2022-06-22 | [config reload] Fixing config reload when timer based services are disabled (#2200) (#2226) (HEAD -> 202205, github/202205) [Sudharsan Dhamal Gopalarathnam]
* 600979e 2022-06-16 | Fix header for the output table following 'show ipv6 interface' command (#2219) [Dror Prital]

sairedis:
* 14fba45 2022-06-13 | [vslib]: Fixbug in cleanup MACsec device (#1059) (HEAD -> 202205) [Ze Gan]
* ac6e7ad 2022-04-29 | [vslib]: Update packet number of MACsec SA at runtime (#1007) [Ze Gan]

platform-daemons:
* 6ffec22 2022-06-22 | [ycabled] Fix the init values for active-active ports (#266) (HEAD -> 202205) [vdahiya12]

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
vivekrnv added a commit to vivekrnv/sonic-buildimage that referenced this issue Jun 24, 2022
@vivekrnv
[submodule]update sonic-swss submodule
be4d79b 
93af69c [PFC_WD] Avoid applying ZeroBuffer Profiles to ingress PG when a PFC storm is detected (sonic-net#2304)
37349cf [swssconfig] Optimize performance of swssconfig (sonic-net#2336)
84e9b07 [fdborch] fix heap-use-after-free in clearFdbEntry() (sonic-net#2353)
1b8bd94 Create ACL table fails due to incorrect check for supported ACL actions sonic-net#11235 (sonic-net#2351)
1ed0b4b [macsec] Refactor the logic of macsec name map (sonic-net#2348)
f88f992 [mock_tests] Add Sflow Orch UTs (sonic-net#2295)
ec57bf1 [macsec] Update macsec flex counter (sonic-net#2338)
6e0fc85 [ACL] Support stage particular match fields (sonic-net#2341)
efb4530 [orchagent, DTel]: report session support to set user vrf (sonic-net#2326)
d82874d Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig sonic-net#11046" (sonic-net#2332)
0c789e6 Fix qos map test in vs test (sonic-net#2343)
1bb5070 Enhance mock test for dynamic buffer manager for port removing and qos reload flows (sonic-net#2262)
700492f [aclorch] Fix and simplify DTel watchlist tables and entries (sonic-net#2155)

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
@jimmyzhai jimmyzhai mentioned this issue Jun 26, 2022
Merged
7 tasks
jimmyzhai added a commit that referenced this issue Jun 27, 2022
@jimmyzhai
advance submodule sonic-swss (#11260)
d18ff26 
2022-06-24 93af69c: [PFC_WD] Avoid applying ZeroBuffer Profiles to ingress PG when a PFC storm is detected (sonic-net/sonic-swss#2304)
2022-06-24 37349cf: [swssconfig] Optimize performance of swssconfig (sonic-net/sonic-swss#2336)
2022-06-24 84e9b07: [fdborch] fix heap-use-after-free in clearFdbEntry() (sonic-net/sonic-swss#2353)
2022-06-24 1b8bd94: Create ACL table fails due to incorrect check for supported ACL actions #11235 (sonic-net/sonic-swss#2351)
2022-06-24 1ed0b4b: [macsec] Refactor the logic of macsec name map (sonic-net/sonic-swss#2348)
2022-06-23 f88f992: [mock_tests] Add Sflow Orch UTs (sonic-net/sonic-swss#2295)
2022-06-23 ec57bf1: [macsec] Update macsec flex counter (sonic-net/sonic-swss#2338)
2022-06-22 6e0fc85: [ACL] Support stage particular match fields (sonic-net/sonic-swss#2341)
2022-06-22 efb4530: [orchagent, DTel]: report session support to set user vrf (sonic-net/sonic-swss#2326)
2022-06-22 d82874d: Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig #11046" (sonic-net/sonic-swss#2332)
2022-06-22 0c789e6: Fix qos map test in vs test (sonic-net/sonic-swss#2343)
2022-06-17 1bb5070: Enhance mock test for dynamic buffer manager for port removing and qos reload flows (sonic-net/sonic-swss#2262)
2022-06-16 700492f: [aclorch] Fix and simplify DTel watchlist tables and entries (sonic-net/sonic-swss#2155)
prsunny pushed a commit that referenced this issue Jun 28, 2022
@vivekrnv
[submodule]update sonic-swss submodule (#11253)
8dbfb0a 
Swss commits:

93af69c [PFC_WD] Avoid applying ZeroBuffer Profiles to ingress PG when a PFC storm is detected (#2304)
37349cf [swssconfig] Optimize performance of swssconfig (#2336)
84e9b07 [fdborch] fix heap-use-after-free in clearFdbEntry() (#2353)
1b8bd94 Create ACL table fails due to incorrect check for supported ACL actions #11235 (#2351)
1ed0b4b [macsec] Refactor the logic of macsec name map (#2348)
f88f992 [mock_tests] Add Sflow Orch UTs (#2295)
ec57bf1 [macsec] Update macsec flex counter (#2338)
6e0fc85 [ACL] Support stage particular match fields (#2341)
efb4530 [orchagent, DTel]: report session support to set user vrf (#2326)
d82874d Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig #11046" (#2332)
0c789e6 Fix qos map test in vs test (#2343)
1bb5070 Enhance mock test for dynamic buffer manager for port removing and qos reload flows (#2262)
700492f [aclorch] Fix and simplify DTel watchlist tables and entries (#2155)

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>
skbarista pushed a commit to skbarista/sonic-buildimage that referenced this issue Aug 17, 2022
@yxieca @skbarista
[202205][swss][utilities][sairedis][pmon] advance submodule head (son…
c05d88c 
…ic-net#11223)

swss:
* 43b9f06 2022-06-23 | [ACL] Support stage particular match fields (sonic-net#2341) (HEAD -> 202205) [bingwang-ms]
* c7f1add 2022-06-22 | Fix qos map test in vs test (sonic-net#2343) [bingwang-ms]
* 0549f3c 2022-06-22 | Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig sonic-net#11046" (sonic-net#2332) [anilkpan]

utilities:
* 785508d 2022-05-25 | [GCU] Handling type1 lists (sonic-net#2171) (HEAD -> 202205) [Mohamed Ghoneim]
* 56c2c6b 2022-06-22 | [config reload] Fixing config reload when timer based services are disabled (sonic-net#2200) (sonic-net#2226) (HEAD -> 202205, github/202205) [Sudharsan Dhamal Gopalarathnam]
* 600979e 2022-06-16 | Fix header for the output table following 'show ipv6 interface' command (sonic-net#2219) [Dror Prital]

sairedis:
* 14fba45 2022-06-13 | [vslib]: Fixbug in cleanup MACsec device (sonic-net#1059) (HEAD -> 202205) [Ze Gan]
* ac6e7ad 2022-04-29 | [vslib]: Update packet number of MACsec SA at runtime (sonic-net#1007) [Ze Gan]

platform-daemons:
* 6ffec22 2022-06-22 | [ycabled] Fix the init values for active-active ports (sonic-net#266) (HEAD -> 202205) [vdahiya12]

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
vivekrnv pushed a commit to vivekrnv/sonic-buildimage that referenced this issue Aug 26, 2022
@anilkpan
Fix for "orchagent crashed when trying to delete fdb static entry wit…
d82874d 
…h swssconfig sonic-net#11046" (sonic-net#2332)

* Fix updated to not flush static mac
prsunny pushed a commit that referenced this issue Sep 14, 2022
@dgsudharsan
[202012][submodule] Update sonic-swss submodule (#12060)
df9b676 
Update sonic-swss submodule pointer to include the following:
* VxLAN Tunnel Counters and Rates implementation (#1859) ([#2442]
* Fix for "orchagent crashed when trying to delete fdb static entry with swssconfig #11046" ([#2374]
@ZhaohuiS
Copy link
Contributor Author

Fixed in this PR sonic-net/sonic-swss#2332.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BRCM Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zhangyanzhao @anilkpan @ZhaohuiS