system-health service_checker should check containers based on asic presence

[spilkey-cisco]

Why I did it

On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created.

system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file).

system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity.

How I did it

Port container_checker logic from #11442 into service_checker for system-health.

How to verify it

Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: spilkey-cisco (3342a31)

[bmridul]

@abdosi Pls assign folks for review.

[shivuv]

@abdosi Pls assign folks for review.

@abdosi @gechiang : Can you have someone review this?

[abdosi]

@prgeor can you please review/approve this.

[prgeor]

@spilkey-cisco have you tested the scenario where if the fabric card once inserted, can still have hardware issues causing ASIC from detecting? or this is not a valid scenario?

[spilkey-cisco]

@spilkey-cisco have you tested the scenario where if the fabric card once inserted, can still have hardware issues causing ASIC from detecting? or this is not a valid scenario?

That scenario should be covered by the checks in pcied, validating that expected devices exist.

[shivuv]

@abdosi @gechiang @prgeor : If you are ok with this change, please approve and merge it. Also requesting to cherry pick this into 202205 branch.

[gechiang]

@lguohan None of us has permission to merge. Please help review and if approved, merge the PR.
Thanks!

[mssonicbld]

Cherry-pick PR to 202211: #13857

[gechiang]

@yxieca can you help cherry-pick this to 202205 branch?
Thanks!

[mssonicbld]

@spilkey-cisco PR conflicts with 202205 branch

[gechiang]

@spilkey-cisco please submit a separate PR for this in 202205 branch as it could not be cherry-picked cleanly...

[spilkey-cisco]

@spilkey-cisco please submit a separate PR for this in 202205 branch as it could not be cherry-picked cleanly...

Created #13966

[StormLiangMS]

@spilkey-cisco this PR break the test case test_service_checker_with_process_exit, because the log format change, pls run sonic-mgmt test suite if any print log changes, that could break test cases.

+@rlhui for vis

[spilkey-cisco]

@spilkey-cisco this PR break the test case test_service_checker_with_process_exit, because the log format change, pls run sonic-mgmt test suite if any print log changes, that could break test cases.

+@rlhui for vis

I was not aware of this requirement or impact, I will keep it in mind in the future. Perhaps this PR should not have been approved or merged without requesting these test results. Are there any contributor guidelines you can point me to? I'd like to be sure I meet all these requirements before raising PRs or asking for approval in the future.

[gechiang]

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

[spilkey-cisco]

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

The log format change was done because the same process name can exist in multiple containers, especially for multi-asic. It certainly can be reverted, but the system-health errors become ambiguous. Please let me know if it needs to be reverted and if you need me to open a PR for that, or if you will handle the change.

[StormLiangMS]

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

The log format change was done because the same process name can exist in multiple containers, especially for multi-asic. It certainly can be reverted, but the system-health errors become ambiguous. Please let me know if it needs to be reverted and if you need me to open a PR for that, or if you will handle the change.

@spilkey-cisco I think the container name is already in, why we need the later one? It could cause the difference among different releases, pls revert this back.
self.set_object_not_ok('Process', '{}:{}'.format(container_name, process_name), "Process '{}' in container '{}' is not running".format(process_name, container_name))

[StormLiangMS]

@yaqiangz when this one is reverted, I think we should change the testcases back too?

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

I thought it changes the syslog, but from what Yaqiang found, it is a DB info change. @gechiang

[spilkey-cisco]

@spilkey-cisco I think the container name is already in, why we need the later one? It could cause the difference among different releases, pls revert this back. self.set_object_not_ok('Process', '{}:{}'.format(container_name, process_name), "Process '{}' in container '{}' is not running".format(process_name, container_name))

The container name was not displayed in the system-health CLI, only the final string argument '<process>' is not running. On multi-asic systems especially where duplicates of containers exist, the CLI (ie. show system-health summary) does not display the container name:

	     'syncd' is not running
	     'teammgrd' is not running
	     'syncd' is not running
	     'syncd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'teammgrd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'syncd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'teammgrd' is not running
	     'teammgrd' is not running

With the log change, the CLI tells us which container they belong to:

	     Process 'syncd' in container 'syncd0' is not running
	     Process 'syncd' in container 'syncd12' is not running
	     Process 'teammgrd' in container 'teamd0' is not running
	     Process 'teammgrd' in container 'teamd1' is not running
	     Process 'syncd' in container 'syncd1' is not running
	     Process 'syncd' in container 'syncd13' is not running
	     Process 'teamsyncd' in container 'teamd12' is not running
	     Process 'teammgrd' in container 'teamd12' is not running
	     Process 'teammgrd' in container 'teamd13' is not running
	     Process 'vxlanmgrd' in container 'swss12' is not running
	     Process 'vlanmgrd' in container 'swss12' is not running
	     Process 'vxlanmgrd' in container 'swss1' is not running
	     Process 'vlanmgrd' in container 'swss1' is not running
	     Process 'vxlanmgrd' in container 'swss0' is not running
	     Process 'vlanmgrd' in container 'swss0' is not running
	     Process 'vxlanmgrd' in container 'swss13' is not running
	     Process 'vlanmgrd' in container 'swss13' is not running