Add deterministic link bring-up feature for SFF compliant modules #383

longhuan-cisco · 2023-06-26T20:17:08Z

HLD update is in different PR (link: sonic-net/SONiC#1371)

Description

According to Interface-Link-bring-up-sequence.md, add a new thread sff_mgr under xcvrd to provide deterministic link bringup feature for SFF compliant modules (100G/40G).

By default sff_mgr is disabled, to enable sff_mgr on a platform, add 'enable_xcvrd_sff_mgr' and set it to true in pmon_daemon_ctrl json file.

Motivation and Context

Scope of sff_mgr: 100G/40G optics (copper/aoc not in the scope)
Why sff_mgr
The goal of sff_mgr is to make sure SFF compliant modules are brought up in a deterministc way, meaning TX is enabled only after host_tx_ready becomes True, and TX will be disabled when host_tx_ready becomes False. This will help eliminate link stability issue and potential interface flap, also turning off TX reduces the power consumption and avoid any lab hazard for admin shut interface.
What sff_mgr does
sff_mgr is a new thread inside Xcvrd. It will only turn on module Tx if both admin_status=up and host_tx_ready=true.
sff_mgr will skip if neither of these below events happens: transceiver insertion event (including bootup and hot plugin), host_tx_ready change event and admin_status change event. All other cases are ignored.
To detect these events, sff_mgr listens to below DB tables:
- STATE_DB PORT_TABLE's host_tx_ready field for host_tx_ready change event.
- STATE_DB TRANSCEIVER_INFO table's type field for insesrtion event.
- CONFIG_DB PORT_TABLE's admin_status field for admin_status change event, and info such as index/channel/etc.
For platforms/vendors:
This feature is enabled on per platform basis. There could be cases where vendor(s)/platform(s) may take time to shift from existing codebase to the model (work-flows).
By default this feature is disabled. No impact for the platforms in current deployment. To enable it, a flag needs to be added to pmon_daemon_control.json and set to true.
- pre-requisite to enable this feature: platform needs to keep TX in disabled state after module coming out-of-reset, in either module insertion or bootup cases. This is to make sure the module is not transmitting with TX enabled before host_tx_ready becomes True.

How Has This Been Tested?

On platform with sff_mgr enabled which has 100G and 40G optics, verified TX is enabled/disabled properly based on host_tx_ready and in proper order, under cases of module insertion/system bootup. And no-op for the cases of xcvrd process restart.
Verified TX is enabled/disabled properly along with "config interface shutdown/startup" (which changes admin_status and host_tx_ready value)
Verified no impact on platform without sff_mgr enabled.

Additional Information (Optional)

longhuan-cisco · 2023-06-26T21:54:38Z

@prgeor @mihirpat1 could you please take a look at the PR?

CC @shyam77git @jaganbal-a

sonic-xcvrd/xcvrd/sff_mgr.py

sonic-xcvrd/xcvrd/xcvrd.py

eddyk-nvidia · 2023-10-23T20:14:06Z

sonic-xcvrd/xcvrd/sff_mgr.py

+                if mask == 0:
+                    self.log_notice("{}: No change is needed for tx_disable value".format(lport))
+                    continue
+                if api.tx_disable_channel(mask, target_tx_disable_flag):


Should we bring a module to high-power before enabling TX ? In CMIS State machine it is done as a part of CMIS Data Path SM

For now, non-cmis/ccmis optics are brought to High Power mode by default by platform.
The missing part is: enabling high power class by setting page 0 byte 93 bit 2(if advertised power class >=5). Yes, this handling will be added to sff_mgr in separate PR.

When you say "platform", it means there could be different implementation. So, does SFF standard require "platform" to set module to high power? If not, I prefer to add a check here and make sure module is in high power

When you say "platform", it means there could be different implementation. So, does SFF standard require "platform" to set module to high power?

I didn't see standard mentions who (PI or platform) should do it.
In today's sonic PI code, there's no place to turn module to high power mode. Then I assume platform side does something (either via SW or HW), otherwise how 100G optics (which needs high power mode to be functional) can come up on today's system for all platforms/vendors.

I prefer to add a check here and make sure module is in high power

Yes, that was part of the plan for high power handling in sff_mgr, basically sff_mgr will do two things:

check and make sure module in high power mode if needed

check and enable high power class for module if needed (power class >= 5)

raised issue to track high power handling #414

eddyk-nvidia · 2023-10-23T20:14:47Z

sonic-xcvrd/xcvrd/sff_mgr.py

+                else:
+                    self.log_error("{}: Failed to {} TX with channel mask: {}".format(
+                        lport, "disable" if target_tx_disable_flag else "enable", bin(mask)))
+


Do you plan to add the Custom module SI parameters here ?

Yes, custom optics SI is supposed to be handled by sff_mgr. The detail is still in discussion. This will be added via separate PR also.

So, when high-power and custom SI are supported the following sequence should be ensured for bringing a module up:

rx_output_disable=1

high-power

Set customer SI (per lane speed)

rx_output_disable=0

tx_disable=0

Agree ?

So, when high-power and custom SI are supported the following sequence should be ensured for bringing a module up:

rx_output_disable=1

high-power

Set customer SI (per lane speed)

rx_output_disable=0

tx_disable=0

Agree ?

At the time of high-power and custom SI are supported, 2, 3, 5 would be ensured.

sff spec doesn't seem to mention rx_output_disable/enable as mandatory, and they were not in the original discussion of sff_mgr. But 1 and 4 can be discussed or taken care of via subsequent PRs.

As long as we have the main structure of sff_mgr in place, it would be easy for people to add on top of it.

Raised issue to track SI setting: #415

prgeor · 2023-11-27T07:49:25Z

@mihirpat1 could you review?

prgeor · 2023-12-06T15:31:24Z

@longhuan-cisco could you check the conflict

longhuan-cisco · 2023-12-06T19:33:49Z

@longhuan-cisco could you check the conflict

sure, addressed the conflict.

sonic-xcvrd/xcvrd/sff_mgr.py

Junchao-Mellanox · 2023-12-12T04:58:42Z

sonic-xcvrd/xcvrd/sff_mgr.py

+                if mask == 0:
+                    self.log_notice("{}: No change is needed for tx_disable value".format(lport))
+                    continue
+                if api.tx_disable_channel(mask, target_tx_disable_flag):


When you say "platform", it means there could be different implementation. So, does SFF standard require "platform" to set module to high power? If not, I prefer to add a check here and make sure module is in high power

sonic-xcvrd/xcvrd/sff_mgr.py

sonic-xcvrd/xcvrd/xcvrd.py

longhuan-cisco · 2024-02-07T02:27:49Z

@longhuan-cisco why copper/AOC is not in the scope of SFF manager as per PR description?

AFAIK, those doesn't have ability of doing tx_disable/enable

longhuan-cisco · 2024-02-07T02:32:56Z

Any updates on this PR? Haven't seen recent activity and would appreciate an ETA if possible.

@longhuan-cisco : Can you provide latest update on this.

Updated the diff and raised two other PR's based on latest comments from Prince's.

the other PRs:
#430
sonic-net/sonic-buildimage#18054

rlhui · 2024-02-23T06:21:53Z

please resolve conflicts

longhuan-cisco · 2024-02-25T06:08:41Z

please resolve conflicts

conflicts resolved.

sonic-xcvrd/xcvrd/sff_mgr.py

judyjoseph · 2024-03-06T18:28:12Z

@prgeor Please merge if the review is completed.

prgeor · 2024-03-07T00:53:05Z

@longhuan-cisco why AOC is outside scope?

Scope of sff_mgr: 100G/40G optics (copper/aoc not in the scope)

sonic-xcvrd/xcvrd/xcvrd_utilities/xcvr_table_helper.py

longhuan-cisco · 2024-03-07T01:00:32Z

@longhuan-cisco why AOC is outside scope?

Scope of sff_mgr: 100G/40G optics (copper/aoc not in the scope)

@prgeor This was answered at #383 (comment)

AFAIK, those doesn't have ability of doing tx_disable/enable

prgeor · 2024-03-07T04:37:45Z

@longhuan-cisco there is some conflict, can you rebase?

longhuan-cisco · 2024-03-07T18:54:22Z

@longhuan-cisco there is some conflict, can you rebase?

@prgeor I didn't see conflicts, it just said branch "out-of-date". Anyway I updated it to latest master. please check.

This PR is a dependency of sonic-net/sonic-platform-daemons#383 HLD of sff_mgr: sonic-net/SONiC#1371 Why I did it Add enable_xcvrd_sff_mgr flag support for sff_mgr

…nic-net#383) * SFF manager for handling QSFP+/QSFP28 transceiver modules

longhuan-cisco added 9 commits June 21, 2023 10:44

Add initial support for sff_mgr

0af24ab

Remove unnecessary check and add comments

297e22d

Update comments

6eea7d8

Add return value to handle_port_update_event and update comments

04fa51e

Update comments

82ce3f3

Rely on asic_id from event update instead from static port_mapping

43ad2e1

Update comment

94a0c9d

Move sff_mgr to a new file

c93f851

Revert space change

bad4fd8

longhuan-cisco mentioned this pull request Jun 26, 2023

Add details of sff_mgr regarding deterministic bringup for SFF compliant modules sonic-net/SONiC#1371

Merged

bmridul requested review from prgeor, mihirpat1 and Junchao-Mellanox June 28, 2023 22:49

Update comments

4821c81

longhuan-cisco commented Jul 12, 2023

View reviewed changes

sonic-xcvrd/xcvrd/sff_mgr.py Outdated Show resolved Hide resolved

Add admin_status sanity check

dbf2ea5

keboliu reviewed Jul 30, 2023

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd.py Outdated Show resolved Hide resolved

longhuan-cisco requested a review from keboliu August 2, 2023 16:57

longhuan-cisco and others added 2 commits September 5, 2023 17:33

Revert unnecessary space change on xcvrd.py

a66831e

Merge branch 'master' into sff_mgr

741ac26

eddyk-nvidia reviewed Oct 23, 2023

View reviewed changes

Add NotImplementedError catching

689e46b

prgeor added the xcvrd label Nov 27, 2023

Merge branch 'master' into sff_mgr

288da9b

keboliu previously approved these changes Dec 11, 2023

View reviewed changes

Junchao-Mellanox suggested changes Dec 12, 2023

View reviewed changes

Change to get enable_sff_mgr flag from process argument

e03e8dd

Fix typo

e3dcff2

longhuan-cisco mentioned this pull request Feb 13, 2024

Add enable_xcvrd_sff_mgr flag support for xcvrd startup command sonic-net/sonic-buildimage#18054

Merged

10 tasks

rlhui assigned longhuan-cisco Feb 23, 2024

longhuan-cisco added 3 commits February 23, 2024 12:16

Merge branch 'master' into sff_mgr

bb7cdde

Resolve conflicts

51d16c3

Remove unused import and class variable in port_event_helper.py

83f7bc3

eddyk-nvidia approved these changes Feb 29, 2024

View reviewed changes

mihirpat1 approved these changes Mar 1, 2024

View reviewed changes

tshalvi approved these changes Mar 4, 2024

View reviewed changes

prgeor reviewed Mar 5, 2024

View reviewed changes

sonic-xcvrd/xcvrd/sff_mgr.py Show resolved Hide resolved

longhuan-cisco requested a review from prgeor March 5, 2024 23:31

prgeor reviewed Mar 7, 2024

View reviewed changes

sonic-xcvrd/xcvrd/xcvrd_utilities/xcvr_table_helper.py Show resolved Hide resolved

Change to update 'lanes' and 'subport' fields independently

178716c

longhuan-cisco requested a review from prgeor March 7, 2024 03:47

prgeor approved these changes Mar 7, 2024

View reviewed changes

Merge branch 'master' into sff_mgr

236079e

prgeor merged commit 8a5ca2b into sonic-net:master Mar 7, 2024
5 checks passed

noaOrMlnx pushed a commit to noaOrMlnx/sonic-platform-daemons that referenced this pull request Mar 14, 2024

Add deterministic link bring-up feature for SFF compliant modules (so…

13a24fb

…nic-net#383) * SFF manager for handling QSFP+/QSFP28 transceiver modules

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add deterministic link bring-up feature for SFF compliant modules #383

Add deterministic link bring-up feature for SFF compliant modules #383

longhuan-cisco commented Jun 26, 2023 •

edited

longhuan-cisco commented Jun 26, 2023

eddyk-nvidia Oct 23, 2023

longhuan-cisco Oct 24, 2023

Junchao-Mellanox Dec 12, 2023

longhuan-cisco Dec 18, 2023

longhuan-cisco Dec 20, 2023

eddyk-nvidia Oct 23, 2023

longhuan-cisco Oct 24, 2023

eddyk-nvidia Dec 14, 2023

longhuan-cisco Dec 18, 2023

longhuan-cisco Dec 20, 2023

prgeor commented Nov 27, 2023

prgeor commented Dec 6, 2023

longhuan-cisco commented Dec 6, 2023

Junchao-Mellanox Dec 12, 2023

longhuan-cisco commented Feb 7, 2024

longhuan-cisco commented Feb 7, 2024

rlhui commented Feb 23, 2024

longhuan-cisco commented Feb 25, 2024

judyjoseph commented Mar 6, 2024

prgeor commented Mar 7, 2024

longhuan-cisco commented Mar 7, 2024

prgeor commented Mar 7, 2024

longhuan-cisco commented Mar 7, 2024

Add deterministic link bring-up feature for SFF compliant modules #383

Add deterministic link bring-up feature for SFF compliant modules #383

Conversation

longhuan-cisco commented Jun 26, 2023 • edited

Description

Motivation and Context

How Has This Been Tested?

Additional Information (Optional)

longhuan-cisco commented Jun 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prgeor commented Nov 27, 2023

prgeor commented Dec 6, 2023

longhuan-cisco commented Dec 6, 2023

Choose a reason for hiding this comment

longhuan-cisco commented Feb 7, 2024

longhuan-cisco commented Feb 7, 2024

rlhui commented Feb 23, 2024

longhuan-cisco commented Feb 25, 2024

judyjoseph commented Mar 6, 2024

prgeor commented Mar 7, 2024

longhuan-cisco commented Mar 7, 2024

prgeor commented Mar 7, 2024

longhuan-cisco commented Mar 7, 2024

longhuan-cisco commented Jun 26, 2023 •

edited