HW Verify Issue 50 Fiber Link Health

HW Verify — Issue #50: How to check health of the fiber link

Item: Issue #50 (How to check health of the fiber link?). Board status: Needs HW Test · Track: Firmware · Priority: P1.

← back to Hardware Verification

Helper script: software/scripts/hwtest/check_link_health.py automates the counter read/poll and doubles as the check_link_health helper this issue asks for.

python check_link_health.py --host localhost --port 9099 --seconds 30
# while perturbing the fiber (Part 2), invert the verdict:
python check_link_health.py --host localhost --port 9099 --expect-faults

Inducing the actual fiber fault (Part 2) is still manual.

What we are verifying

UIUC saw intermittent issues suspected to be a bad fiber connection, and asked which counters expose malformed/corrupted packets so a link-health check function can be built. This task is to (1) find and read the right counters on hardware, (2) confirm they respond to a real link fault, and (3) capture the recipe so it can be wrapped into a helper.

The deployed transport is Ethernet + RSSI (the PGP core is disabled by default), so the primary link-health source is the RSSI cores' counters, backed up by PGP counters if a PGP link is in use.

Where the counters live

Under ColumnBoard[c].WarmTdmCore.ComCore (firmware/python/warm_tdm/_ComCore.py):

• EthCore.SRP_RSSI and EthCore.Data_RSSI — the two RSSI connections (register access + data). Both default to enabled=False and are in the NoConfig group — you must enable them before their counters read valid.
• PgpCore.Pgp2bAxi[0] — PGP counters, but PgpCore is enabled=False by default (only relevant if a PGP link is actually up).

Key RSSI counters (surf .../protocols/rssi/_RssiCore.py, all RO): OpenConn, DropCnt (dropped segments), RetransmitCnt, ReconnectCnt, ValidCnt, TxFrameRate/RxFrameRate, TxBandwidth/RxBandwidth, and FSM states (ConnState, TxTspState, RxTspState, …).

Key PGP counters (surf .../protocols/pgp/_Pgp2bAxi.py, RO): link-ready booleans RxPhyReady/TxPhyReady/RxRemLinkReady, and error counters RxCellErrorCount, RxLinkDownCount, RxLinkErrorCount, RxFrameErrorCount, RxRemOverflow*Count, TxFrameErrorCount, plus RxFrameCount/TxFrameCount.

Procedure

Part 1 — read a healthy link

1. Start the server against real hardware and connect (see Common bench setup).

2. Enable the RSSI cores and read the baseline counters:

   cc = sess.group.ColumnBoard[0].WarmTdmCore.ComCore
   for rssi in (cc.EthCore.SRP_RSSI, cc.EthCore.Data_RSSI):
       rssi.enable.set(True)
       rssi.ReadDevice()
       print(rssi.path,
             "Open=",   rssi.OpenConn.get(),
             "Conn=",   rssi.ConnState.get(),
             "Drop=",   rssi.DropCnt.get(),
             "Retx=",   rssi.RetransmitCnt.get(),
             "Recon=",  rssi.ReconnectCnt.get(),
             "RxRate=", rssi.RxFrameRate.get())

   On a healthy link: OpenConn True, ConnState connected, DropCnt / RetransmitCnt / ReconnectCnt low and not increasing, frame rates nonzero while data flows.

3. Poll over ~30 s (or enable polling) with normal traffic and confirm the error counters stay flat. Record the baseline values.

Part 2 — perturb the link and watch the counters move

4. Induce a real fault so you can confirm the counters actually register it. Use the gentlest that reproduces — e.g. briefly reseat/partially unseat the fiber, or introduce a lossy connection — while polling:

   import time
   for _ in range(30):
       cc.EthCore.SRP_RSSI.ReadDevice()
       print(cc.EthCore.SRP_RSSI.DropCnt.get(),
             cc.EthCore.SRP_RSSI.RetransmitCnt.get(),
             cc.EthCore.SRP_RSSI.ReconnectCnt.get())
       time.sleep(1)

5. Confirm DropCnt / RetransmitCnt / ReconnectCnt (and/or PGP Rx*ErrorCount if PGP is in use) increment during the fault and that ConnState / OpenConn reflect a drop+reconnect. This proves the counters are a usable health signal.

Part 3 — write down the recipe

6. Capture the exact set of variables that best indicate health (the delta of DropCnt/RetransmitCnt/ReconnectCnt over an interval, plus ConnState) so it can be wrapped into a check_link_health()-style helper. Note whether the RSSI counters or PGP counters were the live source on this bench.

Pass criteria

• RSSI (and/or PGP) counters read cleanly on a healthy link, baseline recorded.
• A deliberately perturbed fiber makes the error/reconnect counters increment and ConnState/OpenConn reflect the fault.
• A documented minimal set of variables + interpretation, ready to wrap into a link-health helper.

Record

• Firmware build stamp + git hash, software commit, conda env:
• Which cores were live (RSSI vs. PGP):
• Baseline vs. during-fault counter values:
• Proposed health-check variable set:

References

• Issue #50 (request for malformed/corrupted-packet counters)
• firmware/python/warm_tdm/_ComCore.py — EthCore (SRP_RSSI, Data_RSSI), PgpCore (Pgp2bAxi[0])
• surf .../protocols/rssi/_RssiCore.py — RSSI counters
• surf .../protocols/pgp/_Pgp2bAxi.py — PGP counters
• RTL: PgpEthCore.vhd, EthCore.vhd, RingRouter.vhd (linkRxGood/linkTxGood)