[22.03] mt7621: VDSL SFP module does not work but works fine in 21.02.x

[david-collett]

Hi,
I have a Proscent 180-T VDSL SFP module which has been working for 12 months on Openwrt 21.02.x in a Mikrotik RB760iGS (hEX S) router. The module does not support auto-negotiation and ships with instructions saying to disable autoneg and set speed to 1000. As such, I use the following ethtool command:

ethtool -s sfp speed 1000 duplex full autoneg off

The ethtool output of the working module is:

Settings for sfp:
	Supported ports: [ TP MII ]
	Supported link modes:   100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	                        1000baseX/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseT/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  1000baseX/Full 
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 7
	Transceiver: external
	Auto-negotiation: off
	Current message level: 0x000000ff (255)
			       drv probe link timer ifdown ifup rx_err tx_err
	Link detected: yes

After upgrading to 22.03 (and installing/running ethtool as above), the module no longer works (Timeout waiting for PADO packets / Unable to complete PPPoE Discovery). Below is the new output of ethtool in 22.03:

Settings for sfp:
    Supported ports: [ TP MII ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  1000baseX/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 7
    Transceiver: external
    Auto-negotiation: off
    MDI-X: Unknown
    Current message level: 0x000000ff (255)
                   drv probe link timer ifdown ifup rx_err tx_err
    Link detected: yes

I noted that the port was different, however ethtool -s sfp port mii does nothing, at least while the interface is up. If I revert to 21.02.3, it immediately works again. It appear the driver (at803x?) has undergone some changes and no longer works for this module in 22.03. Any ideas?

Thanks,
Dave

[LGA1150]

Auto-negotiation is mandatory in 1000baseT mode.

[david-collett]

My issue is that this is a regression. My router that worked on 21.03 no longer works after upgrading to 22.03. Do you have any suggestions I could try?

[arinc9]

@david-collett can you try the image on the link below? Luci is included.

https://files.arinc9.com/OpenWrt/

[david-collett]

@arinc9 What am I testing? i.e. what have you changed? Are these changes in a repository somewhere?

[arinc9]

On 760igs, the phy that your SFP is connected to requires the mac, which is connected to the phy, to set an RX delay. I recently made some changes to mt7621.dtsi that didn't account for this. I will send a patch if your SFP starts working with this image.

[david-collett]

thanks mate, I'll let you know.

[david-collett]

@arinc9

Unfortunately it hasn't made a difference and I've reverted to 21.02.5.
Attached is some output that may be helpful.

fail.txt
win.txt

[arinc9]

Does the sfp driver detect the sfp module?

From my 760igs with a Fiberend RJ45 SFP module:

[   12.396428] sfp sfp1: Host maximum power 1.0W
[...]
[   13.185887] sfp sfp1: module Fiberend         1G-S-T           rev 1.0  sn XXXXXXXXXX    dc XXXXXX  
[   13.204616] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: SFP interface sgmii
[...]
[   97.486534] mtk_soc_eth 1e100000.ethernet sfp: PHY [mdio-bus:07] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
[   97.507502] mtk_soc_eth 1e100000.ethernet sfp: configuring for phy/rgmii-rxid link mode
[   97.524183] at803x_config_aneg: fiber
[   97.531904] br-lan: port 5(sfp) entered blocking state
[   97.542232] br-lan: port 5(sfp) entered disabled state
[   97.553275] device sfp entered promiscuous mode
[  105.816047] mtk_soc_eth 1e100000.ethernet sfp: Link is Up - 1Gbps/Full - flow control rx/tx
[  105.832780] br-lan: port 5(sfp) entered blocking state
[  105.843224] br-lan: port 5(sfp) entered forwarding state

[david-collett]

I'll reflash and check that tomorrow. Unfortunately this router+sfp is my daily driver!

[david-collett]

Yes, it is detected as shown:

[  318.767791] sfp sfp1: Host maximum power 1.0W
[  319.290096] sfp sfp1: module Proscend         180-T            rev V3.4 sn 18234L4TA2180005 dc 21082400

The main difference between the outputs of ethtool on 21.02 and 22.03 is the port (MII vs Twisted Pair), could that be relevant? as mentioned in the original port, I'm unable to make any port change stick with ethtool.

[arinc9]

The main difference between the outputs of ethtool on 21.02 and 22.03 is the port (MII vs Twisted Pair), could that be relevant?

I believe so. I think the problem resides at the Atheros PHY driver. If I remember correctly, SFP support for the driver is maintained by OpenWrt folks. Looking at the kernel log, there are changes to the driver from 21.02 to 22.03. See how PHY [mdio-bus:07] driver [Atheros 8031 ethernet] changes to PHY [mdio-bus:07] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)?

I don't debug drivers so I can't do much at this point. It'd be a good idea to mention this issue on openwrt-devel mailing list.
The rgmii delay issue I mentioned would be another roadblock so at least we cleared that up.

[david-collett]

Yeah, I had noticed there appears to be significant changes to that driver.
I'll join the dev list. Thanks for your help.

[david-collett]

This is still broken in 23.05.0 unfortunately.