KSZ9477 driver for newer versions of linux kernel

[ivisha]

For our new design, based on Xilinx Zynq SoC, we plan to use an Ethernet switch KSZ9477. We use Xilinx PetaLinux 2020.1 that comes with the kernel version 5.4.

We did preliminary testing on EVB-KSZ9477, deploying Linux kernel 4.9 using https://github.com/Microchip-Ethernet/EVB-KSZ9477 according to Microhip Driver Setup Guide.

On the aforementioned Microhip github, KSZ driver files for the linux kernel up to version 5.3 can be found under the folder ..KSZ/linux-drivers/ksz9897. However, there is a significant driver logic difference between the linux kernel versions 5.3 and 5.4 and the Microchip changes are not easily implementable to the kernel 5.4 . On the other hand, all mainline Linux kernel versions 5.x include KSZ9477 driver at linux/drivers/net/dsa/microchip.

Is Microchcip going to commit KSZ9477 driver for newer linux kernel versions soon? Does this driver implemented in the mainline linux kernel by default provide some of the functionalities of the switch, or we need to implement driver for the switch from the ..KSZ/linux-drivers/ksz9897 according to the instructions from the README file for the appropriate kernel version?

[triha2work]

The 5.3 switch driver can be compiled in the 5.4 kernel. Can you tell me the specific error? One major issue is the skb_append_datato_frags function is no longer in the newer kernels probably because no one else uses it. Putting back that function in 5.3 still works, but there is a risk it may not work properly in 5.4. In that case another function should be used to consolidate all socket buffer fragments into one. This decreases the TCP transmit performance significantly.
The DSA driver in the mainline kernel does not require modification in the MAC driver but only provides basic functions, so it may not be suited for some applications.

[ivisha]

In the meantime, we moved to PetaLinux version 2020.2 which ships with Linux 5.4 in attempt to use the KSZ9897 driver for 5.4 included in this repo.

However, it doesn't work because this repo's Cadence MAC driver for 5.4 uses phylink framework (#include <linux/phylink.h>). We replaced all Xilinx-included drivers with patched drivers for 5.4 from here, but (after source fixes on Cadence driver required to pass the compilation stage) the build ultimately fails with various linker errors, suggesting that the patched driver relies on functionality still not available in actual version 5.4.

As a sanity check, we also went through the 5.4 branch on kernel.org to confirm that mainline Cadence MAC driver was based on pre-phylink code through the entire 5.4 branch lifetime:

5.4 branch:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/?h=linux-5.4.y

The latest Cadence MAC driver from 5.4:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/net/ethernet/cadence/macb_main.c?h=linux-5.4.y

(includes linux/phy.h instead of linux/phylink.h and doesn't use new phylink structs at any point)

Please suggest the solution for building patched KSZ9897 driver with kernel 5.4 (i.e. how you tested modifications with version 5.4), or if possible provide Cadence MAC driver which relies on the old linux/phy.h interface instead of phylink framework.

[triha2work]

There are updated drivers for 5.4 and 5.10 in the linux-drivers directory.

[ivisha]

The KSZ9897 5.4 driver patch is not compatible with the mainline 5.4 kernel version (on which Xilinx 5.4 kernel is based on). More specifically, Cadence MAC driver in KSZ9897 5.4 driver patch uses phylink framework, which first appeared in the Cadence MAC driver in the mainline 5.5 version.

So we do not understand what is the “original” 5.4 kernel for which the 5.4 KSZ9897 driver patch is made for, and how to build a patched KSZ9897 driver with the kernel 5.4.

[romatou18]

Hi @micreladmin, @triha2work ,

Same observation here:
#78

Would please be able to assign resources to testing and amending as required the drivers and kernel patch (fec.h fec_mains.c) for kernel version 5.4 ?

In spite of our best efforts we are still unable to link to the KSZ9477 using the RGMII interface via SPI with our IMX8/FEC ethernet controller.

We are looking forward to your help.

Kind regards

The Flightcell.com software team.

[triha2work]

The MAC driver patches were updated with the mainline kernels instead of the ones from linux4sam.

[Bartel-C8]

@triha2work : I think there was a small mistake generating the 006-freescale.patch for linux-linux4sam-2021.04. It's headers changed as well to the linux-linux4sam-2020.10, so for the old version. The patch does not apply properly. Is it possible to regenerate it? Thanks!

https://github.com/Microchip-Ethernet/EVB-KSZ9477/blob/b325a94d8748c39fba8ef6c0de76a24614a99196/KSZ/Atmel_SOC_SAMA5D3/patches/linux-linux4sam-2021.04/006-freescale.patch

[Bartel-C8]

Also, I have issue with the 5.10 kernel (patches). For some reason, the KSZ chip is not found:
failed to read device ID(0x128ee00)

By coincidence, this 0x128ee00 is 0x947700 but shifted left one bit?
This is with IBA not set, if I set this, it seems resolved. But I don't have/want IBA enabled.
(I saw when enabling IBA, this is done: /* Turn off SPI auto edge detection. */, so related to that?)

It has something to do with SPI, but what changed? (It seems nothing in the driver, so must be kernel related)?

atmel_spi f0004000.spi: Using dma0chan4 (tx) and dma0chan5 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller version 0x213 at 0xf0004000 (irq 19)
atmel_spi f8008000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
ksz9477@0 enforce active low on chipselect handle
ksz9897 spi1.0: failed to read device ID(0x128ee00)
atmel_spi f8008000.spi: Atmel SPI Controller version 0x213 at 0xf8008000 (irq 23)

The SPI probe worked one time with the 5.10 kernel, but didn't have link.

atmel_spi f0004000.spi: Using dma0chan4 (tx) and dma0chan5 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller version 0x213 at 0xf0004000 (irq 19)
atmel_spi f8008000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
ksz9477@0 enforce active low on chipselect handle
ksz9897 spi1.0: chip id 0x00947700
libphy: Switch MII bus: probed
random: crng init done
atmel_spi f8008000.spi: Atmel SPI Controller version 0x213 at 0xf8008000 (irq 23)

Log for 5.4 kernel:

atmel_spi f0004000.spi: Using dma0chan4 (tx) and dma0chan5 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller version 0x213 at 0xf0004000 (irq 19)
atmel_spi f8008000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
ksz9477@0 enforce active low on chipselect handle
ksz9897 spi1.0: chip id 0x00947700
libphy: Switch MII bus: probed
random: crng init done
atmel_spi f8008000.spi: Atmel SPI Controller version 0x213 at 0xf8008000 (irq 23)

[Bartel-C8]

@triha2work , I saw you did some IBA changes. Were they intended to fix my issue here by coincidence?

Tried a fresh built with the latest state of master currently.
And: I still see that shifted SPI ID. 0x128ee00 = 0x00947700 << 1

What is weird is, when I upgrade (from a working 5.4 kernel) to 5.10, a soft-reset is performed to boot to the new 5.10 kernel. At that time, everything works.

atmel_spi f0004000.spi: Using dma0chan4 (tx) and dma0chan5 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller version 0x213 at 0xf0004000 (irq 19)
atmel_spi f8008000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
ksz9477@0 enforce active low on chipselect handle
ksz9897 spi1.0: chip id 0x00947700
libphy: Switch MII bus: probed
random: crng init done
atmel_spi f8008000.spi: Atmel SPI Controller version 0x213 at 0xf8008000 (irq 23)
libphy: Fixed MDIO Bus: probed
libphy: MACB_mii_bus: probed
pps pps0: new PPS source ptp0
macb f0028000.ethernet eth0: Cadence GEM rev 0x00020119 at 0xf0028000 irq 42 (60:22:fd:ff:ff:01)

Even after multiple (tested 5 times or so) soft-reboots (reboot command in kernel) it keeps working in 5.10. But after the first power off/on reboot, I get the shifted KSZ ID. From which I never can recover again...
My ports are working (I have link and such), but the driver isn't loaded/configured? (So probably the KSZ is still configured as default?)

atmel_spi f0004000.spi: Using dma0chan4 (tx) and dma0chan5 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller version 0x213 at 0xf0004000 (irq 19)
atmel_spi f8008000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
ksz9477@0 enforce active low on chipselect handle
ksz9897 spi1.0: failed to read device ID(0x128ee00)
atmel_spi f8008000.spi: Atmel SPI Controller version 0x213 at 0xf8008000 (irq 23)
libphy: Fixed MDIO Bus: probed
libphy: MACB_mii_bus: probed
libphy: PHY sw.0:00 not found
macb f0028000.ethernet eth0: Cadence GEM rev 0x00020119 at 0xf0028000 irq 42 (60:22:fd:ff:ff:01)

What could be wrong? Some init/power sequence or so?
Am I missing something here?

Thanks!

[Bartel-C8]

Update:

I got it working if I remove spi-cpha in the DTS for the KSZ. (It also works when removing both spi-cpha and spi-cpol, but not when only removing spi-cpol).
Is the spi-cpol and spi-cpha actually needed? As in de datasheet of the KSZ is stated:

SCL is expected to stay low when SPI operation is idle. SPI operations start with the falling edge of SCS_N and end with
the rising edge of SCS_N

Which means SCL should be low, meaning, no cpol?

and

Input data on SDI is latched on the rising edge of clock SCL. Output data on SDO is clocked on the falling edge of SCL.

Which means no cpha?

As described here? https://dlnware.com/theory/SPI-Transfer-Modes
Or is it all inverted (as actually the images in the datasheet, so it's a bit confusing)?

In the datasheet FIGURE 4-11 and FIGURE 4-12 have SCL high, but later on in FIGURE 6-9 and FIGURE 6-10 SCL is default low? I guess this last one is the correct though?

But for me, it only works, dropping this spi-spha...
Maybe it worked before by chance?

Also in mainline linux kernel:

Removed spi-cpha and spi-cpol flags is this should be handled by the device driver.

torvalds/linux@4f36d97#diff-e837636426ef5e9f969a0836d2e545c92933531182e89b93fdcf58b3ad9c13ff

[Bartel-C8]

@triha2work : Could you confirm/deny my above findings? Thanks!

In both cases I was using the default SPI mode, as configured in the driver. And I was using an unaltered build with the drivers for 5.4 and 5.10 here...

Do you have an idea of the spi-cpha ? I changed that in the DTS. We are using the KSZ9477S.

[triha2work]

SPI mode 0 or 3 works for KSZ9477 SPI access. In theory mode 3 works better because it may provide more time for hardware to process the command. All KSZ switches are accessed using SPI mode 3 in Linux drivers. If mode 0 works better than mode 3 then just use mode 0.
KSZ9897 chip has a hack to improve high speed access when it detects a high frequency is used after hardware reset. But it actually causes problem in reading later so the Linux driver always disables that feature in driver initialization. Now I do not know if that hack causes this issue as the chip may be accessed before the SPI driver loads. But if mode 0 does not cause any problem then please use it.

[triha2work]

spi-cpol and spi-cpha are used to specifiy the SPI mode. Without them the mode is 0, With them the mode is 3.
The hardware hack is called SPI Auto Edge Detection.