Hardware Clock Synchronization Issue(NXP LS1021A TSN Switch)

[saiguvvala]

Hi,
I am working on the NXP LS1021A TSN switch and facing issues while trying to sync the hardware clock of NXP switch with other different board. I am getting high offset values while using PTP4L commands. Can you help me out by providing the information about decreasing the offset values.

I have run the commands ptp4l -i eth0 -m on the master device and ptp4l -i eth0 -m -s on the slave device
Output log:
ptp4l[2670.323]: selected /dev/ptp0 as PTP clock
ptp4l[2670.344]: port 1: INITIALIZING to LISTENING on INITIALIZE
ptp4l[2670.344]: port 0: INITIALIZING to LISTENING on INITIALIZE
ptp4l[2671.907]: port 1: new foreign master 00049f.fffe.ef0606-1
ptp4l[2675.908]: selected best master clock 00049f.fffe.ef0606
ptp4l[2675.908]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[2677.908]: master offset 1814389063 s0 freq +1000000 path delay -11282235
ptp4l[2678.908]: master offset 1886559517 s1 freq +1000000 path delay -20511764
ptp4l[2680.908]: master offset 121269846 s2 freq +1000000 path delay -15896999
ptp4l[2680.908]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[2681.908]: master offset 184210931 s2 freq +1000000 path delay -15896999
ptp4l[2682.908]: master offset 251765244 s2 freq +1000000 path delay -20511764
ptp4l[2683.908]: master offset 314706772 s2 freq +1000000 path delay -20511764
ptp4l[2684.908]: master offset 377648525 s2 freq +1000000 path delay -20511764
ptp4l[2685.908]: master offset 440589373 s2 freq +1000000 path delay -20511764
ptp4l[2686.908]: master offset 504437035 s2 freq +1000000 path delay -21418673
ptp4l[2687.908]: master offset 567378447 s2 freq +1000000 path delay -21418673
ptp4l[2688.908]: master offset 630240205 s2 freq +1000000 path delay -21339819
ptp4l[2689.908]: master offset 693181169 s2 freq +1000000 path delay -21339819
ptp4l[2690.909]: master offset 755680662 s2 freq +1000000 path delay -20897631
ptp4l[2691.909]: master offset 818622550 s2 freq +1000000 path delay -20897631
ptp4l[2692.909]: master offset 879851362 s2 freq +1000000 path delay -19185932
ptp4l[2693.909]: master offset 944503921 s2 freq +1000000 path delay -20897631
ptp4l[2694.909]: master offset 1007954619 s2 freq +1000000 path delay -21408405
ptp4l[2695.909]: master offset 1070896194 s2 freq +1000000 path delay -21408405
ptp4l[2696.909]: master offset 1132568476 s2 freq +1000000 path delay -20138162
ptp4l[2697.909]: master offset 1194557077 s2 freq +1000000 path delay -19185932
ptp4l[2698.909]: master offset 1257321980 s2 freq +1000000 path delay -19009231
ptp4l[2699.909]: master offset 1320774158 s2 freq +1000000 path delay -19520005
ptp4l[2700.909]: master offset 1386472022 s2 freq +1000000 path delay -22276805
ptp4l[2701.909]: master offset 1451115371 s2 freq +1000000 path delay -23979093
ptp4l[2702.909]: master offset 1512354701 s2 freq +1000000 path delay -22276805
ptp4l[2703.909]: master offset 1571665092 s2 freq +1000000 path delay -18645601
ptp4l[2704.910]: master offset 1626802503 s2 freq +1000000 path delay -10840052
ptp4l[2705.910]: master offset 1689743412 s2 freq +1000000 path delay -10840052
ptp4l[2706.910]: master offset 1751953893 s2 freq +1000000 path delay -10110519
ptp4l[2707.910]: master offset 1814895418 s2 freq +1000000 path delay -10110519

The kernel version and tag are

[root@OpenIL:misc]# uname -v
#1 SMP Fri Feb 1 17:11:55 CST 2019
[root@OpenIL:misc]# uname -r
4.14.47-ipipe
[root@OpenIL:misc]#

[vladimiroltean]

You say "TSN switch" which on this board is the SJA1105T, but then you show commands for PTP synchronization over the non-switched eth0 (eTSEC) port. Is the switch involved in any way at all in this sync process (does it forward any PTP packets)?
The switched ports are labeled ETH2, ETH3, ETH4, ETH5 on the chassis, while the non-switched ports are ETH0 and ETH1.

[saiguvvala]

We are using eth0.
We are open to suggestions for synchronization with even the switch ports

[saiguvvala]

Switch is not involved in sync process. We are trying to sync NXP LS1021ATSN-PA device(using ETH0) with an other different device by using PTP4L. As you mentioned in the other thread, PTP4L gives stable results when the initial offset between the hardware clocks is very less.
We tried to update the hardware clock of the NXP device(using hwclock cmd) to match its value with the other device's clock. But we couldn't. As per the support team, RTC is not available/implemented on this device.

Could you please help us with the below issues:

1. How to access, modify, update the hardware clock of this NXP device. Please share the steps.
2. How to verify if the NXP device and the other device are in sync. Please share the steps.
3. Is there any way other than PTP4L logs to verify the synchronization between the both devices. Please share the steps.
4. It seems SoC clock tries to sync Switch clock before forwarding the packets through switch ports.
   Which ports give better sync results. Is it Switch ports or SoC ports. How do we decide.
5. If you are going to suggest to use switch ports, then please share the steps to follow to do it.
   Also please share the respective switch port ids such seth# for ETH2-5.

Thanks in advance.

[vladimiroltean]

Thank you for your patience. I tested the kernel tag OpenIL-linux-201901 and found the bug.
The reason is this recent patch on the ptp_qoriq driver: ptp_qoriq: support time offset register using.
The 1588 clock driver is common on all Layerscape chips, but apparently the LS1021 doesn't support the TMROFF register. So when this patch is in place, stepping the clock is a no-op, and if the stepping is broken, the servo can't steer the clock fast enough via frequency adjustment to compensate for that (although that's what it's trying to do).
As you noticed, sometimes clock synchronization can be achieved but most of the time it can't. This is due to the amount of initial offset the slave has to compensate for. So to consistently see that synchronization is broken, one has to run phc_ctl /dev/ptp0 set 0 first, in order to maximize the offset.
Reverting the patch has no functional consequence and it fixes your issue, so for the moment please do that.

[vladimiroltean]

To address the rest of your questions:

How to access, modify, update the hardware clock of this NXP device. Please share the steps.

You can interact with the /dev/ptp0 clock using a rich kernel UAPI in C, or use the phc_ctl program from the linuxptp suite (or see how phc_ctl does that).

How to verify if the NXP device and the other device are in sync. Please share the steps.

Inspecting the ptp4l log is one way, inspecting phc_ctl /dev/ptp0 get is another.

It seems SoC clock tries to sync Switch clock before forwarding the packets through switch ports.
Which ports give better sync results. Is it Switch ports or SoC ports. How do we decide.
If you are going to suggest to use switch ports, then please share the steps to follow to do it.
Also please share the respective switch port ids such seth# for ETH2-5.

PTP synchronization over switch ports is in the works, not yet available. For now please use the ETH0 or ETH1 port.

[saiguvvala]

I am using the ETH0 and ETH1 ports for running PTP4L but getting high offset values. Used PHC_CTL to set hardware clock value but the Offset value is not stable. Can you please suggest me solutions for reducing the offset value.

[vladimiroltean]

Have you tried reverting the patch, as suggested?

[saiguvvala]

Yes i have tried the patch that you have suggested, but getting the same results.

[vladimiroltean]

i have tried the patch that you have suggested

Just to be clear, the command you ran was git revert 0301d8d, right?

[saiguvvala]

Where should i apply this command in terminal or GITHUB. I am not a github user and i was manually updating source code files in my local machine.

[vladimiroltean]

I assumed you already had a kernel tree with git history checked out.
If you don't, do the following on your PC:

git clone git@github.com:openil/linux.git openil-linux
cd openil-linux
git checkout master
git revert 0301d8d

Now go to your OpenIL source tree. The commands below will tell the build system to use the kernel sources that you just downloaded, instead of the ones it tries to build by default (which are not tracked by git).

cd openil
make nxp_ls1021atsn_defconfig
export LINUX_OVERRIDE_SRCDIR=$HOME/openil-linux
make linux-rebuild
make

Now either reflash the output/images/sdcard.img file or just the ls1021a-tsn.dtb and uImage files (which go to the mmcblk0p1 vfat partition.

[saiguvvala]

I have done the things what you have suggested and run the ptp4l on both master(NXP) and slave(other device)

1. log on the slave device is
   root@am57xx-evm:~# ptp4l -i eth0 -m -s
   ptp4l[728.483]: selected /dev/ptp0 as PTP clock
   ptp4l[728.505]: port 1: INITIALIZING to LISTENING on INITIALIZE
   ptp4l[728.506]: port 0: INITIALIZING to LISTENING on INITIALIZE
   ptp4l[729.577]: port 1: new foreign master 00049f.fffe.ef0606-1
   ptp4l[733.577]: selected best master clock 00049f.fffe.ef0606
   ptp4l[733.578]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
   ptp4l[734.577]: master offset 4972345492 s0 freq +1000000 path delay 0
   ptp4l[735.578]: master offset 5054742146 s1 freq +1000000 path delay -19455340
   ptp4l[736.578]: master offset 62941356 s2 freq +1000000 path delay -19455340
   ptp4l[736.578]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
   ptp4l[737.578]: master offset 125882679 s2 freq +1000000 path delay -19455340
   ptp4l[738.578]: master offset 180547214 s2 freq +1000000 path delay -11178467
   ptp4l[739.578]: master offset 245757156 s2 freq +1000000 path delay -13446977
   ptp4l[740.578]: master offset 312455200 s2 freq +1000000 path delay -17200861
   ptp4l[741.578]: master offset 375396363 s2 freq +1000000 path delay -17200861
   ptp4l[742.578]: master offset 438336658 s2 freq +1000000 path delay -17200861
   ptp4l[743.578]: master offset 501278069 s2 freq +1000000 path delay -17200861
   ptp4l[744.578]: master offset 564115361 s2 freq +1000000 path delay -17096547
   ptp4l[745.578]: master offset 627056756 s2 freq +1000000 path delay -17096547
   ptp4l[746.578]: master offset 689998242 s2 freq +1000000 path delay -17096547
   ptp4l[747.578]: master offset 752939650 s2 freq +1000000 path delay -17096547
   ptp4l[748.579]: master offset 814577006 s2 freq +1000000 path delay -15792325
   ptp4l[749.579]: master offset 875083075 s2 freq +1000000 path delay -13356938
   ptp4l[750.579]: master offset 938026244 s2 freq +1000000 path delay -13356938
   ptp4l[751.579]: master offset 998707404 s2 freq +1000000 path delay -11099094
   ptp4l[752.579]: master offset 1061648116 s2 freq +1000000 path delay -11099094
   ptp4l[753.579]: master offset 1124589375 s2 freq +1000000 path delay -11099094
   ptp4l[754.579]: master offset 1186676330 s2 freq +1000000 path delay -10245257
   ptp4l[755.579]: master offset 1247604905 s2 freq +1000000 path delay -8233205
   ptp4l[756.579]: master offset 1310545589 s2 freq +1000000 path delay -8233205
   ptp4l[757.579]: master offset 1373800861 s2 freq +1000000 path delay -8547853

2. log on the master (NXP) is

[root@OpenIL:~]# ptp4l -i eth0 -m
ptp4l[968.482]: selected /dev/ptp0 as PTP clock
ptp4l[968.484]: driver changed our HWTSTAMP options
ptp4l[968.484]: tx_type 1 not 1
ptp4l[968.484]: rx_filter 1 not 12
ptp4l[968.485]: port 1: INITIALIZING to LISTENING on INITIALIZE
ptp4l[968.485]: port 0: INITIALIZING to LISTENING on INITIALIZE
ptp4l[968.485]: port 1: link up
ptp4l[975.134]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[975.135]: selected best master clock 00049f.fffe.ef0606
ptp4l[975.135]: assuming the grand master role

There is no difference in the offset and delay values. I am still getting high values as you can see in the log.

[saiguvvala]

With respect to this:

How to access, modify, update the hardware clock of this NXP device. Please share the steps.

You can interact with the /dev/ptp0 clock using a rich kernel UAPI in C, or use the phc_ctl program from the linuxptp suite (or see how phc_ctl does that).

Can you tell me how to do this a little more elaborately? I am not able to follow.

On a further note, the two devices I use are set on two different times by default: Oct 3 2016 and Jan 1 1970. Is there any way that I can bring them both to the current date?

[vladimiroltean]

I will have another look at this tomorrow.
I agree it is also strange that there is no RTC on the board to keep time persistently, especially since there is a coin cell battery in there. Will double check that.

[vladimiroltean]

I didn't realize that you were testing the LS1021A-TSN board in master mode only. The patch I suggested you to revert was for slave mode only (offset correction). That explains why you see no difference.
But I am able to synchronize an e1000e slave to the LS1021A-TSN master with no problem. Maybe there are limitations in the AM57xx as to how much initial offset it can correct. If the times are closer do they sync?

So your question is how to modify the PHC time manually? Have you tried something like this?

$ phc_ctl /dev/ptp0 set 1555000000
phc_ctl[4128.703]: set clock time to 1555000000.000000000 or Thu Apr 11 16:26:40 2019

I still don't have an answer for the RTC question.