Update TI bits in linux-iot2050

[jan-kiszka]

Align the BSP kernel regarding its TI downstream or backport bits with latest ti-linux. Also update the prueth firmware along that. Should resolve #368 and supersedes #374.

This does not including a stable update of the underlying CIP kernel yet as we are waiting for a recent cip-rt release.

[attila-hannibal]

Dear All!

We tried the Siemens reference image (Chao's branch). The eno1 interface was requested to use DHCP. After several power cycles the eno1 couldn't get IP address. Based on the statistics it seems this the well known issue, that the RX packets got stuck at driver level and couldn't reach the higher levels. Long story short: the DHCP responses (RX packets) are lost in the driver and the Linux cannot configure the IP.

The statistics in the ethtool dump shows non-zero RX counters. However ifconfig / netstat shows zero RX bytes:
ethtool.txt
ifconfig.txt
journalctl-b.log
netstat.txt

Will try Jan's branch variant

[attila-hannibal]

I tried a clean build from jan/kernel-update branch

The network interfaces don't work at all.
After 30 seconds a kernel dump appears on the debug console:
...
[ 136.998977] ------------[ cut here ]------------
[ 137.003638] NETDEV WATCHDOG: eno2 (icssg-prueth): transmit queue 0 timed out
[ 137.010788] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:467 dev_watchdog+0x314/0x320
[ 137.019038] Modules linked in: ti_am335x_adc kfifo_buf irq_pruss_intc rfkill icssg_prueth pru_rproc icss_iep ptp cp210x pps_core usbserial ti_k3_r5_remoteproc ti_cal videobuf2_dma_contig ti_am335x_tscadc v4l2_fwnode videobuf2_memops pci_endpoint_test videobuf2_v4l2 videobuf2_common pruss at24 optee_rng rng_core fuse ip_tables x_tables ipv6
[ 137.049140] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.10.145-cip17 #1
[ 137.055739] Hardware name: SIMATIC IOT2050 Advanced PG2 (DT)
[ 137.061388] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 137.067382] pc : dev_watchdog+0x314/0x320
...
and then endlessly the following comes:
...
[ 180.006989] icssg-prueth icssg0-eth eno2: xmit timeout
[ 184.870978] icssg-prueth icssg0-eth eno2: xmit timeout
[ 189.990981] icssg-prueth icssg0-eth eno2: xmit timeout
...

This version is even worse, please double check at your side

logs attached:
console.txt

[jan-kiszka]

Thanks for reporting. I haven't seen this on any boards here so far. Same issue when going two commits backward (70ebcfa)?

[jan-kiszka]

@attila-hannibal please also check if a specific element of your network infrastructure contributes to this (eg. as specific switch, compared to cross-links).

[attila-hannibal]

Hello @jan-kiszka

I tried the mentioned commit 70ebcfa but the behaviour is the same, eno* interfaces don't work, kernel dump appears after 1-2 minutes
I had a guess that maybe the TI pruss firmware is too new, so I made a version rollback "08.06.00.001" -> "08.02.00.002", but did not help

[jan-kiszka]

And if you leave out the firmware update completely? In my tests, our current firmware still worked.

BTW, please also explore my other question if your network infrastructure influences this. I have no luck reproducing it, boxes run for hours with all versions.

[attila-hannibal]

By using the binary artefact from: https://github.com/siemens/meta-iot2050/suites/10438058123/artifacts/516591580

The only change besides the root password config at first startup I modified the /etc/network/interfaces file:

# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source /etc/network/interfaces.d/*

auto eno1
iface eno1 inet dhcp

auto eno2
iface eno2 inet static
address 192.168.214.230
netmask 255.255.255.0

interfaces does not work, kernel dumps after while
note: during the ping test I pulled out then plugged in the UTP cable
console.txt

[jan-kiszka]

Ok, we can rule out build issues on your side.

Some more things to rule out still: please have a look at my other suggestions.

[attila-hannibal]

Using the Chao's branch as source
The iot2050 eno2 was directly connected to a workstation PC using static IP's

• iot2050 IP: 192.168.214.230
• workstation IP: 192.168.214.20
  This time eno1 was not connected.
  test: pinging from iot2050 to workstation
  The iot2050 tries to resolve the MAC for the IP, this is answered from the PC, see the attached capture.
  arp.zip
  Since the ARP responses are lost, the iot2050 continues the resolution.
  The ARP table remains empty:
  root@iot2050-debian:~# arp
  Address HWtype HWaddress Flags Mask Iface
  192.168.214.20 (incomplete) eno2

The ifconfig shows no RX data
eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.214.230 netmask 255.255.255.0 broadcast 192.168.214.255
inet6 fe80::8ef3:19ff:fe6c:ee42 prefixlen 64 scopeid 0x20
ether 8c:f3:19:6c:ee:42 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2872 bytes 809966 (790.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

The dmesg has the "icssg-prueth icssg0-eth eno2: timeout waiting for command done" line

So the error is present with simple point-to-point connection

[jan-kiszka]

Please don't change two variables a the same time (here: sources AND network setup).

Would still like to see

• a test with the firmware from current master and basic driver update (69fae11)
• a complete tcpdump taken on the device

In addition, please confirm

• issue is gone when using current master
• a second device does expose the same problem

[attila-hannibal]

After the many clean recompilation I recognized that we may use a different SW than the "Chao's branch", it should be an older variant of your master branch. I guess in the past the image to be written was iot2050-image-example-iot2050-debian-iot2050.wic.img (with .img suffix) and the current build creates an other filename without .img suffix.
When I did the git branching and started the build, the old file remained in the build directory and I burned the old one to the USB stick.
The reason I'm thinking the kernel variant is 5.10.104-cip3 in the logs that was already replaced in October and both of your dev branches use 5.10.145 and the master, too.
Sorry it was my mistake

I have the same Ethernet ports non-working issue with the Chao branch, too.

Let me example our test setups, we have two iot2050 instances

• the first device is installed in our testlab, that does not use the Chao's source, but and older variant of the master branch compiled on 21st September 2022. This one produces the incoming RX packet loss in the driver level. This issue happens randomly (this is our original problem), but when it occurs, no packet is transferred to higher level, not at all. So the RX side got stalled right after the beginning, not just after X minutes or so. This setup is normally connected to switches but for last time we used a direct connection to the workstation. From this perspective there was only one change. Here we changed only the networking to simplify the environment, no DHCP, no switch, just direct connection to PC using static IP
• the second device is at my desk where I connect the eno1 interface to corporate network that supports DHCP, while the eno2 is static IP and uses a direct connection to my laptop. When I'm switching the branches I use this device and this setup produces the kernel dump, this setup really uses the latest build compiled from scratch. The non-working Ethernet ports happen by using jan/kernel-update and chao/catch-up-icssg-driver branches. In case I use the actual master branch state (e2f9367) the network interfaces are working in terms of basic RX-TX data, ping works. Not sure about the "original problem" written above, we need to perform many restarts

will do the logs you requested later.

[attila-hannibal]

we may start an other approach.
When the "stalled Ethernet port" problem happens we have this log in the dmesg: "icssg-prueth icssg0-eth eno2: timeout waiting for command done".
This error is written in the "emac_set_port_state()" function: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/net/ethernet/ti/icssg_config.c?h=ti-linux-5.10.y#n580
The function sends 4 commands (each command is an uint32_t) to the R30 / R32 register addresses, then polls the registers if the TI firmware on the other end cleared them or not. When any of the 4 registers is not cleared the driver prints this error, this is we get randomly.

So based the symptom it seems the TI firmware running on the PRU got stock somehow and cannot handle the commands.

Do you know if we can have some debug possibility to verify if the firmware running on the PRU is "healthy" or have some issues?

[attila-hannibal]

we may start an other approach. When the "stalled Ethernet port" problem happens we have this log in the dmesg: "icssg-prueth icssg0-eth eno2: timeout waiting for command done". This error is written in the "emac_set_port_state()" function: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/net/ethernet/ti/icssg_config.c?h=ti-linux-5.10.y#n580 The function sends 4 commands (each command is an uint32_t) to the R30 / R32 register addresses, then polls the registers if the TI firmware on the other end cleared them or not. When any of the 4 registers is not cleared the driver prints this error, this is we get randomly.

So based the symptom it seems the TI firmware running on the PRU got stock somehow and cannot handle the commands.

Do you know if we can have some debug possibility to verify if the firmware running on the PRU is "healthy" or have some issues?

This a command register dump when the problem occurred: the first value should be 0xffff0000 (EMAC_NONE), too, but it stays 0xffbb0000

root@DM8CF3196AE6B5:~# devmem2 0xb0005ac
/dev/mem opened.
Memory mapped at address 0xffffbebe5000.
Read at address  0x0B0005AC (0xffffbebe55ac): 0xFFBB0000
root@DM8CF3196AE6B5:~# ^C
root@DM8CF3196AE6B5:~# devmem2 0xb0005b0
/dev/mem opened.
Memory mapped at address 0xffffb2770000.
Read at address  0x0B0005B0 (0xffffb27705b0): 0xFFFF0000
root@DM8CF3196AE6B5:~# devmem2 0xb0005b4
/dev/mem opened.
Memory mapped at address 0xffff809b4000.
Read at address  0x0B0005B4 (0xffff809b45b4): 0xFFFF0000
root@DM8CF3196AE6B5:~# devmem2 0xb0005b8
/dev/mem opened.
Memory mapped at address 0xffff90003000.
Read at address  0x0B0005B8 (0xffff900035b8): 0xFFFF0000

[jan-kiszka]

I'm trying my contacts to TI. Maybe we will get some further hints how to analyze this best.

[attila-hannibal]

Some further testing result:
SW level: jan/kernel-update, this determines the TI pruss firmware: 08.06.00.001

test scenarios:

1. eno1: dhcp, eno2: static, UTP cables connected at startup -> none of the interfaces are working
2. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno1, then eno2 -> eno1 does not work, eno2 works
3. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno2, then eno1 -> eno2 works, eno1 does not work
4. eno1: dhcp, eno2: static, only eno2 UTP cables is connected at startup -> eno2 works
5. eno1: static, eno2: dhcp, UTP cables connected at startup -> both interfaces are working
6. eno1: unconfigured, eno2: static, UTP cables connected at startup, after login configure eno1 (command: "ifconfig eno1 0.0.0.0 0.0.0.0 && dhclient") -> both interfaces are working
7. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.02.00.002 -> none of the interfaces are working
8. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.00.00.004 -> none of the interfaces are working

I think we can narrow down the issue: during the boot when the eno1 is connected already and configured as dhcp, there should be some kind of deadlock that causes the issue. It seems only eno1 has such a behaviour (see scenario 5) ), also when the interface configured "late" (see scenario 6) ) it works. That may answers why our Linux image has the non-persistent behaviour, because something depends on the timing and/or boot step sequence.

the 4 command registers for scenario 1) seems all fine

root@iot2050-debian:~# memtool md 0xb0005ac
0b0005ac: ffff0000 ffff0000 ffff0000 ffff0000                ................
0b0005bc: 70020000 00000000 00000000 00000000                ...p............
0b0005cc: 00000000 00000000 00000000 00000000                ................

[applea9]

@attila-hannibal Do you have tested with scenario that both eno1 and eno2 are static? I met issue with this scenario before.

[BaochengSu]

@jan-kiszka I found the commit 4a80b17 really helps a lot from the rebase suffering, however, irrelevant to this issue.

So I've just extracted it to a separate PR #414.

[BaochengSu]

Some further testing result: SW level: jan/kernel-update, this determines the TI pruss firmware: 08.06.00.001

test scenarios:

1. eno1: dhcp, eno2: static, UTP cables connected at startup -> none of the interfaces are working
2. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno1, then eno2 -> eno1 does not work, eno2 works
3. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno2, then eno1 -> eno2 works, eno1 does not work
4. eno1: dhcp, eno2: static, only eno2 UTP cables is connected at startup -> eno2 works
5. eno1: static, eno2: dhcp, UTP cables connected at startup -> both interfaces are working
6. eno1: unconfigured, eno2: static, UTP cables connected at startup, after login configure eno1 (command: "ifconfig eno1 0.0.0.0 0.0.0.0 && dhclient") -> both interfaces are working
7. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.02.00.002 -> none of the interfaces are working
8. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.00.00.004 -> none of the interfaces are working

I think we can narrow down the issue: during the boot when the eno1 is connected already and configured as dhcp, there should be some kind of deadlock that causes the issue. It seems only eno1 has such a behaviour (see scenario 5) ), also when the interface configured "late" (see scenario 6) ) it works. That may answers why our Linux image has the non-persistent behaviour, because something depends on the timing and/or boot step sequence.

the 4 command registers for scenario 1) seems all fine

root@iot2050-debian:~# memtool md 0xb0005ac
0b0005ac: ffff0000 ffff0000 ffff0000 ffff0000                ................
0b0005bc: 70020000 00000000 00000000 00000000                ...p............
0b0005cc: 00000000 00000000 00000000 00000000                ................

Hi @attila-hannibal,

We've tried these scenarios with the action build https://github.com/siemens/meta-iot2050/actions/runs/3948486240, however none is reproduced.

So there might be something nuanced different between our setup and yours which lead to the non-producible. So it would be helpful if we have below information (some of them just a double confirmation to make sure we are on the same page):

• The reproduce rate for these scenarios. From your description I get a feeling that it was very easy to reproduce the issue within the scenarios, i.e. you don't have to perform lots of reboot to trigger the issue.
• Which hardware version? We were trying on two PG2 advanced iot2050, with 1YA2 and FS: 04,
• FW version in your setup. We've tried with two different firmware, one is the released v01.03.01 firmware, the other is the firmware built from the above action.
• Is there any additional peripheral such as PCIE card, DP monitor, etc. in your setup? A picture of your device setup could be more helpful. We only have the usb-sd-card reader and the FTDI uart cable connected in our setup.
• What is the network segments for both eno1 and eno2? We are using two different segments for static (192.168.200.0/24) and dhcp(192.168.1.0/24).
• Is there any other images in the eMMC or SD card slot? This is just in case to avoid accidentally booting to another image. You can check the current boot target via fw_printenv | grep boot_targets, or you can check the image build ID from /etc/os-release file to make sure the current booting image is with build id fd691b7.
• For the static profile that directly connected to the PC, what kind of Ethernet port are you using on the PC side, is it the PC native port or some USB network adapter? We've tried both.
• How do you define not working in the issue? is it not pinging? or DHCP IP not showing up? or static IP losing? We were using the ping command to determine the working/non-working.
• What is your DHCP environment? We think it is your company DHCP?
• What is the network tool you are using? We are using nmtui and we think you are using the /etc/network ?

[attila-hannibal]

Some further testing result: SW level: jan/kernel-update, this determines the TI pruss firmware: 08.06.00.001
test scenarios:

1. eno1: dhcp, eno2: static, UTP cables connected at startup -> none of the interfaces are working
2. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno1, then eno2 -> eno1 does not work, eno2 works
3. eno1: dhcp, eno2: static, UTP cables not connected startup, after login connect eno2, then eno1 -> eno2 works, eno1 does not work
4. eno1: dhcp, eno2: static, only eno2 UTP cables is connected at startup -> eno2 works
5. eno1: static, eno2: dhcp, UTP cables connected at startup -> both interfaces are working
6. eno1: unconfigured, eno2: static, UTP cables connected at startup, after login configure eno1 (command: "ifconfig eno1 0.0.0.0 0.0.0.0 && dhclient") -> both interfaces are working
7. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.02.00.002 -> none of the interfaces are working
8. eno1: dhcp, eno2: static, UTP cables connected at startup + using TI pruss firmware: 08.00.00.004 -> none of the interfaces are working

I think we can narrow down the issue: during the boot when the eno1 is connected already and configured as dhcp, there should be some kind of deadlock that causes the issue. It seems only eno1 has such a behaviour (see scenario 5) ), also when the interface configured "late" (see scenario 6) ) it works. That may answers why our Linux image has the non-persistent behaviour, because something depends on the timing and/or boot step sequence.
the 4 command registers for scenario 1) seems all fine

root@iot2050-debian:~# memtool md 0xb0005ac
0b0005ac: ffff0000 ffff0000 ffff0000 ffff0000                ................
0b0005bc: 70020000 00000000 00000000 00000000                ...p............
0b0005cc: 00000000 00000000 00000000 00000000                ................

Hi @attila-hannibal,

We've tried these scenarios with the action build https://github.com/siemens/meta-iot2050/actions/runs/3948486240, however none is reproduced.

So there might be something nuanced different between our setup and yours which lead to the non-producible. So it would be helpful if we have below information (some of them just a double confirmation to make sure we are on the same page):

• The reproduce rate for these scenarios. From your description I get a feeling that it was very easy to reproduce the issue within the scenarios, i.e. you don't have to perform lots of reboot to trigger the issue.
• Which hardware version? We were trying on two PG2 advanced iot2050, with 1YA2 and FS: 04,
• FW version in your setup. We've tried with two different firmware, one is the released v01.03.01 firmware, the other is the firmware built from the above action.
• Is there any additional peripheral such as PCIE card, DP monitor, etc. in your setup? A picture of your device setup could be more helpful. We only have the usb-sd-card reader and the FTDI uart cable connected in our setup.
• What is the network segments for both eno1 and eno2? We are using two different segments for static (192.168.200.0/24) and dhcp(192.168.1.0/24).
• Is there any other images in the eMMC or SD card slot? This is just in case to avoid accidentally booting to another image. You can check the current boot target via fw_printenv | grep boot_targets, or you can check the image build ID from /etc/os-release file to make sure the current booting image is with build id fd691b7.
• For the static profile that directly connected to the PC, what kind of Ethernet port are you using on the PC side, is it the PC native port or some USB network adapter? We've tried both.
• How do you define not working in the issue? is it not pinging? or DHCP IP not showing up? or static IP losing? We were using the ping command to determine the working/non-working.
• What is your DHCP environment? We think it is your company DHCP?
• What is the network tool you are using? We are using nmtui and we think you are using the /etc/network ?

Hello @BaochengSu

I used the iot2050-example-image.zip as wic image from the build you linked in. I copied it to a 64GB large USB3 stick. I also replaced the SPI flash content with the iot2050-pg2-image-boot.bin from the build to have the same FW/SW as you have.
content of the /etc/network/interfaces files is the same as before:
root@iot2050-debian:~# cat /etc/network/interfaces
source /etc/network/interfaces.d/*

auto eno1
iface eno1 inet dhcp

auto eno2
iface eno2 inet static
address 192.168.214.230
netmask 255.255.255.0

(no other change has been made to you reference image)
Answering your question 1-by-1:

1. the issue with the networking is persistent now. I tried 10 restarts and all the 10 were faulty. The failure is explained in point 8.

2. I'm attaching a picture from the sticker on the machine. we use the same HW variant
   

3. I use now the one from the build artifacts

4. nothing special, I'm attaching a picture. I use the pendrive as boot source / rootfs. The eMMC is erased. I have the 2 UTP cables connected to the networking ports, the debug UART and the 24V power supply
   

5. the eno1 (DHCP) connection should obtain an IP address from network range: 10.23.0.*** , but it does not work, see 8. the eno2 (static) uses 192.168.214.*, see above the /etc/network/interfaces file

6. There is no SD card at all, the eMMC is erased, the system boots from USB stick. I'm attaching the os-release file content:
   release.txt the build is correct!

7. the static connection use an U-Green USB-Ethernet adapter, that has an "AX88179 Gigabit Ethernet" chip on it.

8. "The problem" - that happens always when I use your latest image(s) - the eno1 DHCP connection does not get any IP address, nothing at all. Using the eno2 port the iot2050 cannot ping my laptop (laptop ip: 192.168.214.1), I executed ping on both machines, tcpdump on my machine shows no ARP response:
   14:43:48.621038 ARP, Request who-has 192.168.214.230 tell 192.168.214.1, length 28
   14:43:49.643279 ARP, Request who-has 192.168.214.230 tell 192.168.214.1, length 28
   14:43:50.668756 ARP, Request who-has 192.168.214.230 tell 192.168.214.1, length 28
   14:43:51.690955 ARP, Request who-has 192.168.214.230 tell 192.168.214.1, length 28
   14:43:52.715215 ARP, Request who-has 192.168.214.230 tell 192.168.214.1, length 28
   tcpdump on the iot2050 is attached in the console log:
   debug_console.txt
   on the iot2050 it seems the ARP responses are sent but is not visible on the other side. After a while kernel dump happens and I get the "icssg-prueth icssg0-eth eno2: xmit timeout" periodically
   NOTE: as written before if the eno1 <-> eno2 are swapped in the /etc/network/interface files (and of course I swap the UTP cables). the problem does not appear. So please reproduce exactly the same network setup as I have.

9. The DHCP is provided by the company I work for. What information do you need? See point 8.'s note: if I swap the eno1 and eno2 the DHCP works. Also there are about ~40 machines on this network those are working flawlessly, therefore I think nothing wrong with the DHCP environment.

10. I use the systemd's built-in. See the timeout in the debug logs ( A start job is running for Raise network interfaces...)

Best regards
Attila

[jan-kiszka]

Wait, I missed that so far:

content of the /etc/network/interfaces files is the same as before:
root@iot2050-debian:~# cat /etc/network/interfaces
source /etc/network/interfaces.d/*

auto eno1
iface eno1 inet dhcp

auto eno2
iface eno2 inet static
address 192.168.214.230
netmask 255.255.255.0

We are using Network Manger in the default image. I'm sure if /etc/network/interface.d is properly evaluated at all. And even if: there is also /etc/NetworkManager/system-connections/eno1-default.

[attila-hannibal]

Hello @jan-kiszka

Ok, I confirm the "persistent network issue" is solved by using the correct network-manager.
I cleaned the /etc/network/interfaces file and configured the /etc/NetworkManager/system-connections/ connection-files via the nmtui tool, the system works as expected.

Now we have to go back to our original problem when the network loss happened randomly, and when this problem happens we have the "icssg-prueth icssg0-eth eno2: timeout waiting for command done" in the dmesg log

[jan-kiszka]

Great to hear. Hope the image can now help validating if that issue is gone as well. Please let us know when there are news or further findings/questions.

[lyxsiemens]

After several power cycles the eno1 couldn't get IP address

@attila-hannibal
Could you tell me how did you do the power cycle? by "reboot" command or reset button or power cut?

[jan-kiszka]

@BaochengSu, @AsuraZeng, can we proceed with the MR? It addresses what #374 was fixing, avoids related regressions, and aligns with latest ti-linux (now 08.06.004, as recommended by T). It may just not resolve all issues of the prueth, but that should be shared with ti-linux at this point.

[BaochengSu]

Given that TI is about to release the 8.6 SDK soon, I am ok to proceed with 8.6 catchup.