Hi @Ayush1325! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

There are network echo-server/client samples that too could have been used for testing. Those samples support both TCP and UDP, and either one can be disabled if not needed.

The configs look mostly ok, just wondering why you have CONFIG_NET_BUF_DATA_SIZE=256, default is 128 and it should be fine for 802.15.4 networks.
There is also difference between client/server buffer counts but that should not matter here.

The error message tells that you run out of buffers. So if you are certain that buffer count is enough, then I suspect there is a memory leak (not likely), or you are receiving too much data and the device is not able to process it.

You can use network shell to monitor the buffer count usage. The command is net mem, you need to enable CONFIG_NET_BUF_POOL_USAGE=y. The net allocs can also show detailed information about allocations, you need to enable CONFIG_NET_DEBUG_NET_PKT_ALLOC=y to see that information.

There are network echo-server/client samples that too could have been used for testing. Those samples support both TCP and UDP, and either one can be disabled if not needed.

Well, I used separate applications since I wanted to test across zephyr forks. But yes, maybe it would be better to use samples.

The configs look mostly ok, just wondering why you have CONFIG_NET_BUF_DATA_SIZE=256, default is 128 and it should be fine for 802.15.4 networks. There is also difference between client/server buffer counts but that should not matter here.

The error message tells that you run out of buffers. So if you are certain that buffer count is enough, then I suspect there is a memory leak (not likely), or you are receiving too much data and the device is not able to process it.

While I cannot comment on memory leak, I can confidently say that the device is capable of processing this much data. The Zephyr fork that BeagleBoard has is able to run this example without any problem. In fact I can run this example with rx and tx packet and buffer count set to 16.

Also, after some testing, it seems that I get the Cannot allocate packet on server only if the client is built against upstream Zephyr. Everything works fine if the client is built against the fork (irrespective of the Zephyr source the server is using). So maybe the problem is somewhere in the subg send code?

You can use network shell to monitor the buffer count usage. The command is net mem, you need to enable CONFIG_NET_BUF_POOL_USAGE=y. The net allocs can also show detailed information about allocations, you need to enable CONFIG_NET_DEBUG_NET_PKT_ALLOC=y to see that information.

Here is the output from net allocs and net mem

uart:~$ net allocs
Network memory allocations

memory          Status  Pool    Function alloc -> freed
0x20009020       free      RX   drv_rx_done():356 -> reass_timeout():290
0x20009064       free      RX   drv_rx_done():356 -> reass_timeout():290
0x200090a8       free      RX   drv_rx_done():356 -> reass_timeout():290
0x20009284       free      RX   drv_rx_done():356 -> pkt_alloc_with_buffer():1487
0x20008e44       free      TX   ieee802154_acknowledge():83 -> ieee802154_acknowledge():94
0x200091b8       free      RX   drv_rx_done():356 -> net_icmpv6_input():371
0x20009174       free      RX   drv_rx_done():356 -> net_icmpv6_input():371
0x20009130       free      RX   drv_rx_done():356 -> net_icmpv6_input():371
0x200090ec       free      RX   drv_rx_done():356 -> reass_timeout():290
0x200090a8       free      RX   drv_rx_done():356 -> fragment_add_to_cache():562
0x20009064       free      RX   drv_rx_done():356 -> fragment_add_to_cache():562
0x20009020       free      RX   drv_rx_done():356 -> fragment_add_to_cache():562
uart:~$ net mem
Fragment length 128 bytes
Network buffer pools:
Address         Total   Avail   Name
0x20001278      16      16      RX
0x20001294      16      16      TX
0x2000143c      16      16      RX DATA (rx_bufs)
0x20001474      16      16      TX DATA (tx_bufs)
No external memory pools found.

Hi @Ayush1325 - Would it be possible to add a log message that prints both the time and failure count, and then to create a graph of the failures over time? (this would require incrementing a counter when the error is encountered).

Additionally, it would be best to either test upstream <-> upstream, or fork <-> fork, but mixing versions introduces many more moving parts so a straight comparison becomes more difficult. Perhaps that's the goal here - to find the difference between upstream and the fork that causes this issue.

Definitely important to note, is that upstream has had a number of IEEE 802.15.4 and cc13xx subg radio improvements over time, and they might not have been captured downstream.

If this issue was encountered in the process of performing an OTA update, it will likely not be fixable, because the physical layer (subg) itself is not constant between versions. If that is the case, there may need to be an alternative means of upgrading devices.

@cfriedt Sorry for the late reply, was busy with college exams. Since the packet drops have been somewhat fixed by #69098 , I have updated udp-server and udp-client to use TCP sockets (since that is what I need to use in my actual firmware).

Here are the results for upstream <-> upstream test:

*** Booting Zephyr OS build v3.6.0-364-g668f6d87ec49 ***
[00:00:00.170,135] <dbg> udp_client: main: Socket Created
[00:00:00.399,322] <inf> udp_client: TCP client started
[00:00:03.532,958] <inf> udp_client: Total time: 652482382 ns
[00:00:03.532,989] <inf> udp_client: Time per request: 13049647 ns
[00:00:03.533,020] <inf> udp_client: Time per request: 13 ms

And here are the results for fork <-> fork test:

*** Booting Zephyr OS build bcf-sdk-0.2.1-3386-g75ae14ddc766 ***
[00:00:00.128,051] <dbg> udp_client: main: Socket Created
[00:00:00.431,793] <inf> udp_client: TCP client started
[00:00:02.448,669] <inf> udp_client: Total time: 150090404 ns
[00:00:02.448,699] <inf> udp_client: Time per request: 3001808 ns
[00:00:02.448,730] <inf> udp_client: Time per request: 3 ms

The time per request here is the time it roughly takes to send and recv an int over tcp. I am using 50 reqs to calcualte the average.

As you can see, it is pretty evident that there is a large performance gap here. Will inspect the diffs in more detail to see if I can find the cause.

There is no concern for OTA or anything like that since the firmware is completely experimental and new, so no devices in the wild with it.

Hi @Ayush1325 - I hope you did well on your exams 👍

Total time / 50 is probably not the best way to measure radio driver performance because it's effectively measuring the entire chain of software calls.

Unless you're trying to measure the performance of the 15.4 subsystem, but that would likely be independent of the driver in use.

Aside from that, I have other questions:

• What versions of Zephyr (forks) are used in the comparison?
• What performance exactly are we measuring?
• Are assertions enabled?
• Are there additional syscalls in the hot path of version B vs version A?
• is there sufficient buffer space allocated?

If you're thinking about performance indicators for the radio driver, it would be best to look at statistics such as

• dropped packets due to buffer overflow /slow processing
• dropped packets due to error (assuming the RF channel is free of noise, this would indicate bit error at the source, so logical errors)
• amount of time between when a packet is submitted for TX and when the packet is actually transmitted
• amount of time between when a packet is received (isr) and when it is moved to the subsystem
• whether any protocol-specific deadlines are missed

Hi @Ayush1325 - I hope you did well on your exams 👍

Total time / 50 is probably not the best way to measure radio driver performance because it's effectively measuring the entire chain of software calls.

Unless you're trying to measure the performance of the 15.4 subsystem, but that would likely be independent of the driver in use.

Aside from that, I have other questions:

• What versions of Zephyr (forks) are used in the comparison?

The fork is a version of Zephyr 3.4 + a lot of additional patches. The fork has deviated too much from upstream and is in no state for rebase (all the previous rebase attempts have rendered the fork in non-functional state). I have tried to cherry pick and upstream all the patches that should be required, but seems like something is missing.

• What performance exactly are we measuring?

Round trip time of TCP packet. In my firmware, I want to send some packets which will then cause the node to respond.

• Are assertions enabled?

CONFIG_ASSERT is set to false.

• Are there additional syscalls in the hot path of version B vs version A?

I don't think so. I am using the exact same udp-server and udp-client application.

• is there sufficient buffer space allocated?

Yes, I do not get any buffer space warnings after setting them to 64. Kind of overkill considering I am only sending 50 messages.

If you're thinking about performance indicators for the radio driver, it would be best to look at statistics such as

• dropped packets due to buffer overflow /slow processing
• dropped packets due to error (assuming the RF channel is free of noise, this would indicate bit error at the source, so logical errors)
• amount of time between when a packet is submitted for TX and when the packet is actually transmitted
• amount of time between when a packet is received (isr) and when it is moved to the subsystem
• whether any protocol-specific deadlines are missed

Thanks will try the above things. Are there any helpers or shell stuff that can help profile the different net layers?

So I switched to using zperf sample instead of udp-server and udp-client apps.

Testing UDP:

uart:~$ zperf udp upload 2001:db8::2 5001 10 50 1M
Remote port is 5001
Connecting to 2001:db8::2
Duration:       10.00 s
Packet size:    50 bytes
Rate:           1000 kbps
Starting...
ping 2001:db8::2 timeout
Rate:           1.00 Mbps
Packet duration 390 us
-
Upload completed!
Statistics:             server  (client)
Duration:               10.17 s (10.02 s)
Num packets:            899     (900)
Num packets out order:  0
Num packets lost:       0
Jitter:                 3.03 ms
Rate:                   35 Kbps (35 Kbps)

Testing TCP:

uart:~$ zperf tcp upload 2001:db8::2 5001 2 50
Remote port is 5001
Connecting to 2001:db8::2
Duration:       2.00 s
Packet size:    50 bytes
Rate:           10 kbps
Starting...
ping 2001:db8::2 timeout
-
Upload completed!
Duration:       2.07 s
Num packets:    101
Num errors:     0 (retry or fail)
Rate:           19 Kbps

Anything higher than 2 sec in TCP triggers ieee802154_cc13xx_cc26xx_subg: Cannot allocate packet errors.

@Ayush1325 - a bisect will allow you to find the specific commit that breaks your performance criteria.

I would suggest finding an upstream commit where your performance works as expected, and then find a recent upstream commit where your performance does not work as expected, and then proceed with a git bisect.

If your bisection is successful (i.e. you are able to narrow down the bug to a specific commit), it would probably help to diagnose your issue.

@Ayush1325 - a bisect will allow you to find the specific commit that breaks your performance criteria.

I would suggest finding an upstream commit where your performance works as expected, and then find a recent upstream commit where your performance does not work as expected, and then proceed with a git bisect.

If your bisection is successful (i.e. you are able to narrow down the bug to a specific commit), it would probably help to diagnose your issue.

Well, I don't think that's going to be possible. I added the subg support to upstream (by cherry picking commits from fork) a few months ago and performance was pretty much the same as right now. So upstream does not have any commit that works better.

I can only say that the hardware is capable of much better performance. Eg: Running zperf in fork <-> fork setup gives rate of 67 Kbps. So around a difference of 100% in performance.

Alright, I can confirm that 356c8cbe63ae01b3ab438382639d25bb418a0213 with some patches to enable networking for bcf (combination of #69098, #67229, #65048) does give 67 kbps.
Let's get to some bisecting now.

After series of git bisects, it seems the bad commit in question is: a94877b . Causes drop in udp performance from 67 kbps to 38 kbps

Hi @Ayush1325 !

Sorry that I chime in only now, I was ill and couldn't read my messages before.

The commit you point to fixes a bug in the CC13/26xx SubG driver which announced a "hard MAC" CSMA/CA capability which it didn't implement. The change enables the L2 stack's "soft MAC" CSMA/CA implementation instead.

What you're observing might just be standard-conforming random CSMA/CA backoffs that intentionally reduce throughput to enhance co-existence. This can be "fixed" by switching CSMA/CA off. The fact that some packages are delivered without additional latency may just be due to the fact that CSMA/CA chooses not to backoff at random intervals.

I did extensive regression testing on that change, see #58439 and above all #58439 (comment).

There you'll also find an in-depth explanation and demo of the increased latencies due to CSMA/CA as well as sample configurations with CSMA/CA turned off that prove no regression in throughput.

Not sure why you see package drops, though? Some timeout maybe that does not take potential max backoffs into account?

Please let me know if I'm misinterpreting something here. I didn't have time to look deeply into all prior comments.

Hope this helps?

@fgrandel Thanks for the explanation. Switching to ALOHA seems get me around 56 kbps in the zperf test. Not completely sure how that will play with greybus though so let's see.

Not sure why you see package drops, though? Some timeout maybe that does not take potential max backoffs into account?

It has been fixed by #69098

While the previous bisect, I also found another problem in zperf. It seems one of the commits in #62942 breaks the ping that zperf upload test does. So I get ping 2001:db8::2 timeout.
Here is the full log of what I am talking about:

uart:~$ zperf udp upload 2001:db8::2 5001 10 50 1M
Remote port is 5001
Connecting to 2001:db8::2
Duration:       10.00 s
Packet size:    50 bytes
Rate:           1000 kbps
Starting...
ping 2001:db8::2 timeout
Rate:           1.00 Mbps
Packet duration 390 us
-
Upload completed!
Statistics:             server  (client)
Duration:               10.17 s (10.02 s)
Num packets:            899     (900)
Num packets out order:  0
Num packets lost:       0
Jitter:                 3.03 ms
Rate:                   35 Kbps (35 Kbps)

While the previous bisect, I also found another problem in zperf. It seems one of the commits in #62942 breaks the ping that zperf upload test does. So I get ping 2001:db8::2 timeout.

Hi @Ayush1325 , thank you so much for tracking down troubles with BeagleConnect Freedom. I personally know several people who'll be very grateful to you!

It would be very helpful if you could identify the specific commit that breaks zperf. @jukkar and myself will probably be able to better support you if you hone in a bit more on the offending commit. I reviewed that PR and couldn't see any obvious problem there. The change looks very reasonable to me.

@fgrandel Thanks for the explanation. Switching to ALOHA seems get me around 56 kbps in the zperf test.

This sounds reasonable if you have ACK switched on. If you switch ACK off you should see ~120 Kbps throughput if you optimize package fragmentation, again see my test results. Faster than that is physically almost impossible due to the SubG SUN PHY's low bit rates. The rest is due to MAC/PHY header overhead and 802.15.4 inter-frame space (IFS) requirements.

Not completely sure how that will play with greybus though so let's see.

I know little about greybus protocol but it probably depends on the throughput and package size required by your remote driver. In any case 802.15.4 over SubG SUN PHY would not usually be your ideal protocol stack as it is not designed for high throughput and reliability. The greybus demos I've seen usually had a fast and reliable ethernet connection. Maybe @cfriedt or @vaishnavachath know more about this?

In any case upper layers will have to guarantee delivery or tolerate packet loss as 802.15.4 (like any other radio protocol) will never give you 100% reliability no matter how much you tune your PHY/MAC protocol settings.

Maybe @cfriedt or @vaishnavachath know more about this?

In any case upper layers will have to guarantee delivery or tolerate packet loss as 802.15.4 (like any other radio protocol) will never give you 100% reliability no matter how much you tune your PHY/MAC protocol settings.

Exactly - TCP or a similar protocol with retries and receipt acknowledgement needs to be used. TCP is the simplest, but if using UDP, then Thread or Matter might also work.

It would be very helpful if you could identify the specific commit that breaks zperf. @jukkar and myself will probably be able to better support you if you hone in a bit more on the offending commit. I reviewed that PR and couldn't see any obvious problem there. The change looks very reasonable to me.

So I did do a git bisect, but the thing is that a lot of commits there seem to prevent zperf from building. So I am left with this:

❯ git bisect skip
There are only 'skip'ped commits left to test.
The first bad commit could be any of:

19d1dd7
3baaa72
44151d5
c473626
e3fb634
fe5638c
1927308
8f74148
fa4e978

Also, the commits surrounding this one seem to have other problems due to which I am getting 1-2kbs speed, but I was only looking for the commit that breaks the ping timeout for now.

@jukkar Could you help here? @Ayush1325 tracked down zperf troubles to the ping API refactoring, it seems. Just by inspecting your code I couldn't see anything wrong with it, but maybe you have an idea?