net_buf: support large size #97440

MarkWangChinese · 2025-10-13T08:36:13Z

change uint16_t as size_t to support large size of data.

The user case:
In USB, the net_buf is used. usb msc may transfer 64KB (65536) data in one net_buf. So the uint16_t can't handle 65536.

include/zephyr/net_buf.h

pdgendt

Please see https://github.com/zephyrproject-rtos/zephyr/pull/97440/files#r2425645621
Besides that, the following functions/structs need to be updated too:

uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
struct net_buf_simple_state;
static inline void net_buf_simple_save(const struct net_buf_simple *buf,
				       struct net_buf_simple_state *state);
struct net_buf;

pdgendt · 2025-10-13T09:55:06Z

Make sure the tests in tests/lib/net_buf work with this new option, and that this is included in CI.

include/zephyr/net_buf.h

MarkWangChinese · 2025-10-14T02:16:27Z

Make sure the tests in tests/lib/net_buf work with this new option, and that this is included in CI.

Tested tests\lib\net_buf\buf on one NXP platform (lpc55s69), the result is OK as follow. I tried to test tests\lib\net_buf\buf_simple too, but it can't be built successfully.

===================================================================
PROJECT EXECUTION SUCCESSFUL
*** Booting Zephyr OS build v4.2.0-5562-g3a3a6a6c7e65 ***
Running TESTSUITE net_buf_tests
===================================================================
START - test_net_buf_1
 PASS - test_net_buf_1 in 0.001 seconds
===================================================================
START - test_net_buf_2
 PASS - test_net_buf_2 in 0.001 seconds
===================================================================
START - test_net_buf_3
 PASS - test_net_buf_3 in 0.001 seconds
===================================================================
START - test_net_buf_4
 PASS - test_net_buf_4 in 0.001 seconds
===================================================================
START - test_net_buf_big_buf
 PASS - test_net_buf_big_buf in 0.001 seconds
===================================================================
START - test_net_buf_byte_order
 PASS - test_net_buf_byte_order in 0.001 seconds
===================================================================
START - test_net_buf_clone_no_ref_count
 PASS - test_net_buf_clone_no_ref_count in 0.001 seconds
===================================================================
START - test_net_buf_clone_ref_count
 PASS - test_net_buf_clone_ref_count in 0.001 seconds
===================================================================
START - test_net_buf_clone_reference_counted_zero_sized_buffer
 PASS - test_net_buf_clone_reference_counted_zero_sized_buffer in 0.001 seconds
===================================================================
START - test_net_buf_clone_user_data
 PASS - test_net_buf_clone_user_data in 0.001 seconds
===================================================================
START - test_net_buf_comparison
 PASS - test_net_buf_comparison in 0.001 seconds
===================================================================
START - test_net_buf_fixed_append
 PASS - test_net_buf_fixed_append in 0.001 seconds
===================================================================
START - test_net_buf_fixed_pool
 PASS - test_net_buf_fixed_pool in 0.001 seconds
===================================================================
START - test_net_buf_linearize
 PASS - test_net_buf_linearize in 0.001 seconds
===================================================================
START - test_net_buf_multi_frags
 PASS - test_net_buf_multi_frags in 0.001 seconds
===================================================================
START - test_net_buf_user_data
 PASS - test_net_buf_user_data in 0.001 seconds
===================================================================
START - test_net_buf_user_data_copy
 PASS - test_net_buf_user_data_copy in 0.001 seconds
===================================================================
START - test_net_buf_var_pool
 PASS - test_net_buf_var_pool in 0.001 seconds
===================================================================
START - test_net_buf_var_pool_aligned
 PASS - test_net_buf_var_pool_aligned in 0.001 seconds
===================================================================
TESTSUITE net_buf_tests succeeded

------ TESTSUITE SUMMARY START ------

SUITE PASS - 100.00% [net_buf_tests]: pass = 19, fail = 0, skip = 0, total = 19 duration = 0.019 seconds
 - PASS - [net_buf_tests.test_net_buf_1] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_2] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_3] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_4] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_big_buf] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_byte_order] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_no_ref_count] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_ref_count] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_reference_counted_zero_sized_buffer] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_clone_user_data] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_comparison] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_fixed_append] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_fixed_pool] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_linearize] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_multi_frags] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_user_data] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_user_data_copy] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_var_pool] duration = 0.001 seconds
 - PASS - [net_buf_tests.test_net_buf_var_pool_aligned] duration = 0.001 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION SUCCESSFUL

pdgendt · 2025-10-14T06:35:50Z

include/zephyr/net_buf.h


+#if defined(CONFIG_NET_BUF_LARGE_SIZE)
+/** @brief net buf size type */
+typedef size_t net_buf_size_t;


If we look at #96903, should we prefer uint32_t?

Not blocking, just a question.

Comment addressed

jfischer-no · 2025-10-14T11:05:42Z

include/zephyr/net_buf.h

+#if defined(CONFIG_NET_BUF_LARGE_SIZE)
+/** @brief net buf size type */
+typedef size_t net_buf_size_t;
+#else
+/** @brief net buf size type */
+typedef uint16_t net_buf_size_t;
+#endif
+


I do not think that it makes any sense to have it configurable, nor use size_t. This will introduce a lot of mess because no one would know what actual maximal size is. Either it remains uint16_t, or we change it to uint32_t. And USB in Zephyr for sure does not require it to be larger than uint16_t.

And USB in Zephyr for sure does not require it to be larger than uint16_t.

Could you explain more about it?
Windows will write/read 64KiB data to/from usb disk. If the buffer of msc is configured as 64KiB and give the 64KiB buffer to controller at one time, then the controller can transfer all the 64 KiB bytes without host stack and msc class involved.

I do not think that it makes any sense to have it configurable, nor use size_t. This will introduce a lot of mess because no one would know what actual maximal size is.

@jfischer-no I agree about not using size_t, since it needs to be more explicit than that, however no code should be using the struct member types for doing any "maximal size" derminations. net_buf offers separate APIs like net_buf_max_len(), net_buf_headroom() and net_buf_tailroom() to determine how much storage space is available.

If the entity (controller?) reading or writing the data could support scatter-gather API like readv/writev, then we would not need to do any of these changes as it would be possible to setup net_buf so that they are chained together and the reads/writes would happen to correct net_buf "automatically". Anyway, just some idea to think about.

jhedberg · 2025-10-14T11:57:27Z

include/zephyr/net_buf.h

-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
+net_buf_size_t net_buf_simple_max_len(const struct net_buf_simple *buf);


I think this is an oversight in the API. The public API has strived to keep the uint16_t as much of an internal detail as possible, while using size_t for both input and return parameters. I think this one should probably just be changed to size_t to be consistent with the rest of the API.

As for the point @jfischer-no raised regarding the predictability of what the buffer can actually contain, I agree that internal (within the structure) a fixed size type (e.g. uint32_t) makes sense.

Thanks @jhedberg , I do the changes, please help to review whether it is your thought.

jhedberg · 2025-10-16T09:19:24Z

include/zephyr/net_buf.h

-	uint16_t len;
+	uint32_t len;

 	/** Amount of data that net_buf_simple#__buf can store. */
-	uint16_t size;
+	uint32_t size;


I think you misunderstood me a little bit. I wasn't proposing to change this unconditionally to uint32_t, rather you'd need some explicit enabling of the larger size, through Kconfig, like you had before. Otherwise you're causing memory overflows to platforms what are very constrained (we have at least some build configurations for 16k platforms where even just a few bytes increase in RAM consumption will cause the build to fail).

jhedberg · 2025-10-16T09:20:01Z

include/zephyr/net_buf.h

 * @return Number of bytes usable behind the net_buf_simple::data pointer.
 */
-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
+size_t net_buf_simple_max_len(const struct net_buf_simple *buf);


This should be in a separate commit, with the commit message explaining that it's just aligning this with the rest of the net_buf APIs.

This is still a breaking API change, though.

jhedberg · 2025-10-16T09:20:21Z

include/zephyr/net_buf.h

-	uint16_t offset;
+	uint32_t offset;
 	/** Length of data */
-	uint16_t len;
+	uint32_t len;


Same here. This should be build-time conditional.

Got it, I will add back the net_buf_size_t too. Do you have concern about using typedef? IMO, both are OK.

#if defined(CONFIG_NET_BUF_LARGE_SIZE) /** @brief net buf size type */ typedef uint32_t net_buf_size_t; #else /** @brief net buf size type */ typedef uint16_t net_buf_size_t; #endif ... net_buf_size_t len; ...

or

... #if defined(CONFIG_NET_BUF_LARGE_SIZE) uint16_t len; #else uint32_t len; #endif ...

Considering that you need to have the correct type in multiple structs and multiple member variables, I think the typedef would be the simplest option.

That said, I'd appreciate some more thoughts from @jfischer-no, since he seemed to be against this. @jfischer-no changing unconditionally to a larger size will not work without also finding a solution for our memory constrained platforms that can't currently take any additional memory hit wrt to net_buf sizes.

jhedberg · 2025-10-16T09:20:34Z

include/zephyr/net_buf.h

-			uint16_t len;
+			uint32_t len;

 			/** Amount of data that this buffer can store. */
-			uint16_t size;
+			uint32_t size;


And here.

Initially I think we should make the larger size opt in. We can then discuss if eventually it'd be opt-out, and then we make the memory-constrained configurations explicitly select the smaller size.

jhedberg · 2025-10-16T09:21:34Z

include/zephyr/net_buf.h

 * @return Number of bytes usable behind the net_buf::data pointer.
 */
-static inline uint16_t net_buf_max_len(const struct net_buf *buf)
+static inline size_t net_buf_max_len(const struct net_buf *buf)


Separate commit.

jhedberg · 2025-10-16T09:21:46Z

lib/net_buf/buf_simple.c

 }

-uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf)
+size_t net_buf_simple_max_len(const struct net_buf_simple *buf)


Separate commit

jhedberg · 2025-10-20T09:24:03Z

lib/net_buf/Kconfig

+config NET_BUF_LARGE_SIZE
+	bool "Network buffer large size"
+	help
+	  Enable support for large network buffers.


IMO this should specify exactly what the larger size is.

I add the uint32_t in the description. If the config name needs to be changed too, please let me know.

I must admit I am not a fan of Kconfig like this for changing an API. What happens when another subsystem in Zephyr relies on a specific maximum size, but the user selects the Kconfig, changing the size? We been through similar constructs in the past, none turned out to work very well.

I agree with Johan's thought as #97440 (comment).
IMO, today's MCUs all seems to have quite a bit of RAM. and this change only increase 4 bytes for every net buf. 🙂

As discussed on Discord, a cleaner (and hopefully less controversial) way forward would be to introduce accessor functions to the net_buf parsing state (i.e. data pointer & length), something like:

size_t net_buf_len(struct net_buf *buf) { return buf->b.len; } void *net_buf_data(struct net_buf) { return buf->b.data; }

The above would also pave the way to remove the union from within net_buf, which only exists as a convenience to avoid having to access fields through the embedded net_buf_simple.

Since this is a rather big effort, it's not realistic to consider it for Zephyr 4.3 anymore. However, size_t return value fix in this PR is still valid, IMO (along with a migration guide entry), so I think it would be good to get that separated and merged for the next release.

OK, thanks. It can hide the len of the struct net_buf from the users, users only use size_t.
But we still need to change the type of 'len' by CONFIG_NET_BUF_LARGE_SIZE option to support large data size, right?

Sure, I will create another pr to change the return value of net_buf_max_len.

I really do not think we need a Kconfig for this. What do others think?

I really do not think we need a Kconfig for this. What do others think?

The current uint16_t types are a memory footprint optimization. I don't feel confident to claim that no one needs or wants that optimization anymore. And if we don't make that claim I can't think of other reasonable options besides being able to enable or disable the optimization through Kconfig.

jhedberg · 2025-10-20T09:25:10Z

tests/lib/net_buf/buf/testcase.yaml

    tags:
      - net_buf
+  libraries.net_buf.large_buf:
+    min_ram: 16


Not so sure this will help. One memory constrained platform which uses net_bufs (for Bluetooth) is nRF51 with 16k RAM, so the above wouldn't exclude it. I'd propose to use 32.

Updated as 32.

change uint16_t as uint32_t to support large size of data with one Kconfig. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

add test through testcase.yaml. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

sonarqubecloud · 2025-11-19T07:35:28Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

pdgendt reviewed Oct 13, 2025

View reviewed changes

include/zephyr/net_buf.h Outdated Show resolved Hide resolved

MarkWangChinese marked this pull request as ready for review October 13, 2025 09:13

zephyrbot added the area: Networking Buffers net_buf/net_buf_simple API & implementation label Oct 13, 2025

zephyrbot requested review from jhedberg, jukkar and rlubos October 13, 2025 09:15

zephyrbot assigned jhedberg Oct 13, 2025

MarkWangChinese requested review from jfischer-no and tmon-nordic October 13, 2025 09:19

pdgendt previously requested changes Oct 13, 2025

View reviewed changes

tmon-nordic reviewed Oct 13, 2025

View reviewed changes

include/zephyr/net_buf.h Outdated Show resolved Hide resolved

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 3a3a6a6 to 938fd0f Compare October 14, 2025 02:12

zephyrbot added the area: Tests Issues related to a particular existing or missing test label Oct 14, 2025

zephyrbot requested a review from nashif October 14, 2025 02:14

jukkar previously approved these changes Oct 14, 2025

View reviewed changes

pdgendt reviewed Oct 14, 2025

View reviewed changes

pdgendt previously approved these changes Oct 14, 2025

View reviewed changes

jfischer-no requested changes Oct 14, 2025

View reviewed changes

jhedberg reviewed Oct 14, 2025

View reviewed changes

MarkWangChinese dismissed stale reviews from pdgendt and jukkar via e78e3dc October 16, 2025 08:49

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 938fd0f to e78e3dc Compare October 16, 2025 08:49

jhedberg requested changes Oct 16, 2025

View reviewed changes

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from e78e3dc to 9b3c27a Compare October 20, 2025 08:17

zephyrbot added the area: Networking label Oct 20, 2025

zephyrbot requested a review from ssharks October 20, 2025 08:18

jhedberg reviewed Oct 20, 2025

View reviewed changes

henrikbrixandersen added the Breaking API Change Breaking changes to stable, public APIs label Oct 20, 2025

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 9b3c27a to 5a67648 Compare October 21, 2025 02:17

MarkWangChinese mentioned this pull request Oct 24, 2025

net_buf: change return type of net_buf_max_len as size_t #98144

Open

MarkWangChinese added 2 commits November 19, 2025 15:24

net_buf: support large size

ce529b3

change uint16_t as uint32_t to support large size of data with one Kconfig. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

net_buf: add test for large buf

56a7aaf

add test through testcase.yaml. Signed-off-by: Mark Wang <yichang.wang@nxp.com>

MarkWangChinese force-pushed the feature/net_buf_large_size_support branch from 5a67648 to 56a7aaf Compare November 19, 2025 07:25

		uint16_t net_buf_simple_max_len(const struct net_buf_simple *buf);
		net_buf_size_t net_buf_simple_max_len(const struct net_buf_simple *buf);

net_buf: support large size #97440

Are you sure you want to change the base?

net_buf: support large size #97440

Uh oh!

Conversation

MarkWangChinese commented Oct 13, 2025

Uh oh!

Uh oh!

pdgendt left a comment

Choose a reason for hiding this comment

Uh oh!

pdgendt commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MarkWangChinese commented Oct 14, 2025

Uh oh!

pdgendt Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarkWangChinese Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

pdgendt commented Oct 13, 2025 •

edited

Loading

pdgendt Oct 14, 2025 •

edited

Loading

MarkWangChinese Oct 16, 2025 •

edited

Loading