drivers: net: native_sim_sockets: new driver #65116

mniestroj · 2023-11-13T10:48:05Z

Add driver for 'native_sim' target that implements offloaded socket
networking by the use of host networking stack and wrapped BSD sockets API.

This driver has following advantages over existing networking drivers for
emulated platforms that are already in tree:

no TUN/TAP use means that no additional setup is required on the host
side:
- possible to use it within unpriviledged Docker containers, either for
  development or in CI
possibility to use and test offloaded sockets
(CONFIG_NET_SOCKETS_OFFLOAD=y) with emulated target, which allows
to increase tests coverage of this feature, without requirement of using
hardware

Native Simulator host libc has different error codes than embedded libc
used by Zephyr. Convert between those.

Depends on (and includes):

Doc preview:
https://builds.zephyrproject.io/zephyr/pr/65116/docs/boards/posix/native_sim/doc/index.html#nsim-per-offloaded-sockets
https://builds.zephyrproject.io/zephyr/pr/65116/docs/connectivity/networking/native_sim_setup.html#using-offloaded-sockets

jukkar · 2023-11-13T13:17:06Z

drivers/net/native_sim.h

+struct nsim_sockaddr {
+	uint16_t sa_family;      /* Address family */
+	char     sa_data[];      /* Socket address */
+};
+
+struct nsim_sockaddr_in {
+	uint16_t sin_family;     /* AF_INET */
+	uint16_t sin_port;       /* Port number */
+	uint32_t sin_addr;       /* IPv4 address */
+};
+
+struct nsim_sockaddr_in6 {
+	uint16_t sin6_family;    /* AF_INET6 */
+	uint16_t sin6_port;      /* Port number */
+	uint32_t sin6_flowinfo;  /* IPv6 flow info */
+	uint8_t  sin6_addr[16];
+	uint32_t sin6_scope_id;  /* Set of interfaces for a scope */
+};


Why do we need to have separate nsim sockaddr structs here? I mean sockaddr is a sockaddr always, no?

The problem that the introduction of struct nsim_* is trying to solve is to have a common layer between the two libc worlds:

Native Simulator, which is compiled with host libc

Zephyr, which is compiled with embedded libc

In order to pass sockaddr values between those two worlds, we need a structure that is defined the same way. This is where structs like struct nsim_sockaddr_in6 are used, since they are locally defined and libc-independent. This is just to provide bridge between two conflicting libc implementations.

jukkar · 2023-11-13T13:20:15Z

drivers/net/native_sim_errno.h

+
+#include <errno.h>
+
+#define NSIM_EPERM 1         /**< Not owner */


Can't we say here

#define NSIM_EPERM EPERM ....

Similar as described in #65116 (comment), NSIM_* values are defined to bridge between Native Simulator and Zephyr libc implementations. If we would define #define NSIM_EPERM EPERM, NSIM_EPERM value would be potentially different in Zephyr, when there is embedded libc used, and different in Native Simulator when there is host libc used. So by defining #define NSIM_EPERM 1 we make sure that the value is same in those two contexts.

Hmm, don't we need to just convert the errno values between zephyr <-> host, so in either zephyr or in host, meaning we could use the errno value defined in one of them?

Hmm, don't we need to just convert the errno values between zephyr <-> host, so in either zephyr or in host, meaning we could use the errno value defined in one of them?

When building for host, we don't have access to Zephyr's errno.h. When building for Zephyr, we don't have access to host's errno.h. Even if we did, there would be symbol conflicts. So the NSOS_MID_ (previously NSIM_) in nsos_errno.h are supposed to be available to both host and Zephyr, making sure that their values are exactly the same, regardless of used libc implementation.

aescolar

A few comments so far

General requests:

There is no documentation yet in this PR. I guess we should add something in at least these 2:
- https://docs.zephyrproject.org/latest/connectivity/networking/networking_with_host.html
- https://docs.zephyrproject.org/latest/boards/posix/native_posix/doc/index.html#peripherals
  (note Swap documentation from native_posix to native_sim #65057 is moving docs around)

Note: I focus on the "native_simulator" side of things. For the networking side other people are better.

drivers/net/Kconfig

drivers/net/native_sim.h

drivers/net/native_sim_sockets.c

drivers/net/native_sim_adapt.c

drivers/net/native_sim_errno.c

drivers/net/native_sim_adapt.c

aescolar

Some extra comments

drivers/net/native_sim_sockets.c

drivers/net/native_sim_adapt.c

github-actions · 2024-01-13T00:29:27Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

beriberikix · 2024-01-15T12:47:45Z

Please remove the stale label. Thanks!

aescolar · 2024-03-02T07:54:16Z

The compliance check is a false positive which will need waiving.

@jukkar @rlubos To me it looks good, your review and blessing as networking maintainers would be the missing step.

nordicjm

Needs converting to hwmv2

aescolar · 2024-03-04T08:25:30Z

Please rebase this PR.
Now that the hwmv2 branch has been merged this PR needs rebasing on top of the latest main.
Note that the boards & socs folder has moved, and many boards names as used in tests yaml
and overlays have changed.

Add driver for 'native_sim' target that implements offloaded socket networking by the use of host networking stack and wrapped BSD sockets API. This driver has following advantages over existing networking drivers for emulated platforms that are already in tree: * no TUN/TAP use means that no additional setup is required on the host side: * possible to use it within unpriviledged Docker containers, either for development or in CI * possibility to use and test offloaded sockets (CONFIG_NET_SOCKETS_OFFLOAD=y) with emulated target, which allows to increase tests coverage of this feature, without requirement of using hardware Native Simulator host libc has different error codes than embedded libc used by Zephyr. Convert between those. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Extend driver to support DNS by offloading getaddrinfo() and freeaddrinfo() APIs. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Add support for IPv6 socket offloading, next to existing IPv4 support. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

This allows to use Zephyr TLS subsystem with Native Sim offloaded sockets. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Use NSI_HW_EVENT() in order to periodically check for events in host sockets. Whenever there is a socket event ready to be processed by Zephyr, raise native_sim (newly introduced) CPU interrupt, so that Zephyr driver can signal readiness with k_poll(). Maintain a list of Zephyr poll() executions in Zephyr context. Iterate through them whenever there is some event to be processed. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

So far only non-blocking accept() and recvfrom() were suported. This patch implements blocking behavior, with the use of poll(fd, POLLIN) as helper mechanism. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

'http_get' is one of the few samples that just require connection to internet (google.com over HTTP or HTTPS) and check if HTTP GET was successful. Use it for minimal coverage of Native Simulator offloaded sockets: * DNS resolving * basic TCP transfer * compatibility with Zephyr TLS subsystem (which required some fcntl() operations) Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

'native_sim' builds without any issues as part of twister and it also works when manually setting up Ethernet virtual device (using tools/net-tools/net-setup.sh). Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Add configuration for Native Simulator offloaded sockets. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Document how offloaded sockets can be used with 'native_sim' for networking capability. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

Document offloaded sockets driver, which is an alternative to Ethernet driver for networking applications. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev> Co-authored-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>

aescolar · 2024-03-22T12:40:19Z

(The checkpatch warning is a false positive which should be waived)

@nordicjm the PR has been rebased, please re-review or remove the rejection
@rlubos @jukkar your review as maintainers of the networking subsystem would be very welcome

nordicjm

Build system parts OK

mniestroj requested review from rlubos, tbursztyka and jukkar as code owners November 13, 2023 10:48

zephyrbot added area: native port Host native arch port (native_posix) area: Samples Samples area: Sockets Networking sockets area: Networking labels Nov 13, 2023

zephyrbot requested review from aescolar, kartben, nashif and ssharks November 13, 2023 10:48

zephyrbot assigned rlubos Nov 13, 2023

mniestroj force-pushed the native-sim-offloaded-sockets branch from a85afe9 to 724054f Compare November 13, 2023 10:54

jukkar reviewed Nov 13, 2023

View reviewed changes

aescolar requested changes Nov 13, 2023

View reviewed changes

aescolar reviewed Nov 13, 2023

View reviewed changes

drivers/net/native_sim_sockets.c Outdated Show resolved Hide resolved

drivers/net/native_sim_adapt.c Outdated Show resolved Hide resolved

aescolar added the Maintainer At least 2 days before merge, maintainer review required label Nov 13, 2023

github-actions bot added the Stale label Jan 13, 2024

aescolar removed the Stale label Jan 15, 2024

mniestroj force-pushed the native-sim-offloaded-sockets branch from 724054f to c64ddd2 Compare February 23, 2024 14:21

zephyrbot added the area: Base OS Base OS Library (lib/os) label Feb 23, 2024

zephyrbot requested review from andyross, dcpleung and peter-mitsis February 23, 2024 14:22

mniestroj force-pushed the native-sim-offloaded-sockets branch 4 times, most recently from 7f434fb to bd95400 Compare February 23, 2024 15:55

mniestroj force-pushed the native-sim-offloaded-sockets branch from 822c363 to 41f55a5 Compare March 1, 2024 15:38

aescolar previously approved these changes Mar 2, 2024

View reviewed changes

nordicjm requested changes Mar 4, 2024

View reviewed changes

nordicjm removed the hwmv2-likely-conflict DNM until collab-hwmv2 has been merged label Mar 4, 2024

mniestroj force-pushed the native-sim-offloaded-sockets branch from 41f55a5 to 7b99a03 Compare March 19, 2024 16:44

zephyrbot requested a review from pdgendt March 19, 2024 16:45

mniestroj added 6 commits March 19, 2024 17:51

drivers: net: nsos: add DNS offload support

cb3bf9c

Extend driver to support DNS by offloading getaddrinfo() and freeaddrinfo() APIs. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

drivers: net: nsos: add IPv6 support

dc155ac

Add support for IPv6 socket offloading, next to existing IPv4 support. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

drivers: net: nsos: add F_GETFL and F_SETFL support

14fecd4

This allows to use Zephyr TLS subsystem with Native Sim offloaded sockets. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

drivers: net: nsos: support blocking accept() and recvfrom()

753ee3b

So far only non-blocking accept() and recvfrom() were suported. This patch implements blocking behavior, with the use of poll(fd, POLLIN) as helper mechanism. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

mniestroj dismissed aescolar’s stale review via 6219b53 March 19, 2024 16:52

mniestroj force-pushed the native-sim-offloaded-sockets branch from 7b99a03 to 6219b53 Compare March 19, 2024 16:52

mniestroj and others added 5 commits March 19, 2024 17:54

samples: net: echo_server: add native_sim to allowed platforms

67c4d53

'native_sim' builds without any issues as part of twister and it also works when manually setting up Ethernet virtual device (using tools/net-tools/net-setup.sh). Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

samples: net: echo_server: add config for native_sim offloaded sockets

aec9980

Add configuration for Native Simulator offloaded sockets. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

doc: networking: document native_sim with offloaded sockets

040b3dd

Document how offloaded sockets can be used with 'native_sim' for networking capability. Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>

mniestroj force-pushed the native-sim-offloaded-sockets branch from 6219b53 to f76e513 Compare March 19, 2024 16:54

mniestroj requested review from nordicjm and aescolar March 20, 2024 16:50

aescolar approved these changes Mar 22, 2024

View reviewed changes

nordicjm approved these changes Mar 22, 2024

View reviewed changes

rlubos approved these changes Mar 22, 2024

View reviewed changes

carlescufi merged commit 0f5b5b6 into zephyrproject-rtos:main Mar 22, 2024
25 of 26 checks passed

mniestroj deleted the native-sim-offloaded-sockets branch March 22, 2024 17:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

drivers: net: native_sim_sockets: new driver #65116

drivers: net: native_sim_sockets: new driver #65116

mniestroj commented Nov 13, 2023 •

edited by aescolar

jukkar Nov 13, 2023

mniestroj Nov 13, 2023

jukkar Nov 13, 2023

mniestroj Nov 13, 2023

jukkar Nov 13, 2023

mniestroj Feb 23, 2024

aescolar left a comment •

edited

aescolar left a comment

github-actions bot commented Jan 13, 2024

beriberikix commented Jan 15, 2024

aescolar commented Mar 2, 2024

nordicjm left a comment

aescolar commented Mar 4, 2024

aescolar commented Mar 22, 2024

nordicjm left a comment

drivers: net: native_sim_sockets: new driver #65116

drivers: net: native_sim_sockets: new driver #65116

Conversation

mniestroj commented Nov 13, 2023 • edited by aescolar

jukkar Nov 13, 2023

Choose a reason for hiding this comment

mniestroj Nov 13, 2023

Choose a reason for hiding this comment

jukkar Nov 13, 2023

Choose a reason for hiding this comment

mniestroj Nov 13, 2023

Choose a reason for hiding this comment

jukkar Nov 13, 2023

Choose a reason for hiding this comment

mniestroj Feb 23, 2024

Choose a reason for hiding this comment

aescolar left a comment • edited

Choose a reason for hiding this comment

aescolar left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 13, 2024

beriberikix commented Jan 15, 2024

aescolar commented Mar 2, 2024

nordicjm left a comment

Choose a reason for hiding this comment

aescolar commented Mar 4, 2024

aescolar commented Mar 22, 2024

nordicjm left a comment

Choose a reason for hiding this comment

mniestroj commented Nov 13, 2023 •

edited by aescolar

aescolar left a comment •

edited