sockets: Implementing poll(POLLOUT) support

[pfalcon]

Currently, zsock_poll(POLLOUT) implemented as unconditionally returning this flag. This means that if there're no free packet/data buffers, then user application will be effectively in busy-polling loop: POLLOUT will be returned, and app will try to send() to a socket and will get EAGAIN, will call poll() and immediately get POLLOUT, repeated.

There can be different solutions to this, of different level of overhead, "precision" (i.e. how big there's a chance still that POLLOUT will be returned from poll(), yet EAGAIN from send()), and changes required to Zephyr kernel and/or IP stack code. Btw, the fact that overhead and/or changes to code beyond BSD Sockets layer would be required, was the reason why such support wasn't implemented initially and postponed until later.

Two solutions on roughly opposite sides of "precision" spectrum:

1. The first idea is to limit number of outgoing inflight packets for a socket. If socket is currently sending out pkt, it's not POLLOUT'able. Implementation of that would require adding a semaphore to each socket (aka net_context, because socket is nothing but a net_context). send() would take semaphore, "sent" callback would give it. That alone doesn't guarantee high precision though, as a socket might not have an inflight pkt, and yet pkt pool might not have a free pkt either. To solve precision "problem", would need to associate a static outgoing pkt buffer with each socket. That's another field in socket structure, and (much) higher pressure on pkt pool. This still isn't going to resolve too much though, due to 2-level net allocation structure of Zephyr: to send out a pkt, it's required to allocate net_pkt, then allocate one of more net_buf, and attach to net_pkt. To guarantee something, would require to keep static net_pkt and net_buf per each socket.

2. Another idea is to poll pkt pool for free pkt slots. By definition, this method is not precise, because several sockets may compete for a single pkt buffer. And it's even less precise due to 2-level alloc structure, described above.

[pfalcon]

I'd like to start with the approach 2 as proof of concept, promising minimal data space overhead, and potentially benefiting Zephyr beyond just sockets subsystem. So, we'd need to k_poll() on pkt pool, which is backed struct k_mem_slab. And here's caveat: this structure is not k_poll'able. That's why I'm saying that going this way would benefit Zephyr, but minimizing adhoc split between k_poll'able and just thread-blocking structures.

[pfalcon]

@nashif : FYI, this can be seen as a continuation of discussion in #5378 (comment) . This documents what's really (as in: non-trivially) missing in sockets implementation. This might affect async socket usage (which is advanced socket usage, not expected to be the first line of socket usage anyway; it does affect advanced applications, like MicroPython Zephyr port already though).

[pfalcon]

@erwango : FYI, if you're interested why MicroPython's uasyncio module and picoweb web micro-framework based on it are not usable with Zephyr port, that's it. Actually, as the above says, we're talking about no-efficient working, not "non-working", so I actually need to give it a try.

[AndreyDodonov-EH]

This is still the case, right?
At least in 2.7.
I tried using this to tell webserver running on Zephyr to slow down, but both select and poll return immediately.

[rlubos]

Right, this has not been resolved yet.

[carlescufi]

Implemented in #45626