Event Re-Registration

[rrichardson]

There are a couple fundamental flaws in using Edge Triggered without One Shot.

Data Loss due to non notification
The first and most major is that if the read() in response to a read-event doesn't consume all of the data from the socket's buffer, it will never be notified about data again, which means the read-event won't be called again and the data will remain forever alone.

Starvation
The model of needing to consume all data out of a socket can lead to starvation of other sockets if there is an extreme amount of data being sent into that socket. A better model would be to consume a small amount of data per each socket, in turn, and go back to epoll_wait.

There are two solutions to this:

Level Triggered
This is easier for people to get right, as you'll always be notified if data is present in the case of read, or the socket is ready in case of write. The downside is it can be spammy, you don't need to be constantly notified of Writable events if you don't care about writing. A web server is a great example, it only needs to write after it receives a request. The solution is, obviously, to allow an application to un-register its interest in writing when it doesn't care about writing.

Edge Triggered with One Shot
This is the recommended approach. This would also require frequent re-registration, but epol_ctl is a very cheap syscall (cheaper than read() returning EAGAIN)
This would require every read and writer to re-register their interest in reading and writing after every epoll_wait call that returns events for that FD.
This solves both of the above problems. Epoll_wait will return fds who have data waiting regardless of whether it has been resting a while or has recently arrived.

IMO, the best approach is to let the user specify if they want edge triggered vs level triggered. But regardless of that model, there should definitely be a reregister function.

[carllerche]

I accept. How does level triggered & reregistration work on other platforms (not counting windows)?

[rrichardson]

IIRC, kqueue is edge triggered only, but it does support one-shot.

[carllerche]

Quick google search:

Yeah that's right. I discovered that too while testing kqueue. Without
EV_CLEAR, kevent acts as level-triggered (it always returns as long as a
condition holds, socket readable/writeable etc). Setting EV_CLEAR causes
kevent to act as edge-triggered (it returns once when state changes and will
block on second call until state changes again).

So it seems possible to implement on those systems. What API do you have in mind? Add an option to EventLoopConfig or something else?

Also, how would a re-registration API differ from the existing register fn?

[rrichardson]

Two ways:

1. The current register doesn't take events, it just registers for everything.
2. EPOLL_CTL_ADD vs EPOLL_CTL_MOD

I'd allow register to selectively specify interest (maybe supply an IoEventKInd of IOEVENTALL as a convenience for people)
and also ad reregister which requires people to specify the events in which they're interested.

[carllerche]

Re 1), register should take the interests, see #33. 2 makes sense, I had unloaded epoll from my brain :P

[carllerche]

Resolved by 0122914