erl_check_io: do not discard ERTS_POLL_EV_IN from active_events

[max-au]

This commit fixes a race condition, which can be reproduced
with running a large amount of bidirectional TCP streams.
When the system is fully loaded, it is possible that socket migrated
to scheduler pollset still fires EPOLLIN event. In this case
active_events field of ErtsDrvEventState structure will desynchronise
from the actual state (with EPOLLIN requested via scheduler pollset).
This, in turn, leads to all ERTS_POLL_EV_IN events disregarded. All
reads from the socket will be stopped.
This can only happen for {active, N} mode of a TCP socket, where
fd migrates back and forth to/from scheduler pollset.

[garazdawi]

A bunch (> 15) of testcases have started to fail sporadically after this change was merged. I'm trying to figure out why. The testcases that fail the most are in kernel:

pg_SUITE:thundering_herd
logger_std_h_SUITE:op_switch_to_flush_file
logger_disk_log_h_SUITE:op_switch_to_flush

I'm trying to understand what is going on, any help you can give would be most appreciated.

[garazdawi]

I think I have narrowed it down to a testcase problem. The recv processes exit with reason normal, so the teardown of the Sender does not work which leaves a bunch of processes that consume a lot of CPU which effects the other testcases.

I'll merge a fix in the relevant places.

[max-au]

Thank you! Indeed, this leaves many senders when testcase does not fail.