Reproducer: soref of deleted socket

[JonathanLennox]

The attached test program -- when run on top of #703 -- will, most of the time, crash with an abort because soref() is called on a freed socket. (It's a very tight race condition, so the test program runs the test repeatedly, with a large number of "CPU hog" threads running simultaneously to force context switching. Make sure to set the number of hogs to be at least twice the number of cores you have on your machine.)

crash_repro.c.txt

This is, I believe, a reproducer for the problem reported in #676.

To repeat my hypothesis there:

The userspace code in sctp_close locks SCTP_INP_WLOCK when it sets SCTP_PCB_FLAGS_SOCKET_GONE in sctp_flags; however, the userspace code to set upcall_socket in sctp_timeout_handler (and other functions) checks that bit in sctp_flags without, as far as I can tell, acquiring that lock. Thus, I suspect, under load, the a thread that's checking for an upcall_socket can get suspended immediately after the check of SCTP_PCB_FLAGS_SOCKET_GONE, and a usrsctp_close call can sneak in.

However, I don't understand the lock hierarchy here. Would it be safe to lock SCTP_INP when getting the upcall_socket, or could that cause a lock inversion?