Concurrent sendmsg()/recvmsg() calls are not allowed

[GoogleCodeExporter]

I discovered that seccomp-sandbox does not currently allow concurrent
sendmsg() and recvmsg() calls.  If one thread is blocked in a
recvmsg() call, a second thread that calls sendmsg() will block.

This is because seccomp-sandbox uses a global mutex (syscall_mutex_)
for all syscalls that require data to be written to a secure memory
area by the trusted process.  The trusted process will handle only one
syscall at a time, and it waits for syscall_mutex_ to be unlocked
before handling another syscall.

I discovered this while trying to hook up Native Client to use
seccomp-sandbox.  Some of the tests deadlocked: there was a background
thread blocked on recvmsg(), while foreground threads would then block
on calls like mmap().

To fix this, I propose two changes:

1) Use one mutex per thread, rather than a global mutex.

2) Change the trusted process so that it does not wait for the
   thread's mutex to be unlocked before processing another syscall
   (which might come from another thread).

The wait in (2) happens in sendSystemCallInternal() in securemem.cc.
This wait should only be necessary if an allowed syscall has a side
effect that must complete for a subsequent allowed syscall to be safe.
I don't think this is the case for any currently allowed syscalls: the
trusted process does not attempt to model state changes of the
sandboxed process; ordering of syscalls, once checked, is not
significant.  (A possible exception is in the IPC syscalls in ipc.cc.)

The only wait needed should be in lockSystemCall(), to prevent a
secure memory area from being reused while it is still in use.

I have got an implementation of these changes which I'll send out
soon.

Original issue reported on code.google.com by mseaborn@chromium.org on 7 Sep 2010 at 3:57

[GoogleCodeExporter]

I discovered today that recvmsg() holds syscall_mutex_ on i386 but not
on x86-64.  This is because recvmsg() is implemented using different
code on the two architectures.  On i386 it reuses "accept_complex"
which calls sendSystemCall() with locked=true.

So there are two cases to consider:

 a) recvmsg() first, which is unblocked by sendmsg() in another thread.

 b) sendmsg() first, which blocks because the socket's buffer is full,
    and is unblocked by recvmsg() in another thread.

(a) currently deadlocks only on i386.  This is simple to fix without
refactoring the sandbox to use multiple mutexes.

(b) will deadlock on both i386 and x86-64.  Fixing this requires
refactoring to use multiple mutexes.

The ability to fill up a socket's buffer using sendmsg() is exposed by
Native Client's IMC syscalls, and although there are currently some
differences in IMC's behaviour between Unix and Windows, we should
probably fix case (b).

Also it might be possible that Chromium relies on case (b) working.

Original comment by mseaborn@chromium.org on 1 Oct 2010 at 11:03