Skip to content
Permalink
Browse files

Add the core implementation of a TLB-shootdown based RCU_TLB mode.

Add a new flavor of userspace RCU that uses TLB shootdowns to force
readers to issue memory barriers. This works along the lines of
the membarrier and signal approaches, but uses a different primitive
to force threads to issue barriers: mprotect(2).

When a thread revokes write permissions on a page, the kernel needs to
do a TLB shootdown to make sure that none of the other CPUs running
code in that address space have a writable mapping for that page
cached. In Linux, this is done by forcing code to invalidate the
mappings to run on every CPU in the address space, and waiting for
completion. The code for the "run this function on another CPU"
mechanism forces the target CPU to issue an smp_mb().

(In practice TLB shootdowns are done when permissions are added, not
just when they are removed, but they needn't be; faults caused by
using a cached entry with less permissions can be fixed up by the page
fault handler. They're also needed when unmapping memory, but
mprotect() seems cheaper than having to mmap() and munmap(). Also TLB
shootdowns aren't needed if the page is non-present because it's never
been backed or has been swapped out, so mlock(2) is used to keep it in
place).

The performance characteristics of this approach are to be roughly
similar to the membarrier approach. Relative to membarrier, the
mprotect()/TLB shootdown approach has one minor disadvantage, one
major disadvantage, and one major advantage:
 * membarrier() should be slightly more efficient on the write
   side. The mprotect() approach needs to make two system calls instead
   of one and the system calls need to actually do some pointless
   modifications of the page table. It shouldn't be that expensive,
   though.
 * The mprotect() version is a total and complete hack. It is
   blatantly abusing an interface in a way that it was not designed to
   be used. No spec or documentation makes any guarentees about
   mprotect() forcing memory barriers. That said, Linux's TLB
   shootdown will cause every running thread in the process to call
   smp_mb(), and I think it is probably difficult to build an
   implementation that does not.
 * mprotect() is actually implemented in production
   kernels. membarrier() is clearly a better solution, but it never
   shipped.

Relative to the signal and mb implementations, the TLB shootdown
version has all of the advantages of membarrier() and continues to
have the "is a total hack" disadvantage.

For clarity, I split the patch up into the actual implementation and
the various Makefile/test/config changes needed to build and test it;
this is the former.
  • Loading branch information
msullivan committed Feb 23, 2015
1 parent edf8de6 commit 04656b468d418efbc5d934ab07954eb8395a7ab0
Showing with 55 additions and 0 deletions.
  1. +48 −0 urcu.c
  2. +7 −0 urcu/static/urcu.h
48 urcu.c
@@ -35,6 +35,7 @@
#include <string.h>
#include <errno.h>
#include <poll.h>
#include <sys/mman.h>

#include "urcu/wfcqueue.h"
#include "urcu/map/urcu.h"
@@ -87,6 +88,16 @@ int rcu_has_sys_membarrier;
void __attribute__((constructor)) rcu_init(void);
#endif

#ifdef RCU_TLB
static int init_done;
/* We use a silly trick:
* removing permission flags with mprotect() will force a TLB shootdown,
* which should force barriers. */
static void *dummy_barrier_page;

void __attribute__((constructor)) rcu_init(void);
#endif

#ifdef RCU_MB
void rcu_init(void)
{
@@ -158,6 +169,24 @@ static void smp_mb_master(int group)
}
#endif

#ifdef RCU_TLB
/* Force a barrier using a TLB shootdown */
static void tlb_barrier() {
/* Make the dummy page writable, then take it away.
* We do this because there is really no need to TLB shootdown
* when /adding/ permissions. */
if (mprotect(dummy_barrier_page, 1, PROT_READ|PROT_WRITE) < 0 ||
mprotect(dummy_barrier_page, 1, PROT_READ) < 0) {
urcu_die(errno);
}
}

static void smp_mb_master(int group)
{
tlb_barrier();
}
#endif

#ifdef RCU_MB
static void smp_mb_master(int group)
{
@@ -460,6 +489,25 @@ void rcu_unregister_thread(void)
mutex_unlock(&rcu_gp_lock);
}

#ifdef RCU_TLB
void rcu_init(void)
{
if (init_done)
return;
init_done = 1;

/* Allocate a dummy page that we can use. */
void *page = mmap(NULL, 1, PROT_READ, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
if (page == MAP_FAILED) urcu_die(errno);

/* Lock the memory so it can't get paged out. If it gets paged
* out, changing its protection won't accomplish anything. */
if (mlock(page, 1) < 0) urcu_die(errno);

dummy_barrier_page = page;
}
#endif

#ifdef RCU_MEMBARRIER
void rcu_init(void)
{
@@ -112,6 +112,13 @@ static inline void smp_mb_slave(int group)
}
#endif

#ifdef RCU_TLB
static inline void smp_mb_slave(int group)
{
cmm_barrier();
}
#endif

#ifdef RCU_MB
static inline void smp_mb_slave(int group)
{

0 comments on commit 04656b4

Please sign in to comment.
You can’t perform that action at this time.