Make Eio_linux.wakeup signal-safe

[talex5]

This will eventually allow Waiters to be used from signal handlers and GC finalizers (once that's lock-free too).

Previously, wakeup couldn't be used from a signal handler because it took a lock (to ensure that the eventfd wasn't closed while it wrote to it). Now, we avoid ever closing eventfds and simply keep free ones in a pool.

/cc @haesbaert

[haesbaert]

This is nice, although the forever leak of fds is kidna evil, especially when we have a single process running tests with multiple runs. Should top at 64 IIRC, since that's maximum number of domains/eventfd pairs.

I can fix the leak with barriers in the future, it's a pretty simple solution, I just have to write it.

[talex5]

I can fix the leak with barriers in the future, it's a pretty simple solution, I just have to write it.

That sounds interesting. How does it work?

[haesbaert]

I can fix the leak with barriers in the future, it's a pretty simple solution, I just have to write it.

That sounds interesting. How does it work?

Reinstating the current problem so we're on the same page

The current problem is t.eventfd might be closed in the middle of wakeup or worse, eventfd might now be something completely different (as in allocated by another Domain).

How barriers could help

Let there be two domains, A and B. A is trying to wakeup B while B is exiting.
t.eventfd would become something akin to a weak pointer Atomic.t Unix.filedescriptor option

At B_2 we remove the external visibility of eventfd, but we can't really close/destroy it since another domain might be in between A_2 and A_3, meaning it got the old value (Some) and will try writing to it, if the write happens before the remote close, we're safe.

Issuing a barrier call makes sure any remote Domain switched fibers at least once, meaning they can't be in the middle of a wakeup, which means any subsequent Atomic.get will see the updated None value, which means I'm now safe to close/destroy the descriptor.

In EIO, a window or epoch could be everyone switched fibers once, so Eio.Domain.barrier() would wait for the next epoch.

(* Domain A trying to wakeup domain B *)
A_0: let wakeup b =
A_1:   match Atomic.get b.eventfd with
A_2:  | Some fd ->
A_3:     Unix.single_write fd wake_buffer 0 8
A_4:  | None -> () (* Target domain must be exiting *)

(* Domain B in the middle of exit *)
B_0: let exit () =
B_1:   let myfd = self.eventfd in (* copy *)
B_2:   Atomic.set self.eventfd None; (* unlink *)
B_3:   Eio.Domain.barrier (); (* publish *)
B_4:   Unix.close myfd (* destroy *)

(* The most naive barrier implementation would be a zero sized
   Stream.t where you just put "nothing" on it, since buffer is zero,
   the reader has to acknowledge, which means it swi tched Fibers *)
let barrier () =
  List.iter (fun t -> Eio.Stream.put t.barrier_box ()) all_domains

Background

This is mostly what RCU does, OpenBSD calls it SMR (Safe Memory Reclamation), DPDK calls it something else, but it's quite popular in C in order to properly know when you can free(3) something. I've implemented this in our OpenBSD fork at work and rewrote most of pf(4) and pfsync(4) with it.

How the barrier is implemented varies, it just needs to guarantee "wait till next epoch". The classic implementation on non preemptive kernels was pinning your thread to every different cpu and just continuing. Barriers need not to be synchronous as well, it's common to be able to "schedule a callback for the next window" so you could say "call Unix.close () in the next epoch".

https://pdos.csail.mit.edu/6.828/2008/readings/rcu.pdf
https://www.osadl.org/fileadmin/dam/presentations/RTLWS11/paulmck-rcu.pdf

[talex5]

That would be useful, but I don't see how it could be implemented. It's not just all_domains: signals and GC finalizers can also run in other systhreads and non-Eio domains.

[haesbaert]

That would be useful, but I don't see how it could be implemented. It's not just all_domains: signals and GC finalizers can also run in other systhreads and non-Eio domains.

We would have to make concessions like "you're only supposed to have Eio Domains", or X is only safe within an Eio context and unsafe if accessed outside.

It would work out for this eventfd case though (I think).

[talex5]

We want to use wakeup to e.g. let a Domainslib domain send a message to an Eio one.

[haesbaert]

We want to use wakeup to e.g. let a Domainslib domain send a message to an Eio one.

that's a bummer

[haesbaert]

Can we change this to something else, free for me is the opposite of alloc.

[talex5]

What name do you suggest? It's the list of free FDs.

[haesbaert]

free_list, fd_list, fd_pool, free_pool, but don't let this block you, there's no point in me bikeshedding it over this, if free is of your preference please go ahead.

[talex5]

OK, I'm going to keep it as it is. free_list can be misread as a command just like free can; fd_list doesn't say why the FDs are there, which seems the most important thing; and the module already has pool in the name. Also, it's only in scope for 6 lines.

[haesbaert]

So regarding this PR, I don't like the name free but other than that I think you should commit.
The benefit of not locking for every wakeup outweights the fd leak imho.