Rhizome parallelization #68
The purpose is to execute rhizome actions on a thread different from that handling routing and MDP stuff, so that a long rhizome action does not jam the communications. This first step adds a "struct fdqueue" type with all the fields specific to an fdqueue. Two instances are created: main_fdqueue and rhizome_fdqueue. All public fdqueue functions now take an fdqueue argument. They are thread-safe. fd_poll() uses synchronization privimives for waiting until the next event, so that the wait can be interrupted for executing a new action earlier.
When poll() is waiting for events on watched file descriptors, it should not block any thread wanting to (un)schedule an alarm. But the mutex cannot be released while executing poll(), because the watched file descriptors must not be modified (or closed!) by another thread. Therefore, poll() now waits for one more file descriptor, a pipe, to become ready. Before acquiring a mutex, fdqueue functions write 1 byte to the pipe. Once the mutex is acquired, they read 1 byte. Thus, poll() is guaranteed to be non-blocking when a thread waits for the mutex. If the pipe is ready for I/O, then fd_poll() release the mutex for 1ms.
Implement rhizome thread function (which consumes its own fdqueue). Add a util function schedules a new (allocated) alarm from only 3 parameters: - the function to execute; - its parameter (void *); - the fdqueue to use. Of course, the parameter of the function must not be stored on the call stack (else it will not exist anymore when the other thread will use it).
Schedule rhizome_retrieve_and_advertise_manifest_alarm in overlay_mdp_service_manifest_response.
I only added the glue to make the current code multithreaded, without changing the logic.
Paul also said:
I read somewhere that you wanted to avoid threads, and use only monothreaded processes. The reasons are unclear to me. Is avoiding residual inter-thread locks the main reason? Could you explain?
On Tue, Aug 13, 2013 at 7:27 PM, ®om firstname.lastname@example.org wrote:
I've split overlay_mdp_dispatch such that internal services that only sent
Instead of allocating a new alarm per frame, we can probably build some
On that point, should we rename the "rhizome" thread to the "background"
I agree, the missing part of the work is to rewrite some algorithms, when
Ah? Where are these variables?
Which services do send packets to "local" mdp clients? Which are these "local" mdp clients?
But, that way, it could only apply for passing one frame from one thread to another.
My idea was to pass "runnables" (a generic function+argument to post whatever action you want). In practice, the alarms I scheduled do not always post frames (see parallel.h
I don't know if the overrhead of these
The way I've implemented it uses the same mechanism both for main thread and rhizome thread. As a consequence, if rhizome blocks waiting for main thread to be idle, then main thread will also block waiting for rhizome thread to be idle. The situation where the main thread needs to post a runnable on the rhizome thread occurs (1 2 3), but maybe it can be avoided…
I've considered this background thread to be rhizome-specific: another service would have its own thread too… Although, even a single service could have several threads.
I think it is a good idea.
Ideally, I think Rhizome could simply work as any other service on top of MDP: it would open an MDP socket on a predefined port and exchange with other peers, without any lower-level knowledge and, above all, without being referenced by any lower-level code.
This would remove the need of "internal services" hack: each service would use its own port dynamically (like with TCP or UDP).
In that case, Rhizome would create its own thread to handle its stuff separately, without impacting overlay* code.
What do you think?