Rewrite ipdevpoll's multiprocess mode to use generic worker model

[jmbredal]

Inspired by Apache's worker model. The multiprocess mode should spawn a number of slave processes that are assigned collector jobs by a master scheduler. The model should even allow for remote worker processes, for distributed collection.

How to launch remote worker processes is out of scope for this task, but should be considered when designing the communication between the scheduler process and the workers.

---

Imported from Launchpad using lp2gh.

• date created: 2013-11-05T09:26:17Z
• owner: mbrekkevold
• the launchpad url was https://bugs.launchpad.net/bugs/1248090

[lunkwill42]

I would like one more feature out of this: I want workers to kill themselves (only to be restarted) after having run a configurable number of jobs.

Python isn't particulary keen on releasing memory to the OS, once allocated, causing some of the workers (especially one's processing topo jobs) to increase memory usage heavily over time.

This is analogous to Apache's MaxRequestsPerChild, and will offer simple protection against inadvertent "memory leaks".

[lunkwill42]

Closing this, as both implementing PRs have been merged :-)