Workers will now accept a PERCHILD env variable. This will process multiple jobs in each child to reduce the amount of forks and drastically improve performance. Not setting the variable, or having a value of 1, is equivalent to the previous behaviors.
Did not write unit tests, but did document in comments and did adhere to code style used.
Deployed this in production in our (chartboost.com) high-traffic (1k jobs per second) environment and is running smoothly so far. Will report back tomorrow if anything is new.
multi-job per fork
remove extra log
Wouldn't this compromise the safety provided by forking all of the job processing into another thread? I'd be fairly nervous about losing the graceful handling of failed jobs. Am I misunderstanding this?
That's what I'm seeing here, too. Hey, @kballenegger - how far are we from the mark on this one?
You're not wrong, it trades safety for scalability. Forks are extremely expensive.
I don't remember the specifics of how I'd implemented this though; it has been over a year. There may be better ways of doing this though internal acks via IPC or ZMQ sockets.
This PR, unfortunately, won't apply cleanly anymore :-(
@Dynom - I am not that surprised… it is over 2 years old!