Skip to content

Child batch #35

Open
wants to merge 3 commits into from

4 participants

@kballenegger

Workers will now accept a PERCHILD env variable. This will process multiple jobs in each child to reduce the amount of forks and drastically improve performance. Not setting the variable, or having a value of 1, is equivalent to the previous behaviors.

Did not write unit tests, but did document in comments and did adhere to code style used.

Deployed this in production in our (chartboost.com) high-traffic (1k jobs per second) environment and is running smoothly so far. Will report back tomorrow if anything is new.

@jbrower
jbrower commented Jan 14, 2013

Wouldn't this compromise the safety provided by forking all of the job processing into another thread? I'd be fairly nervous about losing the graceful handling of failed jobs. Am I misunderstanding this?

@danhunsaker

That's what I'm seeing here, too. Hey, @kballenegger - how far are we from the mark on this one?

@kballenegger

You're not wrong, it trades safety for scalability. Forks are extremely expensive.

I don't remember the specifics of how I'd implemented this though; it has been over a year. There may be better ways of doing this though internal acks via IPC or ZMQ sockets.

@Dynom
Dynom commented Aug 31, 2013

This PR, unfortunately, won't apply cleanly anymore :-(

@kballenegger

@Dynom - I am not that surprised… it is over 2 years old!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.