Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Create forking worker #3

Closed
nesquena opened this Issue · 9 comments

4 participants

@nesquena
Owner

Resque uses forking to control memory management and bloat and I think this could be a good idea to have a fork worker to apply the same principles.

@jpfuentes2

I think that's a critical feature of resque which might prohibit me from switching to beanstalkd/backburner. Have you received reports/issues of users having this type of problem?

@nesquena
Owner

Yes I have a few people that mentioned to me running into this issue, I have not had this problem yet in my own applications. It really depends on the types of jobs you are running. So far I just create multiple workers (separate processes) and have them process the jobs similar to delayed_job: https://github.com/collectiveidea/delayed_job

I am not convinced that a forked worker is necessary for a large set of cases. That said, I am actually interested in having a threaded and forking worker available so people can choose based on their needs.

@nesquena
Owner

@ShadowBelmolve

Check out:

https://github.com/nesquena/backburner/blob/workers/lib/backburner/worker.rb
https://github.com/nesquena/backburner/blob/workers/lib/backburner/workers/simple.rb

What do you think of this? I kind of like using subclassing here to create workers because it requires less code in each individual worker and then usage of workers to be more explicit.

If you like it, can you try porting the forking_thread_worker to this format?

@ShadowBelmolve

@nesquena yes. My only problem is with tests. I'm not good with this yet so I'll make only the basic ones.

@nesquena
Owner
@nesquena
Owner

Good reference for a simple forking worker michaeldwan/stalker@3862676 by @michaeldwan

@danielfarrell

FYI, I've got a start on this here:

https://github.com/danielfarrell/backburner/tree/forking-worker

I need to write a test still before I send a pull request.

@nesquena
Owner

@danielfarrell Awesome, thanks Daniel. Let me know when the tests are ready and I will merge and update all the documentation.

@nesquena
Owner

Closed with 0.3.1 thanks to @danielfarrell

@nesquena nesquena closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.