Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for transient processing of jobs #191

Open
rwdaigle opened this issue Jan 8, 2014 · 7 comments · May be fixed by #256
Open

Allow for transient processing of jobs #191

rwdaigle opened this issue Jan 8, 2014 · 7 comments · May be fixed by #256
Assignees
@ukd1
Labels
feature
Milestone
4.1.0

Comments

@rwdaigle
Copy link

rwdaigle commented Jan 8, 2014

It would be convenient to have a single entry point into QC that processes all jobs, then exits (i.e. a non-daemon based worker). Something resembling:

$ QC_EXIT_ON_COMPLETION=true rake qc:work

or perhaps a new rake task all together?

The use case here is to reduce the need to have a long-lived daemon to keep alive for systems with low numbers of jobs, or jobs that can wait for a cron-based process to spawn and work on them. On Heroku this would allow users to have an hourly task with Scheduler to process the queue instead of a whole worker dyno.
@ryandotsmith
Copy link
Contributor

As I alluded in a tweet, you can almost do this today.

worker = QC::Worker.new
QC.count.times {woerker.work}

The problem is that QC::Worker#work spins on the worker's @running instance variable in the case that there are no jobs. So if a job is worked between the time your call QC.count and worker.work then you will end up blocking the thread.

I think it would be nice if QC::Worker#work did not block and instead returned nil in the case it was unable to find a job. That would make the aforementioned snippet safe.

Thoughts?

@rwdaigle
Copy link
Author

rwdaigle commented Jan 9, 2014

That solves the problem, though it requires a small bit of knowledge about the internals of QC. I guess it's nothing to wrap your code snippet in a rake task or some other helper method, though.

@ryandotsmith
Copy link
Contributor

Great point. If we had the right primitives, we could easily wrap them up into a QC provided Rake task.

@toshe
Copy link

toshe commented Jan 10, 2015

+1 from me.

Perhaps we need some kind of a lock mechanism to prevent other workers from doing work while the queue is emptied? (the case that @ryandotsmith described).
This would also help with multi-threaded environments.

@senny
Copy link
Contributor

senny commented Mar 4, 2015

👍 would be nice to have this.

@rwdaigle are you interested in working on a patch?

@senny senny added the feature label Mar 4, 2015
@rwdaigle
Copy link
Author

rwdaigle commented Mar 4, 2015

@senny I'll put it on my list, but suspect it will not be in the near term that I'll be able to work on it.

senny added a commit that referenced this issue Mar 6, 2015
@senny
add Worker#work_off and rake qc:work_off.
a73aa07 
This provides a way to work off all the jobs in the queue
and exit afterwards.

Closes #191.
@senny senny linked a pull request Mar 6, 2015 that will close this issue
Open
@senny
Copy link
Contributor

senny commented Mar 6, 2015

@rwdaigle I took a stab at this #256 let me know if this addresses your needs.

@ukd1 ukd1 added this to the 4.0.0 milestone Jul 23, 2019
@ukd1 ukd1 self-assigned this Jul 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ukd1 ukd1

Labels
feature
Projects
None yet
Milestone
4.1.0
Development

Successfully merging a pull request may close this issue.

add Worker#work_off and rake qc:work_off. QueueClassic/queue_classic
5 participants
@rwdaigle @senny @ryandotsmith @ukd1 @toshe