Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Kue Callback System #180

Closed
winduptoy opened this Issue · 6 comments

4 participants

@winduptoy

Not really an issue, but I wanted to share my code showing how I got callback functionality using Kue in a separate worker process from my main application logic. If anyone can help me improve my code or suggest a better way of doing this, I'd love to hear it.

Maybe this could be a feature in Kue?

https://gist.github.com/winduptoy/4991718

@drudge
Owner

I implemented a similar client/worker model for scaling purposes, only using redis. I think this is something that belongs at the application layer.

@drudge drudge closed this
@winduptoy

Duh, I totally reinvented the wheel there. Seems like using redis alone is a much simpler approach, thanks for sharing!

@bjackson

How did you go about doing that @winduptoy? I'm a little confused, myself.

@behrad
Owner

@bjackson What dya really wanna do?
You can use redis pub/sub Or event Kue's progress event to communicate between your modules.

@bjackson

I'm confused about Kue, in the context of threading, specifically how jobs are processed. I have a somewhat long CPU-bound task that I'd like to put in the background. I think each task would take less than a second under a non-trivial load, but that's still too much time to run in the main thread. Does Kue's job processing run in a separate thread, or should I have jobs processed in a separate process and communicate the result back to the main thread?

@behrad
Owner

I'm confused about Kue, in the context of threading, specifically how jobs are processed

Kue is a node.js process, and a node.js process is a single threaded process. So each of your processes will share the same thread/cpu core.
if you want parallelism please check related section in Kue documentation to see who node.js clustering helps you distribute you work across multiple os processes on cpu cores.

I have a somewhat long CPU-bound task that I'd like to put in the background

Theoretically node.js is not suitable for cpu-bound tasks, so you may finish to refactoring your task to a C/Python/... code which is called by you node.js workers.

Does Kue's job processing run in a separate thread, or should I have jobs processed in a separate process and communicate the result back to the main thread?

As I mentioned you can create an extra kue worker process to consume your job and pass the result back to your producer process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.