New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can Bull be Used in a Remote Polling Scenario? #790

Closed
travissutton opened this Issue Nov 17, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@travissutton
Copy link

travissutton commented Nov 17, 2017

I'm currently leveraging bull in our project for it's seemingly intended purpose but also have the need to implement a queue system that allows for remote workers to poll an API for tasks to be completed. From what I'm reading on the documentation it doesn't look like this is going to work as the remote workers have no connection to the API's redis connection. Rather than doing a queue.process I need to do something like queue.poll to get a job, then have that job locked for some duration. The worker would then post back to the API with updates, the API would then need to be able to readily access the Job on-demand (assuming queue.getJob). Finally, I would need to either complete or fail the job from the API side based on a final update from the worker (all I can find here is queue.getJob->job.progress(100) - hoping this actually closes out a job??).

I'm going to try this with queue.getWaiting but from the documentation, this doesn't seem like I'm going to be able to still do what I ultimately am trying to do.

Is remote polling/job work-flow not supported in bull currently? If so, are there API calls that are not documented that I should be using? Thanks for the help.

@manast

This comment has been minimized.

Copy link
Member

manast commented Nov 17, 2017

ok. Not sure I completely understand your case, but If I got it right what you want is to be able to process jobs on demand instead of processing them as soon as they arrive to the queue, is it correct?
There is no API for doing that, but you probably could write such a function yourself and monkey patch the queue (or submit a PR if it makes sense to have it as a feature in bull). Basically you should write a method that does a getNextJob( https://github.com/OptimalBits/bull/blob/master/lib/queue.js#L920) and directly after a processJob(https://github.com/OptimalBits/bull/blob/master/lib/queue.js#L838) where you specify your handler function (processing function for the job) and you probably do not want to call blpoprpush and instead just call moveToActive which returns a job if there is any or undefined otherwise. Good luck!

@travissutton

This comment has been minimized.

Copy link
Author

travissutton commented Nov 17, 2017

You have it correct for the most part. I have it working using private functions to see if it was even possible. Here's the workflow:

  1. Job comes into queue via some other service.
  2. Remote agent asks queue service over http if there are any jobs via queue.getNextJob, so that it get the moveToActive call made.
  3. Queue service returns Job data to agent, agent parses data and does work. This includes job.id for referencing back to Queue Service
  4. Agent provides update to Queue Service via http - progress, etc - using combination of queue.getJob and job.progress
  5. Agent provides completed notification to Queue service via http - Queue service does queue.getJob->job.progress(100)->job.moveToCompleted
  6. Agent asks Queue Service if it has another task to do (restart)

The requirement of being able to process on-demand was the key. The end result seems like it may work really well, though I don't like that I'm using private functions. Are there any plans to make those methods public? Specifically the getNextJob method?

@manast

This comment has been minimized.

Copy link
Member

manast commented Nov 17, 2017

No, we do not have plans for that I am afraid. I would recommend that you write a few unit tests specifically for testing the code using the private functions, in that way hopefully you will catch eventual changes to those APIs when upgrading and you will have a chance to react accordingly.

@manast manast added the question label Nov 17, 2017

@travissutton

This comment has been minimized.

Copy link
Author

travissutton commented Nov 21, 2017

Will do & thank you for the help!

@aleccool213

This comment has been minimized.

Copy link
Contributor

aleccool213 commented May 23, 2018

I am going to make a PR which specifies this as a pattern and possibly exposes some private funcs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment