job processing results

[lbeschastny]

It's not actually an issue, it's more like a suggestion.

Sometimes you want to get the result of the job processing. So, it'll be great to have some option to store the results of the processing.

The results can be returned from the processing function through the promise:

kickq.process('task', function() {
  var res = do_some_work();
  return When.resolve(res);
});

or through the callback:

kickq.process('task', function(job, data, next) {
  var res = do_some_work();
  return next(null, res);
});

This result could be stored inside the job just like the data and returned along with the job itself through the hotjobPromise or using kickq.get function..

I don't think this option should be turned on by default, but sometimes it may be really useful.

[thanpolas]

Indeed, that would be usefull... But only in the context of a job that has a hotjob... ?!

There's currently a different signature in the next() callback... Quoting:

• callback {Function(optErr, optCallback)} One of the two ways to complete a job is to invoke the provided callback function. If the job was processed successfully then invoke the callback with no arguments. If the job failed invoke the callback with an error message, string.
  • optErr {null | string} If processing failed give an error description.
  • optCallback {Function(err)} An optional callback function that gets invoked after the Kickq finishes the Job Item post-process operations.

I'm not sure now, but I'm gonna examine if / why optCallback is useful and i'll get back on this...

[lbeschastny]

I don't think it will be useful only for hotjobs.

Consider the following two use cases:

1. Suppose that there is a job which may end with multiple statuses. Each status represents different type of successful completion of the job. So, after processing my job I'll be able to check its status to get some insights.
2. Sometimes it's better to create an ordinary job and then to poll it for the result instead of waiting for one. For example, the job may be created by the request from another application (e.g. from the client) which will ask about it's results later.

As for current callback signature, it's perfectly compatible with my suggestion:

callback = function(optErr, optRes, optCallback) {
  if (typeof optRes === 'function') {
    optCallback = optRes;
    optRes = null;
  }
  ...
};

[thanpolas]

I got a little confused with the first example and multiple statuses...

If the job is not a hotjob, and we enable this feature, then we essentially add to kickq a "persistent storage service layer" which doesn't sound too sexy to my ears... It'll shift us out of focus and it really isn't kickq's responsibility to store application / worker data...

There is one more issue to think of when considering this option, the Jobs lifetime. The default setting will permanently purge complete jobs after 1day. One can configure Kickq to purge the jobs the moment they complete...

---

Having said that, i still think there is a case to be made for the hotjobs... altering the cb signature like you suggest will work just fine...

What i currently have in my mind is that the data originating from the worker will never be stored in redis. They will only be transported through pubsub channels and if there's a hotjob that listens it'll pick them up...

I'm trying to avoid storing worker generated data to redis as a design choice...

update Changed phrasing a bit

[lbeschastny]

I thing you're right.
The advantages of not storing any additional data will overcome any disadvantages of not being able to use this feature with ordinary jobs.

I kept in mind the purge mechanism and never mean any persistency, though I missed the case with immediate purging.
The only thing that troubled me was the need to store additional data in redis. But using pubsub will solve this problem.

[lbeschastny]

@thanpolas, you convinced me that this my suggestion is ill-suited for ordinary jobs, but it's still a good case for hotjobs, and I think it's worth implementing.
Don't you agree?