Add support for additional indexes other than job id

[kbsanders]

It would be helpful to have additional ways to query jobs other than the job id. For example by user id, group id, batch id, etc.

This would allow jobs to be created, viewed and maintained by individual users, among other things.

Here is how I could envision it working, off the top of my head:

var kue = require('kue');
var jobs = kue.createQueue();

// creation...

jobs.create({ // job type/indexes
    type: 'report'
  , indexes: {  
      user_id: '123'   // added to q:indexes:user_id:* set in redis
    , group_id: 'abc'  // added to q:indexes:group_id:* set in redis
    , batch_id: 'xyz'  // added to q:indexes:batch_id:* set in redis
    // ... etc ...
  }
},{ // job payload/data
    report_id: 'quarterly_profits'
  , format: 'pdf'
  , orientation: 'L'
  , pageSize: 'Letter'
  , rangeStart: '2013-01-01'
  , deliver_to: 'me@somewhere.com'
}).save();

// perhaps a better way to handle creation ...

jobs
  .create(...)
  .index({user_id: '123', group_id: 'abc'})
  .save();

// retrieval...

// returns all pending jobs which have job ids 
// indexed under q:indexes:user_id:123

Job.find({
  state: ['delayed','inactive'],
  user_id: '123' 
},function(err,foundJobs){

  if(err){
    console.log(err);
    return;
  }

  foundJobs.forEach(function(job){
    // do something interesting
  });

});

// stats...

// aggregates stats for all jobs indexed for
// q:indexes:group_id:abc

Job.stats({
  group_id: 'abc'
},function(err,stats){

  // stats contains state counts, etc for 
  // all jobs belonging to group id 'abc'

});

[behrad]

+1, but first I want to make sure reds indexes are not leaking as before and are a solid base to add more indexing features on top. So can you test and claim that it doesn't leak indexes in heavy concurrent thunders? i.e. when adding, changing state fast, removing jobs?

second, we want index some part of our job data, so it would be more nice to write

jobs
  .create('report', { 
      title: 'Report Job', 
      user_id: '123', 
      group: {
         id: 'abc'
      } 
   } )
  .index( ["user_id", "group.id"] )
  .save();

This feature may also be used by apps to define how Kue should provide "Job Uniqueness" for them. (#267)

[kbsanders]

👍 on indexing on user-defined job data fields. Seems like a smart way to go.

As for leaks...

I'm fairly new to redis but perhaps everything could be wrapped inside a transaction using MULTI and EXEC? That would ensure that when creating a job, all indexes are guaranteed to be created and when removing a job all indexes are guaranteed to be removed.

http://redis.io/topics/transactions
http://redis.io/commands/MULTI

I'm not familiar with the reds search leak issue but perhaps transactions could benefit there as well?

[behrad]

You may want to read #94
and #218

there was a problem when removing jobs (on some unknown conditions) which
index removal was not done.