Proposition to reduce search key explosion

[aventuralabs]

I'm new to Kue, but thus far I really like the interface and I find it to be well thought out.

However, I've been encountering problems similar to those discussed in #94, #98 and others.

Here's my proposal to fix it (and happy to submit a pull request):

1. While reds was created for full-text search, this feature is not germane to a job queue. Nonetheless, it is useful.
2. The Job constructor should have a third parameter e.g. Job(type, data, lookups). Lookups will be a string of fields that full-text searching is performed on, separated by spaces. For example 'key3 key21 key33'.
3. In the update function, the json passed to getSearch will only include the keys specified by the user. For backwards compatibility, if lookup is left undefined, the traditional indexing will occur. If an empty string is submitted, no indexing will be done.
4. The Queue object will then have Queue#lookup that allows a user to search for particular jobIds.

Using an extreme case as an example results in fairly sizable savings:

var kue = require('./index.js');

var jobs = kue.createQueue();

var arr = [];
for (var i = 0; i < 500; ++i) {
  arr.push(Math.random());
}

for (var i = 0; i < 1000; ++i)
  var job = jobs.create('test-job', { W : '12345678', A : 'flying', D : 'squirrels', numbers : arr }).save();

Before proposed changes:

used_memory_human:60.80M
db0:keys=2516,expires=0

After proposed changes:

used_memory_human:12.08M
db0:keys=2010,expires=0

Note that the majority of the keys are attributable to objects that have been created but not removed.

Finally, this would also consolidate job removal as the number of search entries to remove would be reduced substantially.

Let me know thoughts and I'll submit a pull request.

Brandon

[aventuralabs]

Closing this as content is reflected in the above pull request.