Highland.js

[cshum]

Not really an issue. Just another tool may worth looking into: http://highlandjs.org/

Highland is basically a Node.js stream interface for lodash + async.
A very good combination since levelup exposes Node.js Readable stream for querying data.

What may look like using Highland e.g.
https://github.com/fergiemcdowall/search-index/blob/master/lib/matchers/matcher.js

var H = require('highland');

H(reverseIndex.createReadStream({
      start: 'TF~*~' + beginsWith,
      end: 'TF~*~' + beginsWith + '~~~'
}))
.filter(function(data){
  return data.key.substring(data.key.length, data.key.length - 2) == '~~';
})
.map(function(data){
  return [data.key.split('~')[2], data.value.length]; //suggestions
})
.sort(function (a, b) {
  return b[1] - a[1]; //sortedSuggestions
})
.take(10) //splice(0, 10);
.map(function(data){
  return data[0]; //simpleSortedSuggestions
})
.collect()
.pull(callback);

Streaming data can avoid processing the whole batch of arrays into memory.
Which may help when search index reaches certain point of memory consumption.

I am very interested in this project. Would love to send some PRs once I get around with the code.

[mewwts]

Hi @cshum! I would not mind at all working with streams through highland. Great to hear of your interest, and PRs are more than welcome! Thanks for contributing 👍

[fergiemcdowall]

@cshum thats a good point about how streaming data can avoid processing whole batches of arrays into memory. The code looks great, and would be a most welcome addition to the project.

[fergiemcdowall]

The highland stuff that has been put in so far is really good.

[fergiemcdowall]

Closing for now, and agree with the general principle that streams should be used instead of large arrays wherever possible.