A full MongoDB query language implementation with INDEXES for querying your levelup/leveldb database.
Clone or download
Latest commit 557d87c Apr 6, 2015
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data initial commit Jul 29, 2013
test no more byteup May 13, 2014
.gitignore initial commit Jul 29, 2013
.travis.yml remove 0.8 support May 13, 2014
README.md Update README.md Apr 1, 2015
index.js all tests passing Jul 29, 2013
jsonquery-plan.js better undefined support Aug 14, 2013
package.json 1.0.2 Feb 25, 2015

README.md

jsonquery-engine

A full MongoDB query language implementation with INDEXES for querying your levelup/leveldb database.

This is a plugin for level-queryengine.

build status

Installation

Install through npm:

$ npm install jsonquery-engine

Usage

var levelQuery = require('level-queryengine'),
    jsonqueryEngine = require('jsonquery-engine'),
    pairs = require('pairs'),
    levelup = require('levelup'),
    db = levelQuery(levelup('my-db', { valueEncoding: 'json' }));

db.query.use(jsonqueryEngine());

// index all the properties in pairs
db.ensureIndex('*', 'pairs', pairs.index);

// alternatively you could just index the properties you want:
// db.ensureIndex('num');
// db.ensureIndex('tags');

db.batch(makeSomeData(), function (err) {
  // compound mongodb / jsonquery query syntax
  db.query({ $and: [ { tags: 'tag1' }, { num: { $lt: 100 } } ] })
    .on('data', console.log)
    .on('stats', function (stats) {
      // stats contains the query statistics in the format
      //  { indexHits: 1, dataHits: 1, matchHits: 1 });
    });
});

Example Queries

I'm using my jsonquery module to implement that final, ultimate mongodb syntax.

This module adds awesome INDEX support to the syntax, so you're not just filtering your entire database stream, but taking advantage of any indexes that are set up using level-queryengine

Here are some sample queries from the test suite. They all will take advantage of any indexes for filtering before looking up values.

// will use indexes for quick retrieval if present
{ 'name': 'name 42' }

// if both fields are present, then indexes will be used before hitting values
{ $or: [ { num: 420 }, { name: 'name 42' } ] }

// $ands are smart so that if one of the fields is indexed, that will be used for retrieval
{ $and: [ { tags: 'tag1' }, { num: { $lt: 100 } } ] }

// can search efficiently for items in array. eg: { tags: [ 'tag1', 'tag4' ] }
{ tags: 'tag4' }

// will still require a full index scan, but depending on your data it won't need to do a full db scan
{ 'name': { $ne: 'name 42' } }

// smart enough to use levelups sorted indexes to efficiently do range queries BEFORE fetching data
{ 'num': { $gte: 500 } }

// smart enough to turn these both into { 'num': { $lte: 500 } } and use and index range lookup
{ $not: { 'num': { $gte: 500 } } }
{ 'num': { $not: { $gte: 500 } } }

// index scan
{ num: { $mod: [200, 0] } }

// will use indexes
{ num: { $in: [420, 70] } }

// $nins suck - table scan
{ num: { $nin: [420, 70] } }

// will use indexes for efficient retrieval
{ tags: { $all: ['tag2', 'tag4'] } }

// will use indexes for efficient retrieval
{ tree: { $elemMatch: { a: 42, b: 43 } } }

// will use indexes for efficient retrieval
{ 'tree.a': 42 }

// index scan
{ 'name': /^name 4/ }

Indexing Strategy Support

Currently two index strategies are supported:

  • 'property' (default) - index the property defined by the indexName. If you don't pass in any emitFunction (or indexType) then this indexing strategy will be used by default.
  • 'pairs' - used by the pairs module and jsonquery-engine to index "pairs" of object properties to allow arbitrary object queries with a reasonable tradeoff between index size and query performance.

To use the alacarte 'property' system:

db.query.use(jsonqueryEngine());

// index these properties
db.ensureIndex('num');
db.ensureIndex('tree.a');

db.query(...);

To use the 'pairs' strategy, which effectively indexes almost EVERY field, with a nice balance between selectiveness and index size:

var pairs = require('pairs');
db.query.use(jsonqueryEngine());

// index all pairs of properties
db.ensureIndex('*', 'pairs', pairs.index);

db.query(...);

This will enable you to do effective ad-hoc queries on practically any field. But, be aware the pairs indexing can be VERY large.

TODO

This project is under active development. Here's a list of things I'm planning to add:

  • There are still some bugs in the jsonquery query syntax. Eg: { name: { first: 'bob' } } doens't currently work properly.
  • support the 'full-path' indexing strategy.
  • joins?