Skip to content
Mongoose plugin for weighted search and pagination
JavaScript Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ci
src
.babelrc
.eslintrc.json
.gitignore
.npmignore
.yarnrc
CHANGELOG.md
CODE_OF_CONDUCT.md
LICENSE
PULL_REQUEST_TEMPLATE.md
README.md
package.json
yarn.lock

README.md

mongoose-athena

A plugin to add weighted search and pagination to your schema.

Codefresh build status

Usage

Install

yarn add @salo/mongoose-athena

Add Athena to your schema:

const athena = require('@salo/mongoose-athena');

MySchema.plugin(athena, {
  fields: [{
    name: 'name',
    prefixOnly: true,
    threshold: 0.3,
    weight: 2
  }, {
    name: 'biography',
    minSize: 4
  }]
});

Then, to use it with weighting you can do:

MySchema.athena({
  query: { /* something to filter the collection */ },
  term: 'Athena',
  sort: 'relevancy', // this is the key to trigger weighting
  page: 1,
  limit: 20
});

This will search name and biography for the term 'athena'. If it is sorted by 'relevancy' then a confidenceScore will be attached to the result. The result looks like so:

{
  docs: [], // matching records in the collection
  pagination: {
    page: Number,
    hasPrevPage: Boolean,
    hasNextPage: Boolean,
    nextPage: Number || null,
    prevPage: Number || null,
    total: Number
  }
}

Or you can use it simply to paginate:

MySchema.athena({
  query: { /* something to filter the collection */ },
  term: 'Athena',
  sort: '-created_at', // this will not add `confidenceScore` to the results
  page: 1,
  limit: 20
});

API

Field options

Field Description Type Default
name The field name in your collection String
prefixOnly Whether to only match from the start of the string or anywhere in the string e.g. 'ob' would match 'bob' with this off but not when it's on Boolean false
threshold Value between 0 and 1. It will only count a score if it is greater or equal to this value Float 0
minSize The length of the string to start matching against. e.g. if minSize is 4 then the term 'bob' will not search against the field Int 2
weight A scaling value to multiply scores by so you can weigh certain fields higher/lower than others Int 1

Response

Field Description Type
docs Array of matching documents Array
pagination.page The current page Int
pagination.hasPrevPage Whether or not there is a previous page Boolean
pagination.hasNextPage Whether or not there is a next page Boolean
pagination.nextPage Value of the next page or null Int || null
pagination.prevPage Value of the previous page or null Int || null
pagination.total Total number of matching documents Int

How it works

The crux of it lies in the calculateScore method in the helpers directory. This uses the Jaro-Winkler distance to compute how close your search term is (e.g. 'Athena') to the text in your database. Additionally text is ranked higher if it appears at the start rather than the end of a string so 'Athena Rogers' will have a higher confidenceScore than 'Rogers Athena'.

One thing to note is that the search term is not split on spaces but text on the database is. So using our previous example where term = 'Athena Rogers' the text in the database is split into ['Athena', 'Rogers']. Now, Athena Rogers doesn't directly match 'Athena' or 'Rogers' (it scores 0.93 and 0.41 respectively) but this score is accumulated (0.93+0.41) and then multiplied by the position in the string and any weighting applied to the field. We could split the search term to get direct matches and higher scores but this would considerably slow the calculation of the score down by an order of magnitude as every part of the search term would need matching to every part of the field. In my testing the current approach lends itself to speed and logical weighting.

Pagination

The pagination is based on mongoose-paginate-v2 and mongoose-aggregate-paginate-v2. Athena's implementation is an amalgamation of both libraries and it transparently determines if the query is an aggregate or not.

const aggregate = MySchema.aggregate();
const result = await MySchema.athena({
  query: fullNameQuery,
  limit: 10
});

Publishing

  1. Before opening a PR, run yarn release:prep locally to add changelog and increment version number on your branch
  2. Open a PR from your feature back to master
  3. When the development pipeline completes, hit merge and the publish pipeline will release your changes

Testing

Athena currently has 100% test coverage.

Roadmap

  • Make options (e.g. weighting, minSize) configurable outside of the schema definition.
  • Add more robust tests to ensure there aren't regressions in options going to pagination (e.g. select, sort, etc.).

Prior art (and disclaimer)

I'm not an expert in any of these fields and have very much relied on a few prior projects to reach this point. There's a very high chance there are more efficient ways to accomplish this and I welcome PRs to help this!

That said, many thanks to:

You can’t perform that action at this time.