Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support $search in query syntax #334

Closed
daffl opened this issue May 20, 2016 · 33 comments
Closed

Support $search in query syntax #334

daffl opened this issue May 20, 2016 · 33 comments
Labels

Comments

@daffl
Copy link
Member

@daffl daffl commented May 20, 2016

I am consolidating all the individual issues and discussions here:

This is a proposed special attribute that allows you to fuzzy match a property. Possibly even multiple properties and/or nested documents.

Suggested syntax:

name: {
  $search: ['alice', 'Alice', 'bo', /$bob/i]
}

Following similar syntax to our other special query filters, this would allow you to filter by a singular value, multiple values (treated like an or) and/or regular expressions directly.

Service Adapter Completion

  • NeDB
  • MongoDB
  • Mongoose
  • Sequelize
  • Knex
  • Waterline
  • RethinkDB
  • Memory
  • Localstorage
  • LevelUp
  • Blob?? (maybe to support fuzzy matching filenames?)
@ekryski ekryski mentioned this issue May 20, 2016
22 of 28 tasks complete
@ekryski ekryski added the Backlog label May 21, 2016
@ekryski ekryski added this to the Auk milestone May 21, 2016
@ekryski ekryski added the Feature label May 21, 2016
@ghost
Copy link

@ghost ghost commented Jul 20, 2016

any news on this feature ?
if it is not implemented , how to fuzzy match a property now without this special attribute ?

@daffl
Copy link
Member Author

@daffl daffl commented Jul 20, 2016

You can create a hook that for e.g. MongoDB converts the $search query to a $regex:

app.service('todos').before({
  find(hook) {
    const query = hook.params.query;
    if(query.name.$search) {
      query.name = { $regex: new RegExp(query.name.$search) }
    }
  }
});
@ghost
Copy link

@ghost ghost commented Jul 21, 2016

the way to make fuzzy match for some field in collection before your comment was to use custom route and write native query with same service db engine like "mongo" or "nedb" .
but this way is very good and show me new use case to use hooks in featherjs

@ghost
Copy link

@ghost ghost commented Jul 22, 2016

what about this hook as simple solution for search in mongodb & nedb

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search) }
      }
    }
    hook.params.query = query
    return hook
  }
}

and simply include it in the

src/services/ServiceName/hooks

like

exports.before = {
  all: [],
  find: [globalHooks.searchRegex()]
}
@ruddfawcett
Copy link

@ruddfawcett ruddfawcett commented Aug 17, 2016

For those interested, you can make the above code posted by @alnour-altegani case insensitive by changing

 query[field] = { $regex: new RegExp(query[field].$search) }

to:

 query[field] = { $regex: new RegExp(query[field].$search, 'i') } // note the 'i'
@beeplin
Copy link
Contributor

@beeplin beeplin commented Oct 7, 2016

so why $search, not $regex?
hope it compatible with mongoDB's original syntax, which is available now out-of-box already, so we don't need to change it in the future.

@ekryski ekryski modified the milestones: Auk, Buzzard Oct 17, 2016
@ekryski ekryski mentioned this issue Oct 17, 2016
5 of 18 tasks complete
@cklmercer
Copy link

@cklmercer cklmercer commented Nov 14, 2016

Just stumbled upon this after fighting with the Mongoose adapter for half an hour. This would be a super nice addition.

Any suggestions for a work-around for the feathers-mongoose adapter?

@daffl
Copy link
Member Author

@daffl daffl commented Nov 14, 2016

There is a working hook two comments above. Mongoose already supports it, you just have to convert the query into a regular expression in a before hook.

By now, I am also leaning more towards not adding this. Defining an abstract search format for all databases isn't really possible. Some support %like% queries, others use regular expressions and some do not support searching at all. Fuzzy matches should be implemented with the features of the database you are using.

@cklmercer
Copy link

@cklmercer cklmercer commented Nov 14, 2016

Thanks for the quick reply.

@daffl
Copy link
Member Author

@daffl daffl commented Dec 8, 2016

Most likely it is anything that the SQL LIKE operator supports for your database.

@daffl
Copy link
Member Author

@daffl daffl commented Jan 4, 2017

@sajov Example linked in the comment above: #334 (comment)

@sajov
Copy link

@sajov sajov commented Jan 4, 2017

got it, my fault. Thanks

my solution with implementation of $or for multiple $search

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search, 'i') }
      }
      if(field == '$or') {
        query[field].map((action, index) => {
            let f = Object.keys(action)[0];
            if(action[f].$search) {
                action[f] = { $regex: new RegExp(action[f].$search, 'i') }
            }
            return action;
        });
      }
    }
    hook.params.query = query
    return hook
  }
}

But how you guys deal with integer?

@ghost
Copy link

@ghost ghost commented Jan 6, 2017

@sajov thank you for sharing your work . but, can you give example of how to use this code to search with $or operator ?

@sajov
Copy link

@sajov sajov commented Jan 6, 2017

@ruddfawcett i use it in a datatable context like this

for (var i = 0;i < opts.tableHeader.length; i++) {
                        let q = {};
                        q[opts.tableHeader[i]] = {$search: queryObj.search.value.value};
                        query.$or.push(q);
                    }

here is an working example https://github.com/sajov
I write README right now

@pimvanderheijden
Copy link

@pimvanderheijden pimvanderheijden commented Jan 16, 2017

@sajov Were you able to write a README ? I'd be very interested to see how I could use this case in combination with an $or -array e.g. like

[
    { _id: searchInput },
    { type: searchInput }
]
@sajov
Copy link

@sajov sajov commented Jan 16, 2017

@MidnightP

Hook example which treat $search as { $regex: new RegExp(value, 'i') };
https://github.com/sajov/riot-crud/blob/master/example/src/hooks/index.js#L50-L53

Client example
https://github.com/sajov/riot-crud/blob/master/tags/themes/bootstrap/views/crud-views.tag#L169-L170

query.$or = [
    {fieldA : {$search: value}},
    {fieldB : {$search: value}},
];

hope that answer your question

@pimvanderheijden
Copy link

@pimvanderheijden pimvanderheijden commented Jan 18, 2017

@sajov Thanks!

I rewrote things a bit and ended up using the hook below for a client side query containing this:

orRegex: [
  { 
    fieldA: {
      $search: 'searchinput'
    }
  },
  { 
    fieldB: {
      $search: 'searchinput'
    }
  },
  et cetera...
]

Hook:

return function(hook) {
    const { query } = hook.params

    if( Object.keys(query).includes('orRegex') )  {
      const or = query.orRegex.map((field) => {
        console.log('Object.keys(field)[0]: ', Object.keys(field)[0] )

        const attribute = Object.keys(field)[0]

        return {
          [attribute]: {
            $regex: new RegExp(field[Object.keys(field)[0]]['$search']), $options: 'ix'
          }
        }
      })

      delete(query.orRegex)

      hook.params.query['$or'] = or
    }
    return hook
  }

Suggestions welcome of course

@daffl
Copy link
Member Author

@daffl daffl commented Jan 18, 2017

Maybe @eddyystop has some thoughts if this can be turned into a common hook.

@eddyystop
Copy link
Contributor

@eddyystop eddyystop commented Mar 23, 2017

Just saw this now. I've created a link in feathers-hooks-common feathersjs-ecosystem/feathers-hooks-common#141

@arve0
Copy link
Contributor

@arve0 arve0 commented Jun 25, 2017

Here is a fuzzy match for NeDB, it searches all properties case insensitive:

module.exports = function (options = {}) { // eslint-disable-line no-unused-vars
  return function (hook) {
    if (hook.params.query && hook.params.query.$search) {
      hook.params.query.$where = fuzzySearch(hook.params.query.$search)
      delete hook.params.query.$search
    }
    return hook
  }
}

/**
 * Returns a $where function for NeDB. The function search all
 * properties of objects and returns true if `str` is found in
 * one of the properties. Searching is not case sensitive.
 *
 * @param {string} str search for this string
 * @return {function}
 */
function fuzzySearch (str) {
  let r = new RegExp(str, 'i')

  return function () {
    for (let key in this) {
      // do not search _id and similar fields
      if (key[0] === '_' || !this.hasOwnProperty(key)) {
        continue
      }
      if (this[key].match(r)) {
        return true
      }
    }
    return false
  }
}

I actually found this to be a bit quicker than single property regex. With fuzzy search I got 47 ms average time for service.find(), versus ~68 ms for single field $regex. Times are average of 50 times loop over 7 different search terms, and the DB contains about 14000 rows, each object has four properties, all of them text. Only one property has a lengthy, 140 chars, text field. I guess the implementation is vulnerable to DOS attacks, but $regex should also be.

@arve0
Copy link
Contributor

@arve0 arve0 commented Jun 26, 2017

Based on my comment above I've made to plugins:

arve0 added a commit to arve0/feathers-docs that referenced this issue Jun 26, 2017
Ref feathersjs/feathers#334:

> We will add documentation for searching to the adapters individually.
ekryski added a commit to feathersjs/docs that referenced this issue Jul 24, 2017
* Add note about searching documents

Ref feathersjs/feathers#334:

> We will add documentation for searching to the adapters individually.

* add REST example to $search section
@ulrichborchers
Copy link

@ulrichborchers ulrichborchers commented Aug 1, 2018

Hi,

stumbled into this thread from querying.md:
https://github.com/feathersjs/docs/blob/master/api/databases/querying.md

... which is referring here.

Noticed a little mistake in the querying sample URI for the $in,$nin example.

If I am not mistaken it should be:
/messages?roomId[$in][]=2&roomId[$in][]=5

instead of:
/messages?roomId[$in]=2&roomId[$in]=5

because $in is an array in the query.

@sajov
Copy link

@sajov sajov commented Aug 1, 2018

Hi Ulrich,

/messages?roomId[$in]=2&roomId[$in]=5

get parsed by qs into

/messages?roomId[$in][0]=2&roomId[$in][1]=5

see the option extended

@zsf3
Copy link

@zsf3 zsf3 commented Dec 31, 2018

@sajov Thank you for the hook. Btw, what is plain array in your code? why are you concatenating it query[field]? Some comments might help in understanding this logic please?

@glazs
Copy link

@glazs glazs commented Jan 6, 2019

Hi everybody! I'm trying to use feathers-nedb-fuzzy-search

My query is: http://localhost:3030/messages?$sort[date]=-1&$skip=0&$search=2

And result: {"name":"BadRequest","message":"Invalid query parameter $where","code":400,"className":"bad-request","data":{"$sort":{"date":"-1"},"$skip":"0"},"errors":{}}

error: BadRequest: Invalid query parameter $where
    at new BadRequest (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/errors/lib/index.js:86:17)
    at _.each (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:48:17)
    at Object.keys.forEach.key (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/utils.js:12:39)
    at Array.forEach (<anonymous>)
    at Object.each (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/utils.js:12:24)
    at cleanQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:41:7)
    at filterQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:107:18)
    at Object.filterQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/service.js:46:20)
    at Object._find (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb/lib/index.js:36:47)
    at callMethod (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/service.js:9:20)
error: TypeError: Expected a string
    at module.exports (/home/glazs/workspace/lmt/server/node_modules/escape-string-regexp/index.js:7:9)
    at fuzzySearch (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb-fuzzy-search/index.js:49:22)
    at Object.<anonymous> (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb-fuzzy-search/index.js:33:34)
    at promise.then.hookObject (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/hooks.js:142:73)

What's wrong with my Feathers/NeDB?

@daffl
Copy link
Member Author

@daffl daffl commented Jan 6, 2019

In the latest versions of all database adapters non-standard query parameters have to be explicitly whitelisted.

nedbService({
  whitelist: [ '$where', '$search' ]
});
@jscottsf
Copy link

@jscottsf jscottsf commented Jan 6, 2019

The whitelisting seems to break feathers-vuex since it uses commons. Not related to this thread tho. I'll dig deeper on that and log an issue over there.

https://github.com/feathers-plus/feathers-vuex

@1MikeMakuch
Copy link

@1MikeMakuch 1MikeMakuch commented Jan 9, 2019

what about this hook as simple solution for search in mongodb & nedb

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search) }
      }
    }
    hook.params.query = query
    return hook
  }
}

and simply include it in the

src/services/ServiceName/hooks

like

exports.before = {
  all: [],
  find: [globalHooks.searchRegex()]
}

This from ghost works for me.

@Zalasanjay
Copy link

@Zalasanjay Zalasanjay commented Nov 5, 2019

While using feathers-mongodb-fuzzy-search module If you got an error like Invalid $regex

Then you can try whitelist attribute to white list that keywords in your mongoose service configuration in <service_name>.service.js file

const options = {
    paginate,
    whitelist: ['$regex', '$options']
};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.