Improve Search User API

[mrkvon]

Follow up of #531.

Improve /api/users?search=string

• pagination
• expand the user profile fields returned, or allow user to specify which fields to return
• improve search via usernames; figure out how dots, dashes etc. are treated by text index
  it is working fine for periods, hyphen and underscores. If you put period or hyphen into the search it works the same like a space

[mrkvon]

ping @Akronix

Feel free to update the issue with further requirements.

[Akronix]

How about fuzzy search?
Searching for "Abe" should return all instances of Abel, one of them myself: Abel Por Qué No.
And better, searching for "Avel" (similar names) should find "Abel" as well...

[simison]

Searching for "Abe" should return all instances of Abel

So this was possible with MongoDB regexp queries but the issue was that they weren't relying on indices and thus potentially very slow — is that correct @mrkvon ?

I wonder if we could do 2nd expensive query with regexp if first one doesn't yield results?

And better, searching for "Avel" (similar names) should find "Abel" as well...

I agree that's how it should work ideally. Great if it's easy possible and performant enough with MongoDB Indices but I suspect it won't be either. :-( Investing time and maintenance energy in something like ElasticSearch at this point probably isn't worth the effort on the other hand?

[guaka]

Sounds like premature scaling. How slow is a regexp search really? How many searches do we expect on a daily basis?

…

On Wed, Jan 2, 2019 at 3:51 PM Mikael Korpela ***@***.***> wrote: Searching for "Abe" should return all instances of Abel So this was possible with MongoDB regexp queries but the issue was that they weren't relying on indices and thus very slow — is that correct @mrkvon <https://github.com/mrkvon> ? I wonder if we could do 2nd expensive query with regexp if first one doesn't yeld results? And better, searching for "Avel" (similar names) should find "Abel" as well... I agree that's how it should work ideally. Great if it's easy and performant enough with MongoDB Indices but I suspect it won't be either. :-( Investing time and maintenance energy in something like ElasticSearch at this point probably isn't worth the effort on the other hand? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#944 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADVyPJ3cOErH9gNMVHt9OEZqgmicZu9ks5u_OOHgaJpZM4ZKWUU> .

[simison]

Yep, that's a valid point. If someone whips up a PR I'm happy to help to measure with production data (~34K users).

[guaka]

The admin search I've been working on works pretty well with 34k users - it's quite crude but does the job well. #960

…

On Wed, Jan 2, 2019 at 3:57 PM Mikael Korpela ***@***.***> wrote: Yep, that's a valid point. If someone whips up a PR I'm happy to help to measure with production data (~34K users). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#944 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADVyBOrfcN-c6O96d6EeQRi_Yz2MRfUks5u_OTtgaJpZM4ZKWUU> .

[mrkvon]

potentially very slow

I believe the main issue is being a potential vector of DoS attack. Not the slowness on its own.
So, can we limit amount of requests from one user per second for a specific API endpoint?

Also, unlike a search using indices, this will get worse as the amount of members grows.

[simison]

I believe the main issue is being a potential vector of DoS attack. Not the slowness on its own.
So, can we limit amount of requests from one user per second for a specific API endpoint?

Yes, we already do that for the whole /api/* but adding a separate more strict "bucket" for specific API endpoints is very straightforward.

It's configured in Nginx.

[mrkvon]

Then go on with the regex. Just make a restrictive rate limit and be aware that it's a temporary solution which is not scalable as the amount of members grows.

https://docs.mongodb.com/manual/reference/operator/query/regex/#index-use

Some case-sensitive search (i.e. ?search=abel doesn't find Abel) can use indices.
Eventually, we could create lowercase versions of searched fields to have (pseudo)case-insensitive search with index.

Or we can migrate to a database which supports more option. 😅 👎

[Akronix]

Good discussion.

I'd go ahead with regexp.
Then, when scaling become an actual issue, we can consider moving to elasticsearch.

[simison]

Yep and yep. 👍

We can deploy it as such to prod and follow up with regexp version but it's not urgent; we have something that already works.

[github-actions]

This issue is marked as unloved because it has not had any activity for 180 days.

It doesn't mean it's not important, so please remove the unloved label if you like it, or add a comment saying what it means to you :) If this was a bug, maybe you can test to see if this is still an issue?

However, if you just leave it like this, I'll close it in 14 days to help keep your issues tidy!

Thanks!