[NEW] Search Provider Framework #10110

tkurz · 2018-03-12T21:29:49Z

@RocketChat/core , @mrsimpson , @geekgonecrazy , @engelgabriel, @rodrigok
Relates RocketChat/feature-requests#745, #9651, #8714, #1615

Rocket.Chat Search Providers

In this pull request I introdude a provider concept for chatpal search. Beside the providerService
itself it contains 2 reference implementations:

The Default Provider (package rocketchat-search), which uses the messageSearch function of Rocket.Chat based on MongoDB and rebuilds the current search interface,
The Chatpal Provider (package chatpal-search), which uses an external Service based in Apache Solr. The service can be hosted on-site (and can be downloaded from dockerhub) or be used as a service (the API key can be generated on RC Admin directly.

In addition to the 2 packages it adds some i18n variables and does smaller adaptions to the result model of the messageSearch function.

Providers

A new Provider just extends the provider class and registers itself in the SearchProviderService.

class MyProvider extends SearchProvider {
	constructor() {
		super('myProvider'); //a unique id for the provider
	};
	
	search(text, context, payload, callback) {
		//do some search and call the callback with the result
	};
}

searchProviderService.register(new MyProvider());

Settings

In order to enable Settings within the admin UI for your own provider, you can add it (e.g. in the constructor).

this._settings.add('PageSize', 'int', 15, {
			i18nLabel: 'Search_Page_Size'
		});

The setting values are loaded, when you use your provider. The values can be easily accessed.

this._settings.get('PageSize')

The settings can be set in Search section in the admin ui:

Search UI

Search provider can have their own result template. The template is loaded with data.

{
 searching, //reactive var if search results are loading
 result, //reactive var with the result 
 text, //reactive var with the search text
 settings //the settings of the provider,
 parentPayload, //the main search payload (not reset for new searches)
 payload, //the payload (reseted when new search is issued from search field)
 search //the search function
}

The default search ui looks like the existing ones:

The chatpal UI supports also rooms and users and looks like this:

Search Result

In order to provide a proper validation of the results the search function of the provider must follow at the following (extendable) format:

{
  message: {
    docs:[{_id},...]
  },
  room: {
    docs:[{_id},...]
  },
  user: {
    docs:[{_id},...]
  }
}

I am happily looking forward to your comments and hope you like my approach!

# Conflicts: # .meteor/packages # package-lock.json

CLAassistant · 2018-03-12T21:29:56Z

All committers have signed the CLA.

…fault again

…atpal

tkurz · 2018-04-19T13:09:32Z

Hi all!
All the issues should be fixed now! I think now we are (at least from our side) ready to merge ;)
Best regards
Thomas

mrsimpson · 2018-04-19T18:45:21Z

I am so much looking forward to using this on open.rocket.chat - imho the most important contribution for months!

…st, too

…dlink-gmbh/Rocket.Chat into redlink-local/#8381-search_provider_concept

tkurz · 2018-04-19T19:58:50Z

I just did a minor extension of the suggestion functionality to overcome an issue with specific filters (e.g. for messages only) within search providers. @karlprieb this should not influence the positive review from your side right?

mrsimpson · 2018-04-19T21:02:08Z

@tkurz can you add a recent screenshot - we need more teasers.

graywolf336 · 2018-04-19T21:06:31Z

Yes, if you don't mind adding screenshots that we can use for our release blog post that would be awesome!

tkurz · 2018-04-19T22:41:15Z

Here we go, some screenshots ;)

Basis:

You can now select your Search Provider in the Search Admin page:

Chatpal

Some features of Chatpal, a reference implementation for the search provider framework.

Multiple Result Types (currently Users, Messages, Rooms), also properties for types (e.g. room topics) are searchable

Highlights of search terms within results

Search-Term suggestions

Some more infos:

The Chatpal Backend is based on Solr (a high performant search index), which allows better search (like stemming, language specific pre-processing etc) and high scalability. Chatpal can be installed on-site or used as hosted Service.

sampaiodiego

found a few more things..

everything else looks good.

if you can fix them we can still merge tomorrow.

sampaiodiego · 2018-04-19T23:28:04Z

packages/rocketchat-search/server/service/validationService.js

+					SearchLogger.debug(`user ${ uid } can access ${ msg.rid } ( ${ subscription.t === 'd' ? subscription.username : subscription.name } )`);
+				} else {
+					SearchLogger.debug(`user ${ uid } can NOT access ${ msg.rid }`);
+				}console.log(msg);


oops.. found a console.log

sampaiodiego · 2018-04-19T23:33:12Z

server/methods/messageSearch.js

@@ -226,7 +226,7 @@ Meteor.methods({
 				};
 			}

-			result.messages = RocketChat.models.Messages.find(query, options).fetch();
+			result.message.docs = RocketChat.models.Messages.find(query, options).fetch();


any reason to change the result schema? I'm afraid this could break some clients. if there is no strong reason for this change, I would ask to stick with the old schema.

The result schema has been changed because we wanted to allow some metadata to the search results (e.g. number of matching documents, scoring, etc) In order to keep one schema (necessary e.g. for result validation) for all providers, we changed it also here. Nevertheless, I changed this in the code in api, which uses his function, too . If you want to change there we can have the solutions: add a parameter format to the function, that is default basic but can be set to extended
What do you think?

@tkurz what about adding another property to the result which is exclusively dedicated to the metadata?

result: { messages: [...], meta: { messages: { totalCount: 17177882, pagesize: 50, currentPage: 37 }, user: { language: 'prBR' } }

I don't think it's a good style to change the schema of a response, but having optional parts for which a projection might be specified by the consumer is fine.

Yes, would be a valid solution, but enforces some major changes in the current Chatpal implementation. What about this: Keep the "old" schema in the messageSearch Service and map it in the search provider. so we do not change the current api. IMHO the new schema is more adequate (and common) on search index systems like solr/elastic. Would be easy and would not contain api changes (and this be downwards compatible to 3rd party)

As you said, the new Schema us common for search providers. Since it’s the implementation of the „search“ service, I believe it’s valid to go for .docs - as you also changed the old global search implementation and the api to match, I don’t see a constraint to use this new schema. It was a beta in the current version explicitly anyway. But I think you should discuss this with @diegosampaio (probably on open.rocket.chat) during the course of today. 🇧🇷 ist close to waking up now anyway ;-)

@rafaelks do the mobile apps utilize the real time method or the endpoint for searching? Or do they not search messages yet?

@graywolf336 We're implementing using this API: https://rocket.chat/docs/developer-guides/rest-api/chat/search/.

Thanks, I will take a look!

@graywolf336 the rest endpoint uses still current search service (mongo based). And it returns the same result format as before. So schema change should not influence the app right? In a next version we can change the method (and maybe the api) so it supports both, basic and extended schema version. Then it would also be possible to use the current provider for API as well. But for now it should be fine right?

yup.. seems we're good with mobile apps.. I'll still add a [BREAK] flag for third parties who still rely on DDP method.

looking forward to the next version with metadata on REST endpoints 😃

sampaiodiego · 2018-04-19T23:39:56Z

server/methods/messageSearch.js

@@ -25,7 +25,7 @@ Meteor.methods({
 			if (!Meteor.call('canAccessRoom', rid, currentUserId)) {
 				return result;
 			}
-		} else if (RocketChat.settings.get('Message_GlobalSearch') !== true) {


looks like we don't need the old setting any more right? if so, I would ask to remove Message_GlobalSearch

I have to check this.

You mean that is should be removed from the config?

Rocket.Chat/packages/rocketchat-lib/server/startup/settings.js

Line 1372 in ee06a91

this.add('Message_GlobalSearch', false, {

yes. a migration will also be needed to reflect the current value of Message_GlobalSearch into the new Search.defaultProvider.GlobalSearchEnabled setting.

additionally just found that old templates are also not being used anymore, so we would need to also delete the files:

messageSearch.html

messageSearch.js

and remove its references at:

rocketchat-ui-flextab/package.js#L24

rocketchat-ui-flextab/package.js#L31

Fully agree: @sampaiodiego Is there any standard way for setting migration or do we need to do it programatically (in our code).

the migration is simple.. take a look at migration 112 (you'll need to create the version number 113) .. you just need to find the old setting, grab its value and run an upsert (the settings might have been created yet, so an upsert is more recommended for this case) on the new setting with the old setting's value. after that you can remove the old setting.

Cool, i will commit this as soon it is ready.

@sampaiodiego I added the migration. Can you please check? Should do the trick right?

you forgot to remove the setting :) I just did it

👍

…dlink-gmbh/Rocket.Chat into redlink-local/#8381-search_provider_concept

…tting (fixes RocketChat#10110 (comment))

nkeilar · 2024-01-26T22:22:03Z

My team has been complaining that this feature isn't enabled in our self hosted rocket chat, making us impossible to fully ditch slack. I know its 6 years old, but is there anyone out there who still uses global search - can anyone recommend how to set it up, at this point we'd even contact the task.

tkurz · 2024-01-29T10:23:10Z

Actually it was quite easy, see https://github.com/chatpal and especially https://github.com/chatpal/chatpal-search-standalone to run a search instance. But as the project is retired I cannot guarantee it works. It is Open Source so feel free to make PRs if necessary!!

tkurz added 12 commits February 22, 2018 09:38

first concept

71ea121

first provider implmentation for mongo search

bba0793

Merge branch 'develop-upstream' into #8381-poc

2940c2f

# Conflicts: # .meteor/packages # package-lock.json

add some admin features

155f590

smaller fixes and restructuring

db6211d

many adaptions regarding ui, smaller fix for 'searchMessages'

11775f0

add search example directly to i18n strings

c1fc513

set multiple languages

feb137b

smaller fixes regarding ui, i18n; some documentation

4d0e10b

Merge branch 'develop-upstream' into #8381-poc

91c23d2

Add i18n description for url parameter

caf3628

move Readme to the right place

e0328e8

smaller changes for linter

152fc55

RocketChat deleted a comment Mar 12, 2018

test failed due to class renaming, should fix the test

3e00331

tkurz mentioned this pull request Mar 14, 2018

[NEW] Global message search (beta: disabled by default) #9687

Merged

tkurz added 3 commits March 14, 2018 12:19

enabled stopping of running indexing process in chatpal

12d019f

fixes missing channel names on direct messages

48fe6fb

enable on keyup search for providers without autocompletion

4f19d54

RocketChat deleted a comment Mar 14, 2018

tkurz added 3 commits March 14, 2018 14:51

default search result type for chatpal is now configurable, ALL is de…

09afe77

…fault again

fixes smaller styling issue

ea394f8

filter non valid (non accessible) results

768c11a

RocketChat deleted a comment Mar 14, 2018

fix naming bug in validation service

b4ffaa9

RocketChat deleted a comment Mar 14, 2018

fix some typos and add german translation for search framework and Ch…

e3a28d0

…atpal

karlprieb previously approved these changes Apr 19, 2018

View reviewed changes

Merge branch 'develop' into #8381-search_provider_concept

fe5cf57

sampaiodiego moved this from Backlog to Review/QA in 0.64.0 Apr 19, 2018

tkurz added 2 commits April 19, 2018 21:56

add payload from searches (like e.g. type filter) to suggestion reque…

08449c9

…st, too

Merge branch '#8381-search_provider_concept' of https://github.com/re…

eb1561c

…dlink-gmbh/Rocket.Chat into redlink-local/#8381-search_provider_concept

tkurz dismissed karlprieb’s stale review via eb1561c April 19, 2018 19:57

Merge branch 'develop' into #8381-search_provider_concept

facb8a3

sampaiodiego requested changes Apr 19, 2018

View reviewed changes

tkurz and others added 7 commits April 20, 2018 08:11

fix typo

39e8fcb

remove console log (fixes RocketChat#10110 (comment))

36b79c0

Merge branch '#8381-search_provider_concept' of https://github.com/re…

dfecb5e

…dlink-gmbh/Rocket.Chat into redlink-local/#8381-search_provider_concept

remove current search ui and add migration of Message_GlobalSearch se…

4ef4883

…tting (fixes RocketChat#10110 (comment))

Merge branch 'develop' into #8381-search_provider_concept

af1f3d2

Remove Message_GlobalSearch setting

7acd770

Update and rename v113.js to v114.js

1f1185d

sampaiodiego approved these changes Apr 20, 2018

View reviewed changes

rodrigok changed the title ~~[NEW] Add Search Provider Framework~~ [NEW] Search Provider Framework Apr 20, 2018

rodrigok merged commit 68b0641 into RocketChat:develop Apr 20, 2018

0.64.0 automation moved this from Review/QA to Done Apr 20, 2018

rodrigok mentioned this pull request Apr 28, 2018

Release 0.64.0 #10613

Merged

ggazzo added the Invalid PR Title label Mar 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NEW] Search Provider Framework #10110

[NEW] Search Provider Framework #10110

tkurz commented Mar 12, 2018 •

edited

Loading

CLAassistant commented Mar 12, 2018 •

edited

Loading

tkurz commented Apr 19, 2018

mrsimpson commented Apr 19, 2018

tkurz commented Apr 19, 2018 •

edited

Loading

mrsimpson commented Apr 19, 2018

graywolf336 commented Apr 19, 2018

tkurz commented Apr 19, 2018

sampaiodiego left a comment

sampaiodiego Apr 19, 2018

sampaiodiego Apr 19, 2018 •

edited

Loading

tkurz Apr 20, 2018

mrsimpson Apr 20, 2018 •

edited

Loading

tkurz Apr 20, 2018

mrsimpson Apr 20, 2018

graywolf336 Apr 20, 2018

rafaelks Apr 20, 2018

tkurz Apr 20, 2018

tkurz Apr 20, 2018

sampaiodiego Apr 20, 2018

sampaiodiego Apr 19, 2018

tkurz Apr 20, 2018

tkurz Apr 20, 2018

sampaiodiego Apr 20, 2018

tkurz Apr 20, 2018

sampaiodiego Apr 20, 2018 •

edited

Loading

tkurz Apr 20, 2018

tkurz Apr 20, 2018

sampaiodiego Apr 20, 2018

nkeilar commented Jan 26, 2024

tkurz commented Jan 29, 2024

[NEW] Search Provider Framework #10110

[NEW] Search Provider Framework #10110

Conversation

tkurz commented Mar 12, 2018 • edited Loading

Rocket.Chat Search Providers

Providers

Settings

Search UI

Search Result

CLAassistant commented Mar 12, 2018 • edited Loading

tkurz commented Apr 19, 2018

mrsimpson commented Apr 19, 2018

tkurz commented Apr 19, 2018 • edited Loading

mrsimpson commented Apr 19, 2018

graywolf336 commented Apr 19, 2018

tkurz commented Apr 19, 2018

Basis:

Chatpal

Some more infos:

sampaiodiego left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sampaiodiego Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrsimpson Apr 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sampaiodiego Apr 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkeilar commented Jan 26, 2024

tkurz commented Jan 29, 2024

tkurz commented Mar 12, 2018 •

edited

Loading

CLAassistant commented Mar 12, 2018 •

edited

Loading

tkurz commented Apr 19, 2018 •

edited

Loading

sampaiodiego Apr 19, 2018 •

edited

Loading

mrsimpson Apr 20, 2018 •

edited

Loading

sampaiodiego Apr 20, 2018 •

edited

Loading