Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of mapping search by creator #170

Open
stefandesu opened this issue Apr 21, 2022 · 1 comment
Open

Improve performance of mapping search by creator #170

stefandesu opened this issue Apr 21, 2022 · 1 comment
Labels
backend Requires work on the backend cleanup

Comments

@stefandesu
Copy link
Member

When searching mappings by creator, it was recently adjusted so that the search is performed via a regular expression that allows partial and case-insensitive search. This was done so that when searching for mappings by a creator in Cocoda, searching for "stefan" will also return results for "Stefan" and "Stefan Peters" etc.

When I tested this feature, I was surprised how performant it was despite using a regular expression that, I assumed, did not use any index. However, in retrospect this is obvious that it was only fast because the development instance doesn't have much data in it. With the release instance of jskos-server, we have almost 400k mappings, and the same search takes about 2-3 seconds.

Since there's no index for the kind of search we want, we probably need some kind of n-gram approach, similar to what we're doing for schemes and concepts in addKeywords (utils/searchHelper.js). This is a bigger change that should be well tested, thus I'm creating this issue here.

For now, this doesn't have a big priority because creator search will likely be performed rarely. But we should do something about it in the long run. We should probably also develop some kind of "big data performance tests" so that we generate a large amount of fake data and test how each kind of request performs.

@stefandesu
Copy link
Member Author

As a first step, I change the creator search so that the label fields are only searched for strings that are not a valid URI, and the URI fields are only searched for strings that are a valid URI. This should improve performance for the feature that we recently added to Cocoda (gbv/cocoda#520) and also when searching for owns own mappings in Cocoda.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Requires work on the backend cleanup
Projects
None yet
Development

No branches or pull requests

2 participants