Pagination and search response times

[ghiron]

Hi !

we made some performance tests on search requests after our migration to the ES search engine (#742).. We still have some pretty slow response times (e.g over 5s on ~600k rows).

I looked into code and I see two main reasons of this latency :

• Search results are stored into the database and the system trues to fetch all results (in an extra thread) from the DB even if only one result is required
• Pagination is done at the "controller" layer instead of using the DB/Elasticsearch pagination capability. Because of that, a lot of data is fetched from the DB.

Primary tests I've done, show an improvement on response time and response time stability if search result storage is removed and pagination is used at DB layer.

I plan to make a PR with these modifications :

• Move the Pagination from the controller to DAO layer
• Disable the search result storage

This PR would at this time break a feature like *BaseSubscriptionInterceptor but we welcome more insight into how the feature is used currently to adapt our PR. This PR would also impact several method and interface signatures which will be updated.

We welcome any feedback !

this issue looks related to
#481
#536

[storni]

Hi @ghiron, have you arrived to a solution?

BTW, I've found a way to improve the performance of the search result storage with minimal changes in the code. See #866 for details.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.