Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QueryDsl findAll(Predicate, Pageable) is expensive [DATAMONGO-994] #1921

Closed
spring-projects-issues opened this issue Jul 17, 2014 · 5 comments
Closed

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Jul 17, 2014

Seshu Pasam opened DATAMONGO-994 and commented

Page<T> findAll(Predicate, Pageable) is first doing a count query (to get total count for the Page). Then the actual query is performed.

Querydsl MongodbQuery.java#list() method is used in spring data mongodb code. This method call cursor.size() to construct ArrayList<>(). cursor.size() seems to be making an additional count query call. What looks like a simple findAll(…) call is translating into multiple count queries before the actual query is called.

I opened a bug against querydsl querydsl/querydsl#856

I also opened a improvement for spring-data-commons to add a findAll(Predicate, Pageable) that returns Slice so that all count calls can be avoided


Affects: 1.5.1 (Dijkstra SR1)

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Jul 17, 2014

Oliver Drotbohm commented

So what exactly is it that you think we can do about it? Our count query is indeed necessary to calculate the Page. So by deciding to call the paging method, you know what you get. That Querydsl is triggering another count (important: a different one, counting the number of the subset) is hardly something we can do something about, right?

I saw the ticket with the Slice method but right now there's hardly anything we can do about it as we can't have an overload with an alternative return type

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Jul 24, 2014

Oliver Drotbohm commented

The related ticket in Querydsl has been closed and is due to be released with Querydsl 3.4.2. Upgrading to that version once it has been release should do the trick

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Jul 24, 2014

Seshu Pasam commented

Sorry, somehow missed the comment updates. Few things can be done to help users. It was not very obvious that calling this method is 3 times more expensive. And required debugging code in 3 different projects.

a.) Document the behavior. Javadoc, online spring-data-mongodb reference etc

b.) Although the improvement I proposes in spring-data-common is not applicable to this project, since it was assigned to you, the following could be done.
Add Slice<T> findSlice(Predicate predicate, Pageable page) method in spring-data-commons. Of course this will have implications on other sub-projects.
And eventually implement that in spring-data-mongodb

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Jul 24, 2014

Oliver Drotbohm commented

As indicated, we have a separate ticket for the suggestion you made. This however is not a fix but effectively new API that needs more exhaustive consideration than a bug fix that could've worked around a third party library bug. See we're effectively at the following point:

  1. Pagination might require an additional query. This - and how to avoid it if you don't need the page metadata but access to the page content - is documented in the reference documentation. I've added DATACMNS-548 to improve this and include the mentioning of Slice as return value.
  2. The additional count query Querydsl triggers (which is what I considered this ticket to be about) is nothing we can do about but rely on users upgrading to 3.4.2.
  3. The request for a sliceAll(…) or the like is covered in DATACMNS-541.

Especially 2 is why marked this as works as designed as we can't do anything in Spring Data MongoDB's code base to reduce the price of a pagination query. I agree on the documentation side of things as well as the request for a slicing CRUD method, but that's got to be tracked in the according tickets of Spring Data Commons.

Does that make sense?

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Jul 24, 2014

Seshu Pasam commented

Sounds good. Querydsl issue is already resolved. So there is just one additional call now for the total count. I guess spring-data-commons issue has wider implications.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants