Page<T> findAll(Predicate, Pageable) is first doing a count query (to get total count for the Page). Then the actual query is performed.
Querydsl MongodbQuery.java#list() method is used in spring data mongodb code. This method call cursor.size() to construct ArrayList<>(). cursor.size() seems to be making an additional count query call. What looks like a simple findAll(…) call is translating into multiple count queries before the actual query is called.
So what exactly is it that you think we can do about it? Our count query is indeed necessary to calculate the Page. So by deciding to call the paging method, you know what you get. That Querydsl is triggering another count (important: a different one, counting the number of the subset) is hardly something we can do something about, right?
I saw the ticket with the Slice method but right now there's hardly anything we can do about it as we can't have an overload with an alternative return type
Sorry, somehow missed the comment updates. Few things can be done to help users. It was not very obvious that calling this method is 3 times more expensive. And required debugging code in 3 different projects.
a.) Document the behavior. Javadoc, online spring-data-mongodb reference etc
b.) Although the improvement I proposes in spring-data-common is not applicable to this project, since it was assigned to you, the following could be done.
Add Slice<T> findSlice(Predicate predicate, Pageable page) method in spring-data-commons. Of course this will have implications on other sub-projects.
And eventually implement that in spring-data-mongodb
As indicated, we have a separate ticket for the suggestion you made. This however is not a fix but effectively new API that needs more exhaustive consideration than a bug fix that could've worked around a third party library bug. See we're effectively at the following point:
Pagination might require an additional query. This - and how to avoid it if you don't need the page metadata but access to the page content - is documented in the reference documentation. I've added DATACMNS-548 to improve this and include the mentioning of Slice as return value.
The additional count query Querydsl triggers (which is what I considered this ticket to be about) is nothing we can do about but rely on users upgrading to 3.4.2.
The request for a sliceAll(…) or the like is covered in DATACMNS-541.
Especially 2 is why marked this as works as designed as we can't do anything in Spring Data MongoDB's code base to reduce the price of a pagination query. I agree on the documentation side of things as well as the request for a slicing CRUD method, but that's got to be tracked in the according tickets of Spring Data Commons.