Using the Scroll API with KNP Paginator

[roelveldhuizen]

I am searching for a way to use the Scroll API with the KNP paginator. Now I get the following exception when going to 1918 of my results. I could not find any documentation on this.

Result window is too large, from + size must be less than or equal to: [10000] but was [46032]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. [reason: all shards failed]

Current code looks like bellow, this works fine as long as you dont huge amounts of results.

    /**
     * @Route("/search", name="search")
     */
    public function index(Request $request, PaginatorInterface $paginator, RepositoryManagerInterface $repositoryManager)
    {
        $q = $request->query->get('q');

        $finder = $repositoryManager->getRepository(Product::class);

        $results = $finder->createPaginatorAdapter($q, ['search_type' => 'dfs_query_then_fetch']);
        $pagination = $paginator->paginate(
            $results, /* query NOT result */
            $request->query->getInt('page', 1), /*page number*/
            24 /*limit per page*/
        );

...

        return $this->render('search/index.html.twig', [
            'title' => 'Searching: ' . $q,
            'controller_name' => 'SearchController',
            'pagination' => $pagination,
            'productPrices' => $productPrices,
        ]);

    }

[modstore]

Also interested in this. I increased max_result_window: 50000 for now, but this isn't an ideal solution.

[modstore]

I added a scroll paginator to my fork but I only added the minimal that I needed to get it working: master...modstore:add-scroll-paginator
You can use it like:
$results = $finder->createScrollPaginatorAdapter($query, ['expiryTime' => '20m']);
When I've got more time, I can implement the other methods and add tests if it's useful for others.

[XWB]

@modstore Would you still be interested in making a PR based on your fork?

[modstore]

@XWB I'll go back to it and make a PR as soon as I've got some time.

[XWB]

Excellent :)

[modstore]

Hey @XWB sorry I haven't been able to get to this yet. Unfortunately I'm getting busier at the moment not less, so I'm not sure if/when I'll be able to get to this. I have been using this on a live site for csv generation for quite a while now, so I am happy to make a PR with it as is. Otherwise I guess it's just good as a reference for whoever needs it and can get to it.