Fix pagination in Content Delivery API Index Helper

[Brynjarth]

Resolves: #18683

Description

When rebuilding the Content Delivery API Index it only fetches 10.000 descendants at maximum currently. The logic that was used was flawed and was always stopping at 10.000 items.
This fix updates the logic so that it can fetch all of the descendants and insert them into the index.
We tested this on our own website and this fix is currently working in production.

[github-actions]

Hi there @Brynjarth, thank you for this contribution! 👍

While we wait for one of the Core Collaborators team to have a look at your work, we wanted to let you know about that we have a checklist for some of the things we will consider during review:

• It's clear what problem this is solving, there's a connected issue or a description of what the changes do and how to test them
• The automated tests all pass (see "Checks" tab on this PR)
• The level of security for this contribution is the same or improved
• The level of performance for this contribution is the same or improved
• Avoids creating breaking changes; note that behavioral changes might also be perceived as breaking
• If this is a new feature, Umbraco HQ provided guidance on the implementation beforehand
• 💡 The contribution looks original and the contributor is presumably allowed to share it

Don't worry if you got something wrong. We like to think of a pull request as the start of a conversation, we're happy to provide guidance on improving your contribution.

If you realize that you might want to make some changes then you can do that by adding new commits to the branch you created for this work and pushing new commits. They should then automatically show up as updates to this pull request.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[emmagarland]

Hi @Brynjarth

Thanks for your PR to resolve #18683 with the maximum amount of results.

One of the core contributors team will take a look - I think it's one HQ might want to confirm too since it was in discussion on the issue thread too.

Cheers,

Emma

[AndyButland]

I'm struggling to see here what the problem was the original code, which makes me nervous to accept the change, even if verified in production, without having understood where the issue was.

Can you help me understand the logic error with the original code please?

Let's assume 25000 documents.

First time through, pageIndex = 0, pageSize = 10000 - retrieves 10000 records, so descendants.Length == pageSize and we loop again with pageIndex incremented.

Second time through, pageIndex = 1, pageSize = 10000 - retrieves a second batch of 10000 records, so descendants.Length == pageSize and we loop again with pageIndex incremented.

Third time through, pageIndex = 2, pageSize = 10000 - retrieves the last 5000 records, so descendants.Length != pageSize and we stop.

[gardarthorsteins]

Hi Andy,

Its actually this that fixes the issue: while (descendants.Length > 0 && pageIndex < total);

The problem is that the current code is looking for an exact match while (descendants.Length == pageSize);

But .Where(descendant => _deliveryApiSettings.IsAllowedContentType(descendant.ContentType.Alias)) is filtering out values so the end results is less then 10000. This breaks the loop.

My guess is that only this index has this specific where command and why its the only one breaking and not the other ones.

This means that only projects with higher then 10000 nodes and are using the DisallowedContentTypeAliases feature are having this issue.

This is tested with version 13.8.1

@AndyButland

[AndyButland]

Thanks @gardarthorsteins for the explanation - makes sense. I added an initially failing integration test to first ensure we could replicate the problem via that means, and then to verify your implementation, which looks to work as expected.