Search API - How to prevent duplicate results when using paging (StartRow, RowLimit)

[diabhoil]

What type of issue is this?

Question

What SharePoint development model, framework, SDK or API is this about?

💥 SharePoint Framework

Target SharePoint environment

SharePoint Online

What browser(s) / client(s) have you tested

• 💥 Internet Explorer
• 💥 Microsoft Edge
• 💥 Google Chrome
• 💥 FireFox
• 💥 Safari
• mobile (iOS/iPadOS)
• mobile (Android)
• not applicable
• other (enter in the "Additional environment details" area below)

Additional environment details

Issue description

I have a document library with 599 documents which are more or less very similar. When I perform queries to the search REST API like below, I get - with 2 calls (500+99 entries) - TotalRows = 599 back. But that are not 599 unique rows. I get ~10-50 duplicate entries in the second call for the 99 results which I got in the first 500 rows already. How can I perform the query to get all 599 unique results back. And I already now about Pagination for large result sets, but when I use that, I lose the "default" sorting, because then everything is sorted by DocId.

//this is performed in loop to get more then 500 results
const searchResults = await sp.search({
          Querytext: `(ContentTypeId:0x010100010*) AND (Path:https://tenant.sharepoint.com/sites/XXXX*)`,
          StartRow: startRow,
          SelectProperties: ["Title", "Path", "UniqueID"],
          RowLimit: 500,
          EnableInterleaving: false,
          SourceId: "8413cd39-2156-4e00-b54d-11efd9abdb89",
          TrimDuplicates: false //true wont work here, because the documents are kinda alike
      });

Is there a property I have to set? To double check my results I have included the PnP Modern Search Webpart on one of my pages, displayed the results and checked if every entry on every page (100 entries per page) is unique. And there I got the same problem.

So the number of total results is always correct. But I never get all the unique 599 results back with paging.

[Ashlesha-MSFT]

Thank you for bringing this issue to our attention. We will look into it and get back to you shortly.
If you don’t receive a reply within two working days, please use this escalation form to escalate.

[Ashlesha-MSFT]

@diabhoil,
The behavior you’re seeing is by design in the SharePoint Search API.
When paging is done using StartRow and RowLimit, and the results are sorted by relevance (or any non-deterministic sort), SharePoint Search doesn’t guarantee consistent ordering between requests. As a result, some duplicate or missing items can occur across pages — even though TotalRows is accurate.
To avoid duplicates, you can:
Use a deterministic sort, e.g.:
SortList: [{ Property: "DocId", Direction: "Ascending" }]

This keeps the order consistent, but note it won’t be relevance-based anymore.
Or, if you must preserve relevance sorting, deduplicate results client-side using the UniqueId field.
This is currently expected behavior, not a defect, due to how the Search service handles ranking and pagination.

[diabhoil]

@Ashlesha-MSFT
In my view, not getting all search results, kinda is a "defect". At least an unexpected behavior by the user.

But I already guessed that this is by design and not a "bug". Thats why this is tagged as question.

So am I right that, to get all items in relevant order, I need to perform "two" (or more or less doubled, and only when the results would be paged) queries? One is getting the items in relevant order and one is getting all items in an deterministic way and then fill the results up with the missing items and remove the duplicate ones on client site?

[Ashlesha-MSFT]

@diabhoil,
I understand why this feels like a defect from a user perspective, but based on current Microsoft documentation and design behavior, this is expected when using relevance-based sorting with the SharePoint Search REST API.

When paging through results using StartRow and RowLimit, and results are sorted by relevance, SharePoint Search recalculates ranking each time you issue a query. This means the ordering between requests isn’t guaranteed to remain identical, which can result in duplicate or missing items across pages.

If you need stable paging and all unique items, Microsoft recommends using deterministic pagination based on DocId and IndexDocId, as described in the official documentation: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/pagination-for-large-result-sets

Your idea of using two queries — one for relevance ordering and one for deterministic pagination, then merging and deduplicating client-side — is a community-known workaround, though it’s not officially supported by the API.

[Ashlesha-MSFT]

@diabhoil,
As mentioned earlier, the behavior you’re seeing is expected when using relevance-based sorting with the SharePoint Search API. To get all unique items, deterministic sorting with DocId or client-side deduplication is required.

Could you please confirm that this answers your question? If so, we can go ahead and close this issue.

[diabhoil]

Yes this answers my question. I will close it. Thank you