Search results return `next` parameter even when there are no more results

[allanlasser]

As observed in documentcloud-frontend#1288, a list of search results will include a valid next parameter that points to an empty page of results. That leads to an unnecessary additional network request and a flash of loading state that resolves to an empty list. If we're on the last page of results, next should return null.

[eyeseast]

Example query: https://api.www.documentcloud.org/api/documents/search/?project=224036

[duckduckgrayduck]

• We want to check that the if the num docs < per page, then we will return none for the next page
• Backend should fetch per page +1 documents and it will only return per page documents but if there is exactly per page documents it won't fetch the next (25, 50, 75, 100, etc etc)

[jamditis]

I'd like to work on this. Following the approach discussed by @duckduckgrayduck — for page-number pagination (v1.0), I'm requesting per_page + 1 rows from Solr and using the presence of the extra row to determine whether a next page exists, then truncating before returning results. This avoids relying on results.hits which can be approximate.

For cursor pagination (v2.0), the existing nextCursorMark equality check already reliably detects the last page, so I'm leaving that path unchanged — the N+1 trick doesn't work cleanly with Solr cursors since the cursor mark advances past all returned rows including the probe row.

PR incoming.