SOLR-17053: If all shards fail, fail the request despite shards.tolerant

[aparnasuresh85]

https://issues.apache.org/jira/browse/SOLR-17053

Description

If the request is tolerant to errors, logic in SearchHandler simply sets partialResults to true upon detecting an error in ShardResponse. This means a failure to query any shard would be reported back as partialResults=true, instead of a query failure.

Why: Improve the error handling and reporting in the search system by making sure that query failures are distinct from partial results

Solution

Evaluate if all shards failed while fetching top results. If yes, fail the request by throwing a SolrException, if not set partialResults=true

Tests

Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.

Added a test in TestTolerantSearch: testAllShardsFail() that performs distributed search across 2 shards, but both shards have failures. The test performs the distributed search first by setting shards.tolerant=false and then by toggling it to true. The test confirms that in both cases, a SolrException is thrown. Also ensured existing tests in TestTolerant pass with the new code changes.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

@dsmiley

[dsmiley]

Wonderful! Can you please add a CHANGES.txt entry under 9.5 in the Improvements section?

Suggested text:

Distributed search with shards.tolerant: if all shards fail, fail the request"

[aparnasuresh85]

Wonderful! Can you please add a CHANGES.txt entry under 9.5 in the Improvements section?

Suggested text:

Distributed search with shards.tolerant: if all shards fail, fail the request"

Done.