[BUG]multiple index queries error

[suwei999]

When I perform multiple index queries, an error is reported

Org. opensearch. action. search. searchPhaseExecutionException: all shards failed

The detailed information is as follows,

My data volume is very small, and the cluster is a three node version 2.0. What is the reason for this problem?

[dblock]

What are your queries and setup (OS, etc.)? Is this consistently reproducible with any query or a specific one?

[suwei999]

I also found this issue during a single index query, which occasionally recurs:

[2023-08-04T11:38:16,446][WARN ][r.suppressed             ] [zzba-opensearch-beianzhizhen-1-1] path: /ip_fpxx_140000/_search, params: {typed_keys=true, max_concurrent_shard_requests=5, ignore_unavailable=false, expand_wildcards=open, allow_no_indices=true, ignore_throttled=true, index=ip_fpxx_140000, search_type=dfs_query_then_fetch, batched_reduce_size=512, ccs_minimize_roundtrips=true}
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:644) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:362) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:679) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:459) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:293) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:615) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$6.handleException(TransportService.java:750) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:318) [opensearch-security-2.0.0.0.jar:2.0.0.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1370) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1484) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1458) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:982) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:152) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:884) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:212) [opensearch-security-2.0.0.0.jar:2.0.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$2.sendRequest(OpenSearchSecurityPlugin.java:665) [opensearch-security-2.0.0.0.jar:2.0.0.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:771) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:846) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:834) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.SearchTransportService.sendExecuteDfs(SearchTransportService.java:177) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.SearchDfsQueryThenFetchAsyncAction.executePhaseOnShard(SearchDfsQueryThenFetchAsyncAction.java:118) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:280) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:314) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:293) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:615) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.TransportService$6.handleException(TransportService.java:750) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:318) [opensearch-security-2.0.0.0.jar:2.0.0.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1370) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:301) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:418) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:410) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:158) [opensearch-2.0.0.jar:2.0.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.0.0.jar:2.0.0]

[suwei999]

Hello, we did encounter this issue when using Opensearch. How long do we plan to test this issue?

[dblock]

@suwei999 AFAIK nobody is working on this, these logs don't really explain what's going on and this is a generic error - try to provide more information? dig through logs to see whether there are better errors? what's the cluster state or setup? etc. see my question above?

[msfroh]

Usually the "all shards failed" exception is accompanied by exceptions thrown by the shards themselves, which explain what issue the shards encountered.

@suwei999, are there other exceptions in the logs?