x-pack security plugin concurrency issue

[mushao999]

Elasticsearch version (bin/elasticsearch --version):
6.3.3, 6.8.0, 7.7.0, 7.10.0, 7.16.1 , 8.1.0(build from latest master branch)

Plugins installed: []
with x-apack security enabled, no more plugins

JVM version (java -version):
follow the mini runtime java version of each version

OS version (uname -a if on a Unix-like system):
Linux node2 3.10.0-327.ali2019.alios7.x86_64 #1 SMP Sun Jan 19 18:21:42 CST 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Cluster has an index named index1 and an alias named alias1 which points to index1. We may have some chance to get a IndexNotFoundException of an irrelvant index2 when we call GetAlias API for alias1 if we are creating index2 at the same time.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. build a three nodes cluster(must be multi-nodes cluster)
2. enable x-pack security
3. do not visit cluster from master node(make sure at least two nodes involved)
4. assume cluster endpoint is "http://endpoint:9200", run the follow two script concurrently.
   script 1(create index1, alias1. Then clean and create index2 looply. You can choose not to clean old index, thing will not changed , we just want to keep cluster clean)

 #!/bin/bash
  #create index1
  curl -XPUT -u"elastic:xxx" http://endpoint:9200/index1?pretty
  #create alias
  curl -XPOST -H'Content-type:application/json' -u"elastic:xxx" http://endpoint:9200/_aliases -d'
  {
    "actions": [
      {
        "add": {
          "index": "index1",
          "alias": "alias1"
        }
      }
    ]
  }'
  while true
  do
    echo "==================================================="
    # clean index2 if exists
    curl -u"elastic:xxx" -XDELETE http://endpoint:9200/index2 >/dev/null 2>&1
    # create index 2
    curl -XPUT -u"elastic:xxx" -H'Content-type:application/json' "http://endpoint:9200/index2?pretty"
  done

script2(get alias1 continuously)

#!/bin/bash
  #create index1
  while true
  do
    # cat alias1 while creating index2
    curl -XGET -u"elastic:xxx" http://endpoint:9200/_alias/alias1?pretty
  done

Provide logs (if relevant):
the stack trace of 8.1.0 list as follow:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [index2]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "index2",
        "index" : "index2",
        "stack_trace" : "[index2] org.elasticsearch.index.IndexNotFoundException: no such index [index2]\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1289)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1227)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1183)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:302)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:268)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNamesWithSystemIndexAccess(IndexNameExpressionResolver.java:107)\n\tat org.elasticsearch.action.admin.indices.alias.get.TransportGetAliasesAction.checkBlock(TransportGetAliasesAction.java:77)\n\tat org.elasticsearch.action.admin.indices.alias.get.TransportGetAliasesAction.checkBlock(TransportGetAliasesAction.java:43)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.checkBlockIfStateRecovered(TransportMasterNodeAction.java:132)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:179)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:152)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:52)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:79)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:77)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:169)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:576)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:96)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:123)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:555)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:449)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:972)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:936)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:326)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:442)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:370)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:255)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:129)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:179)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$3(CompositeRolesStore.java:200)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:241)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:198)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:169)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:126)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:165)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:161)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:120)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:77)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:54)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:71)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:67)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:271)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:318)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:315)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:576)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:96)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:123)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:555)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:449)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:972)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:936)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:366)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1020)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:358)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:442)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:370)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:255)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:129)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:179)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$3(CompositeRolesStore.java:200)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:241)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:198)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:169)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:126)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:344)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:276)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:762)\n\tat org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)\n\tat org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)\n\tat org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.lang.Thread.run(Thread.java:833)\n"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [index2]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : "index2",
    "index" : "index2",
    "stack_trace" : "[index2] org.elasticsearch.index.IndexNotFoundException: no such index [index2]\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1289)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1227)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1183)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:302)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:268)\n\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNamesWithSystemIndexAccess(IndexNameExpressionResolver.java:107)\n\tat org.elasticsearch.action.admin.indices.alias.get.TransportGetAliasesAction.checkBlock(TransportGetAliasesAction.java:77)\n\tat org.elasticsearch.action.admin.indices.alias.get.TransportGetAliasesAction.checkBlock(TransportGetAliasesAction.java:43)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.checkBlockIfStateRecovered(TransportMasterNodeAction.java:132)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:179)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:152)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:52)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:79)\n\tat org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:77)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:169)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:576)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:96)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:123)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:555)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:449)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:972)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:936)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:326)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:442)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:370)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:255)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:129)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:179)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$3(CompositeRolesStore.java:200)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:241)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:198)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:169)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:126)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:165)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:161)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:120)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:77)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:54)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:71)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:67)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:271)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:318)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$2.onResponse(SecurityServerTransportInterceptor.java:315)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:576)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:96)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:85)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:123)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:570)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:555)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$11(AuthorizationService.java:449)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:972)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:936)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:366)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1020)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:358)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:442)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:370)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:255)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:129)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:179)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$3(CompositeRolesStore.java:200)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:241)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:198)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:169)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:126)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136)\n\tat org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101)\n\tat org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:344)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:276)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:762)\n\tat org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)\n\tat org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)\n\tat org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.lang.Thread.run(Thread.java:833)\n"
  },
  "status" : 404
}

Most likey cause:
we have analyzed and debuged the code and get the following conclusion:

• GetAlias API execution can be described as this:
  • (Coordinate node) RestGetAliasAction -> SecurityActionFilter -> TransportGetAliasAction
  • (Master node)SecurityActionFilter -> TransportGetAliasAction
• Exception was thrown at the TransportGetAliasAction of master node, where they get cluster state to validate index's existence.(java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java:1227)
• our request did not specify any index explicitly, but the indices param in the request will be rewritten to a list of all indices from the ClusterState in the SecurityActionFilter of coordinate node.(org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java:257)
• So there will be a concurrency issue because we rewrite indices param based on the ClusterState in coordinate node but validate their existence based on ClusterState in master node
• Our case goes a deep further, which seems like we get a new create index in corrdinate node but can't see it in the master node. We think this can be happen because the two phase cluster state plublication can make corrdinate nodes' cluster state newer than the master node.

Need help
Please help reivew our issue and analysis, and give us some suggestion how to fix it.

[mushao999]

This issue by the way, may find the answer of a question marked as todo in the PR #25728. Wildcard resolver throw exceptions regarding non wildcarded expressions because SecurityActionFilter rewrite the wildcard to the concreteIndices. @javanna @martijnvg

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[droberts195]

I think this is the same problem that was investigated from #45652 (comment) onwards in that issue.

#47159 was opened as a result of that, but never worked on. It's great that this issue provides and up-to-date way to recreate the problem, as the old way I found to reproduce it is over 2 years old now and might not work any more.

[mushao999]

So is there any plan to fix it ? I would like to do some job ,if needed. @droberts195

[martijnvg]

Thanks @mushao999 for sharing this bug and the reproduction.

After reading and understanding this reproduction, the fact that the wildcard expression is expanded/resolved on the coordinating node (the node that first accepts a request), but actual usage of the resolved indices from the wildcard expression happens on the elected master node can cause error like this one. Essentially the request contains stale information, in this case resolved indices (which were valid on the coordinating node at time the request was accepted), and one of these indices no longer exists when request is being handled on the elected master node (due to the addition and removal of index2 in the background).

Today due to historical reasons, index expression resolution happens in two places. In the security (IndicesAndAliasesResolver via the SecurityActionFilter) and in IndexNameExpressionResolver. IMO this is the cause this bug that you've reported exists. If index expression resolution happened in a single place and on the node where the action performed then I don't think this bug would occur. Also I think this bug isn't tied to get alias api, but also other APIs are prone to it.

I don't know whether there is a quick fix for this bug. Maybe the get alias api (and other indices based APIs) could instead of always throwing a IndexNotFoundException if an index is missing, ignore that fact that an index is missing if an index originated from a * expression. Not sure whether such a solution is desirable.