Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bulk rejections of monitoring data due to full queues cause large ERROR/WARN traces in the logs #42584

Closed
ismael-hasan opened this issue May 27, 2019 · 3 comments
Closed

Bulk rejections of monitoring data due to full queues cause large ERROR/WARN traces in the logs #42584

ismael-hasan opened this issue May 27, 2019 · 3 comments
Labels
:Data Management/Monitoring Team:Data Management Meta label for data/management team

Comments

@ismael-hasan
Copy link
Contributor

ismael-hasan commented May 27, 2019 •
edited
Loading

Elasticsearch version (bin/elasticsearch --version): 6.7.0, 7.0.0

Plugins installed: []

JVM version (java -version): "10.0.2" 2018-07-17

OS version Windows 10

Description of the problem including expected versus actual behavior:
When there are rejections for REST bulk requests due to full queue the Elasticsearch logs are not verbose; when the rejection is caused by a monitoring bulk request hitting a full queue it generates an ERROR or WARN and a large stack trace.

Checking the output logs in a Logstash instance created to reproduce this behavior, when Elasticsearch rejects monitoring documents from Logstash then Elasticsearch 7.0 returns a 500 error - not 429.
EDIT: I am referring to Logstash, but Kibana and Beats monitoring also receive a 500 error on full bulk queues

The expected behavior is that rejections in monitoring documents due to full queues follow the same process as rejections for normal bulk requests - or at least, that the logs are not that verbose on the error.

Steps to reproduce:

  1. Start a single node Elasticsearch cluster adding the following configuration to easily hit queue limits and with monitoring enabled:
thread_pool.write.size: 1
thread_pool.write.queue_size: 1
  1. Start ingesting documents, so we fill the queue constantly.

The result is that bulk rejections to monitoring data generate log stack traces in the normal ES logs, making difficult to troubleshoot them.
I am also adding the output error in a Logstash instance sending monitoring data to this Elasticsearch instance, in which it seems Elasticsearch is returning a 500 error on a full queue rejection for monitoring

Provide logs (if relevant):

Logs from ES 6.7 on rejected monitoring bulk, related to alerting monitoring, returning ERROR 
[2019-05-27T11:19:35,099][ERROR][o.e.x.w.a.i.IndexActionFactory] [wR18dbv] failed to execute action [Rio6vxW-TmO_mbA5y_HsZg_xpack_license_expiration/add_to_alerts_index]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of processing of [14639][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-alerts-6][0]] containing [index {[.monitoring-alerts-6][doc][Rio6vxW-TmO_mbA5y_HsZg_xpack_license_expiration], source[{
  "metadata" : {
    "severity" : 0,
    "cluster_uuid" : "Rio6vxW-TmO_mbA5y_HsZg",
    "version_created" : 6040099,
    "watch" : "xpack_license_expiration",
    "link" : "license",
    "time" : "2019-07-05T23:59:59.999Z",
    "alert_index" : ".monitoring-alerts-6",
    "type" : "monitoring",
    "expires_days" : [
      60,
      30,
      14,
      7
    ]
  },
  "update_timestamp" : "2019-05-27T09:19:35.002Z",
  "prefix" : "This cluster's license is going to expire in {{#relativeTime}}metadata.time{{/relativeTime}} at {{#absoluteTime}}metadata.time{{/absoluteTime}}.",
  "message" : "Update your license.",
  "timestamp" : "2019-05-09T08:14:37.283Z"
}]}], target allocation id: Z2Td1BK8SG2JJwTuUdrUVw, primary term: 17 on EsThreadPoolExecutor[name = wR18dbv/write, queue capacity = 1, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@42a41269[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 634]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:98) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:682) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:125) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:626) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:133) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:541) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:529) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:872) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performLocalAction(TransportReplicationAction.java:823) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:810) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:171) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:99) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:84) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$handleIndexActionAuthorizationResult$13(AuthorizationService.java:318) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$19(AuthorizationService.java:511) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:67) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$21(AuthorizationService.java:522) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:488) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$20(AuthorizationService.java:520) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$22(AuthorizationService.java:519) ~[?:?]
	at java.util.HashMap.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$23(AuthorizationService.java:516) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:644) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$24(AuthorizationService.java:454) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:644) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeBulkItems(AuthorizationService.java:453) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:315) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:488) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:105) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:440) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:553) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:256) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:84) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:223) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:105) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:84) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:223) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:169) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:152) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:105) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:391) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:412) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.lambda$execute$0(ExecutableIndexAction.java:100) ~[?:?]
	at org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:129) ~[x-pack-core-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.execute(ExecutableIndexAction.java:99) ~[?:?]
	at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) ~[x-pack-core-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:456) ~[x-pack-watcher-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) ~[x-pack-watcher-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:396) ~[x-pack-watcher-6.7.0.jar:6.7.0]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:545) [x-pack-watcher-6.7.0.jar:6.7.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_121]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_121]
Logs from ES 7.0, on a `.monitoring-es...` index, returning a WARN due to full bulk 
[2019-05-22T14:00:07,884][WARN ][o.e.x.m.e.l.LocalExporter] [DESKTOP-I0VERPV] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[DESKTOP-I0VERPV][127.0.0.1:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of processing of [25289][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-es-7-2019.05.22][0]] containing [43] requests, target allocation id: vfqweVGHTOa8la6C39uCug, primary term: 1 on EsThreadPoolExecutor[name = DESKTOP-I0VERPV/write, queue capacity = 1, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2a6e5e2b[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 3006]]];
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:1.8.0_121]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_121]
	at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:659) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:464) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:459) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:74) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:928) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:886) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1230) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1204) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:698) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:741) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:104) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:683) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:125) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:627) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:542) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:530) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:855) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performLocalAction(TransportReplicationAction.java:806) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:793) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:170) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:342) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$handleIndexActionAuthorizationResult$13(AuthorizationService.java:315) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$19(AuthorizationService.java:511) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:67) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$21(AuthorizationService.java:522) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$20(AuthorizationService.java:520) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$22(AuthorizationService.java:519) ~[?:?]
	at java.util.HashMap.forEach(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$23(AuthorizationService.java:516) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:644) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeBulkItems$24(AuthorizationService.java:451) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:644) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeBulkItems(AuthorizationService.java:450) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:312) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:260) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:248) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:244) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:247) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:258) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:224) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:190) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:193) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:192) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:171) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:157) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:434) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:547) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:253) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:601) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.ingest.IngestService$4.doRun(IngestService.java:419) [elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_121]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_121]
Caused by: org.elasticsearch.transport.RemoteTransportException: [DESKTOP-I0VERPV][127.0.0.1:9300][indices:data/write/bulk[s][p]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of processing of [25289][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-es-7-2019.05.22][0]] containing [43] requests, target allocation id: vfqweVGHTOa8la6C39uCug, primary term: 1 on EsThreadPoolExecutor[name = DESKTOP-I0VERPV/write, queue capacity = 1, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2a6e5e2b[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 3006]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:1.8.0_121]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:98) ~[elasticsearch-7.0.0.jar:7.0.0]
	... 169 more
Logs from Logstash rejected execution for a document not coming from monitoring. It is shown as INFO and with a 429 error code 
[2019-05-27T12:13:13,817][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of processing of [3404][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[test123][0]] containing [index {[test123][_doc][SNfH-GoBEqCkYNoyoRnV], source[{\"asd\":\"asd\",\"@version\":\"1\",\"host\":\"DESKTOP-I0VERPV\",\"@timestamp\":\"2019-05-27T10:13:13.707Z\"}]}], target allocation id: uEF_2j7JRKaBRvguoNYOMA, primary term: 5 on EsThreadPoolExecutor[name = DESKTOP-I0VERPV/write, queue capacity = 1, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@139830e8[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 625]]"})
Logs from Logstash writing monitoring data to ES 7.0, in which ES responds to a bulk rejection of monitoring data with a 500 + ERROR message 
[2019-05-20T14:25:03,446][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://localhost:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", :body=>"{\"took\":43,\"ignored\":false,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"Exception when closing export bulk\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulk [default_local]\",\"caused_by\":{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of org.elasticsearch.ingest.IngestService$4@bbeb8a8 on EsThreadPoolExecutor[name = DESKTOP-I0VERPV/write, queue capacity = 1, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@67405096[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 1415]]\"}}}}}"}
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@original-brownbear
Copy link
Member

I would add that this kind of super noisy logging also happens when monitoring has trouble because of an unavailable primary shard (I've recently had a case from 7.1.1 here where we were logging 100MB/s because of that). I think we should see if we can optimize the logging of failures for the bulk exporter.

@rjernst rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020
@dakrone
Copy link
Member

dakrone commented May 8, 2024

Internal monitoring has been deprecated for quite a while, and we're no longer doing any active development on it. I'm going to close this issue.

@dakrone dakrone closed this as not planned Won't fix, can't repro, duplicate, stale May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Data Management/Monitoring Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dakrone @rjernst @original-brownbear @ismael-hasan @elasticmachine