elasticsearch.log fillup disk

[bc1000]

Elasticsearch version (bin/elasticsearch --version): elasticsearch-5.6.0-1.noarch

Plugins installed: []

JVM version (java -version): "1.8.0_144"

OS version (uname -a if on a Unix-like system): CentOS Linux release 7.4.1708 (Core)

Description of the problem including expected versus actual behavior:
elasticsearch.log file was rapid growing and filling up disk.

Steps to reproduce:

I am not sure, I guess it started this night, I do not know the trigger. elasticsearch is used in combination with zammad-ticket-system (2.0.1).

Provide logs (if relevant):

tail -200 elasticsearch.log at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] Caused by: org.elasticsearch.transport.RemoteTransportException: [JWOnIkN][127.0.0.1:9300][internal:gateway/local/started_shards[n]] Caused by: org.elasticsearch.ElasticsearchException: failed to load started shards at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:171) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.0.jar:5.6.0] ... 3 more Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-09-15T06:10:50Z, (lock=NativeFSLock(path=/var/lib/elasticsearch/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2017-09-15T06:10:50.517216Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:941) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.env.NodeEnvironment.availableShardPaths(NodeEnvironment.java:804) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:120) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.0.jar:5.6.0] ... 3 more [2017-09-16T11:46:33,071][WARN ][r.suppressed ] path: /zammad_production/Ticket/418, params: {index=zammad_production, id=418, type=Ticket} org.elasticsearch.node.NodeClosedException: node closed {JWOnIkN}{JWOnIkNAROS7szXTDWTKcQ}{g5yA6pzBRVy1_VWeNERe0g}{127.0.0.1}{127.0.0.1:9300} at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,071][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[?:?] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) ~[?:?] at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113) ~[?:?] at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[?:?] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1027) ~[?:?] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:301) ~[?:?] at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:147) ~[?:?] at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:447) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:118) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:389) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:384) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:857) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,075][WARN ][r.suppressed ] path: /zammad_production/Ticket/418, params: {index=zammad_production, id=418, type=Ticket} org.elasticsearch.node.NodeClosedException: node closed {JWOnIkN}{JWOnIkNAROS7szXTDWTKcQ}{g5yA6pzBRVy1_VWeNERe0g}{127.0.0.1}{127.0.0.1:9300} at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,075][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[?:?] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) ~[?:?] at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113) ~[?:?] at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[?:?] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1027) ~[?:?] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:301) ~[?:?] at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:147) ~[?:?] at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:447) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:118) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:389) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:384) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:857) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,078][WARN ][r.suppressed ] path: /zammad_production/User/62, params: {index=zammad_production, id=62, type=User} org.elasticsearch.node.NodeClosedException: node closed {JWOnIkN}{JWOnIkNAROS7szXTDWTKcQ}{g5yA6pzBRVy1_VWeNERe0g}{127.0.0.1}{127.0.0.1:9300} at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,078][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[?:?] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) ~[?:?] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) ~[?:?] at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113) ~[?:?] at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[?:?] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1027) ~[?:?] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:301) ~[?:?] at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:147) ~[?:?] at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:447) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:118) ~[elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:389) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:384) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:94) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:857) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:840) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:304) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:224) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:230) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.stop(Node.java:787) [elasticsearch-5.6.0.jar:5.6.0] at org.elasticsearch.node.Node.close(Node.java:809) [elasticsearch-5.6.0.jar:5.6.0] at org.apache.lucene.util.IOUtils.close(IOUtils.java:89) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.apache.lucene.util.IOUtils.close(IOUtils.java:76) [lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:209) [elasticsearch-5.6.0.jar:5.6.0] [2017-09-16T11:46:33,313][INFO ][o.e.n.Node ] [JWOnIkN] stopped [2017-09-16T11:46:33,313][INFO ][o.e.n.Node ] [JWOnIkN] closing ... [2017-09-16T11:46:33,430][INFO ][o.e.n.Node ] [JWOnIkN] closed [2017-09-16T11:46:37,489][INFO ][o.e.n.Node ] [] initializing ... [2017-09-16T11:46:37,644][INFO ][o.e.e.NodeEnvironment ] [JWOnIkN] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.1gb], net total_space [46.9gb], spins? [unknown], types [rootfs] [2017-09-16T11:46:37,644][INFO ][o.e.e.NodeEnvironment ] [JWOnIkN] heap size [1.9gb], compressed ordinary object pointers [true] [2017-09-16T11:46:37,656][INFO ][o.e.n.Node ] node name [JWOnIkN] derived from node ID [JWOnIkNAROS7szXTDWTKcQ]; set [node.name] to override [2017-09-16T11:46:37,656][INFO ][o.e.n.Node ] version[5.6.0], pid[31155], build[781a835/2017-09-07T03:09:58.087Z], OS[Linux/3.10.0-514.26.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_144/25.144-b01] [2017-09-16T11:46:37,656][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch] [2017-09-16T11:46:40,557][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [aggs-matrix-stats] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [ingest-common] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [lang-expression] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [lang-groovy] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [lang-mustache] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [lang-painless] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [parent-join] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [percolator] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [reindex] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [transport-netty3] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded module [transport-netty4] [2017-09-16T11:46:40,558][INFO ][o.e.p.PluginsService ] [JWOnIkN] loaded plugin [mapper-attachments] [2017-09-16T11:46:40,859][WARN ][d.m.attachment ] [mapper-attachments] plugin has been deprecated and will be replaced by [ingest-attachment] plugin. [2017-09-16T11:46:42,803][INFO ][o.e.d.DiscoveryModule ] [JWOnIkN] using discovery type [zen] [2017-09-16T11:46:43,636][INFO ][o.e.n.Node ] initialized [2017-09-16T11:46:43,636][INFO ][o.e.n.Node ] [JWOnIkN] starting ... [2017-09-16T11:46:43,957][INFO ][o.e.t.TransportService ] [JWOnIkN] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2017-09-16T11:46:47,090][INFO ][o.e.c.s.ClusterService ] [JWOnIkN] new_master {JWOnIkN}{JWOnIkNAROS7szXTDWTKcQ}{yTToRRSZRM2uHk5ABx4p9g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2017-09-16T11:46:47,203][INFO ][o.e.h.n.Netty4HttpServerTransport] [JWOnIkN] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2017-09-16T11:46:47,204][INFO ][o.e.n.Node ] [JWOnIkN] started [2017-09-16T11:46:47,466][INFO ][o.e.g.GatewayService ] [JWOnIkN] recovered [1] indices into cluster_state [2017-09-16T11:46:48,636][INFO ][o.e.c.r.a.AllocationService] [JWOnIkN] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[zammad_production][2]] ...]).

[jasontedor]

Closed by #25660

[jasontedor]

Starting in 6.0.0, Elasticsearch will ship with a default configuration that limits the on-disk size of logs. In 5.6, you can apply this configuration yourself.