NullPointerException in bulk operation leading to IndexMissingException

[Volune]

ElasticSearch 1.7.0, 5 nodes, each running in a docker container.

Got the following exception during a bulk index operation

[2015-07-30 13:29:05,558][WARN ][action.bulk              ] [duvel-elasticsearch_3] unexpected error during the primary phase for action [indices:data/write/bulk[s]]
java.lang.NullPointerException
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shards(TransportShardBulkAction.java:128)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.doRun(TransportShardReplicationOperationAction.java:354)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:112)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:74)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$OperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:207)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$OperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:189)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:222)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Any following bulk operation returns a IndexMissingException error

In case that help, here is an extract of the returned message of the first error:

{
    "index": {
        "_index": "ads",
        "_type": "ads",
        "_id": "bdi-190-1901",
        "status": 500,
        "error": "RemoteTransportException[[duvel-elasticsearch_3][inet[/172.17.4.62:9302]][indices:data/write/bulk[s]]]; nested: NullPointerException; "
    },
    "bulkCommand": {
        "operation": {
            "index": {
                "_index": "ads",
                "_type": "ads",
                "_id": "bdi-190-1901"
            }
        },
        "data": {
            "some data": "..."
        }
    }
}

This may be a duplicate of #11671

[clintongormley]

Hi @Volune

Was the index deleted? The problem in #11671 was caused by the indices being deleted, so if yours was too, then it is a duplicate

[Volune]

There was no log indicating that the index was deleted on any node. Also it seemed visible in the "ElasticSearch Head" interface, but empty. Auto-creation of indexes is disabled on the server. Still with the IndexMissingException I'm not sure what to trust.

[clintongormley]

Can you upload your logs from master from before the scroll started?

[Volune]

Sadly I just discovered that I don't have these logs anymore.

[clintongormley]

thanks @Volune

I think that for now we can assume that something similar happened, and close this in favour of #11671

[Volune]

Fine with me, thanks.