Pushing auto-follow-pattern results in a Null pointer exception on elasticsearch side

[seenkley]

Copy - paste from here https://discuss.elastic.co/t/pushing-auto-follow-pattern-results-in-a-null-pointer-exception-on-elasticsearch-side/189666/2

Elasticsearch version (bin/elasticsearch --version):
6.8.1
Plugins installed: []
none.

JVM version (java -version):
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b04, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux ip-10-176-68-180 4.14.62-65.117.amzn1.x86_64 #1 SMP Fri Aug 10 20:03:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

So, we installed elasticsearch 6.8.1 on 6 nodes and are trying to setup CCR with two clusters - 3 nodes each. License is activated and valid, remote settings are set (seed, nodes and cluster name). But now, somehow when I try to push the autofollow pattern like this

curl -X PUT "localhost:9200/_ccr/auto_follow/AutoFollowerPatterns" -H 'Content-Type: application/json' -d'
{
  "remote_cluster" : "elastic-uswest",
  "leader_index_patterns" :
  [
    "visit*"
  ],
  "follow_index_pattern" : "ccr-{{leader_index}}"
}
'

it results into that response:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "remote_transport_exception",
        "reason" : "[10.176.70.184][10.176.70.184:9300][cluster:admin/xpack/ccr/auto_follow_pattern/put]"
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : null
  },
  "status" : 500
}

Digging a bit deeper into the logs of elasticsearch, we can see this message:

[2019-07-09T10:10:49,510][WARN ][r.suppressed             ] [10.176.68.180] path: /_ccr/auto_follow/AutoFollowerPatterns, params: {name=AutoFollowerPatterns}
org.elasticsearch.transport.RemoteTransportException: [10.176.69.160][10.176.69.160:9300][cluster:admin/xpack/ccr/auto_follow_pattern/put]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.xpack.ccr.CcrLicenseChecker.hasPrivilegesToFollowIndices(CcrLicenseChecker.java:318) ~[?:?]
        at org.elasticsearch.xpack.ccr.action.TransportPutAutoFollowPatternAction.lambda$masterOperation$2(TransportPutAutoFollowPatternAction.java:88) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1104) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:985) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:977) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:952) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) [elasticsearch-6.8.1.jar:6.8.1]
...

Our initial thought was, that the security module could cause the error - because of this word in the exception: hasPrivilegesToFollowIndices. So, we removed all unnecessary modules and the error still occurs.

Installed modules:

Steps to reproduce:
Sadly, I cannot reproduce the bug. When I tried it, it always worked fine. It happened on this one particular environment and now we cannot fix it.

[elasticmachine]

Pinging @elastic/es-distributed

[jasontedor]

Do you see any "failed to read authentication" error log messages with an accompanying exception message and stack trace in the logs, around the time that this occurred?

[jasontedor]

Our initial thought was, that the security module could cause the error - because of this word in the exception: hasPrivilegesToFollowIndices. So, we removed all unnecessary modules and the error still occurs.

Also, I don't understand this course of action at all. Are you saying that you do not have security enabled (this code path suggests that you do)? Can you clarify what your license level is (basic, trial, or a paid gold/platinum license)? I strongly discourage you from modifying the installation on disk like this. You are in unsupported territory now.