Upgrading from 6.3.2 to 6.5.1 and Pagerduty Keystore Settings Issues

[danielkasen]

When trying to upgrade with a populated keystore in 6.3 the node starts up but spams the logs with the following:

java.lang.IllegalStateException: setting [xpack.notification.pagerduty.account.watcher-default.secure_service_api_key] is not dynamic
	at org.elasticsearch.common.settings.Setting.newUpdater(Setting.java:551) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.lambda$getValue$2(Setting.java:681) ~[elasticsearch-6.5.1.jar:6.5.1]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_171]
	at java.util.stream.DistinctOps$1$2.accept(DistinctOps.java:175) ~[?:1.8.0_171]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_171]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_171]
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171]
	at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) ~[?:1.8.0_171]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_171]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_171]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_171]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_171]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_171]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:677) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.settings.Setting$AffixSetting$1.getValue(Setting.java:666) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.updater(AbstractScopedSettings.java:554) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:177) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.1.jar:6.5.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

You can get around this by clearing out the Keystore but then I lose my keystore values.

Also, none of this is defined in the _cluster/settings so I'm not sure why it's trying to make it dynamic.

[danielkasen]

This was on the first node of an attempted upgrade so not sure if this happens when you have different ES versions, but it's hard to test without fully committing on the large cluster I have.

[elasticmachine]

Pinging @elastic/es-core-features

[danielkasen]

I was able to continue if I comment this section out on the master nodes. Of course this breaks the integration of now though.

xpack:
  notification:
#    pagerduty:
#      default_account: 'watcher-default'
#      account:
#        'watcher-default':
#          event_defaults:
#            description: "Watch Notification"
#            client: " Watcher Notification"

[danielkasen]

Looks like this is sometimes broken and sometimes work:

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[i-xxxxx][10.0.0.5:9300][cluster:admin/settings/update]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "setting [xpack.notification.pagerduty.account.watcher-default.secure_service_api_key] is not dynamic"
  },
  "status": 500
}