Support/document dynamically reloading Kibana settings without restart

[gigerdo]

Describe the feature:
The main idea is to dynamically reload settings without having to restart the Kibana server. This functionality seems to exist already, but is only documented to reload logging settings.
There is this issue mentioning the feature: #52756

I would specifically like to have dynamic reload for xpack.actions.email.domain_allowlist (Docs).

I tested this and it seems to work: After sending SIGHUP to Kibana, the new allowlist is used. Edit: My original comment was not quite correct. The UI does indeed reload the setting, but the backend/server where the email is actually sent out still uses the old config.

But currently this functionality looks unofficial/undocumented, so it would be great to make it official and document it.

Describe a specific use case for the feature:
My use-case is updating xpack.actions.email.domain_allowlist for a number of separate deployments using Kibana. The setting can change from time to time, and it would mean restarting every Kibana instance, every time the setting changes. But by having the setting reload dynamically, the impact of changing it is reduced to a minimum.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

There are probably other settings - mainly for the actions plugin - that would be nice to have hot-reloadable. But thinking there are some that won't work that way - preconfigured connectors, perhaps?

I'm assuming the ask here is to document the alerting-related config (including task manager, if anything is hot-reloadable there) specifically, versus across all of Kibana, so marking this as a ResponseOps issue. We would probably want our own if if this is more general Kibana-wide ask, due to additional testing/doc needed for this.

[gigerdo]

Yes, for me the most important one is just xpack.actions.email.domain_allowlist. We can always look at individual other settings later if it makes sense to also mark them as dynamic. There for sure are also others that would be nice to dynamically reload to avoid restarts.

[mikecote]

Linking with #132183.

[mikecote]

We should talk to @elastic/kibana-core team to see what the best practices are for plugins when the kibana config changes. Are we supposed to support dynamic changes or only track the first config given to us via the observable.

[afharo]

Are we supposed to support dynamic changes or only track the first config given to us via the observable.

@mikecote as always, IT depends:

The config$ observable uses shareReplay(1). It means it always returns the latest & freshest value. You can access await firstValueFrom(config$) to read your configuration.

For sync operations, you may want to maintain a variable that you overwrite whenever the observable emits: Use this.actionsConfig and update it every time the config$ observable emits.

Now... why it depends?

There are some configurations that are usually linked to a way of setting up the plugin. An extreme use case could be an HTTP route is registered depending on a config parameter. You can't effectively register a new HTTP route after the setup phase is completed, so reacting to a config change in that scenario is not recommended.

Another fun fact to know: the configuration exposed to the browser (which uses a sync API only), is always the freshest (when loading the page), so, if you have configuration used on both, the public and server sides, you should be aware that public might have a different one to server if server doesn't react to changes because public always does. On the opposite side, if the user never reloads the page, it could happen that public uses an outdated version of the config.

I hope it makes sense.

[mikecote]

@afharo this makes sense, thank you for the insight! 🙏 In our case, I believe most of our code would be capable of reacting to live changes of the config$ observable.

We can use this issue to make most if not all of the alerting, actions, task manager, and event log settings react to changes.

[gigerdo]

I would specifically like to have dynamic reload for xpack.actions.email.domain_allowlist (Docs). I tested this and it seems to work: After sending SIGHUP to Kibana, the new allowlist is used.

I just retried using SIGHUP to reload the setting and realized that my original comment is not quite correct: The setting is only reloaded in the UI/browser and not in the backend (as @afharo mentioned). So creating a rule works, but then it fails when executing.

So it really would need some implementation to allow fully dynamic reload of this setting.