Plugins may leave junk in the cluster state, requiring lenient parsing and possibly preventing their removal

[DaveCTurner]

As noted at #21471 (comment), today we assume that anything in the cluster Metadata that we don't recognise must belong to a plugin that has since been uninstalled, and just leniently ignore it. We should be stricter about this, but if we were simply to fail to start in this state then it would not be possible to remove a plugin that records anything in the cluster Metadata unless the plugin implemented a way to remove its metadata whilst still installed. Additionally there's not really a stable interface for plugins to manipulate this state today.

If we do intend to permit plugins to record information in Metadata then I think we should not use a bare Custom manipulated with the internal APIs, and should instead use a plugin-specific structure that better supports uninstallation.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)