Forgot to install a plugin can cause custom cluster state parts to be removed

[martijnvg]

If a plugin adds custom cluster state parts and the plugin isn't installed on all nodes then it could cause custom metadata to be removed by a node just by reading it.

Brought up while reviewing #21243.

[clintongormley]

On the other hand, if you uninstall a plugin, it's the only way of removing its custom metadata. Perhaps we should just recommend using the required plugins setting?

[bleskes]

my kneejerk reaction in the review was to refuse starting when you can't read a local cluster state due to the lack of a plugin and make the plugin manager to remove/archive pieces of the cluster state. This is however not trivial at all as the plugin manager need to reach into ES storage which is something we didn't do before (and may require custom knowledge in the future where current plans will make it more complicated). There is also the question if we want the plugin manager to "archive" this like we do with other config.

Perhaps we should just recommend using the required plugins setting?

yeah.. it's fairly hidden now (it took a couple of iteration to find it, knowing it exists.) Maybe we should promote it to a note in the introductions section or the management intro of the plugins.

[martijnvg]

Right, but that doesn't protected against unintended removal of custom metadata. Maybe we should go even further and require that plugins are also specified in the plugin.mandatory setting? (if a plugin isn't in the this list then we abort starting a node)

[bleskes]

We are working on addressing this as part of the node validaing in zen 2. @ywelsch @DaveCTurner do you have this captured in another place so we can close the issue?

[bleskes]

@DaveCTurner ping re ^^ ?

[DaveCTurner]

The approach we're currently investigating for Zen 2 now looks like it'll be fairly independent of this issue: it won't fix this, but nor will it significantly affect the work required to fix this.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

This is still a legitimate issue, we leniently ignore unknown fields when reading the Metadata from XContent:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java

Lines 2354 to 2360 in b4b68c8

	try {
	Custom custom = parser.namedObject(Custom.class, currentFieldName, null);
	builder.putCustom(custom.getWriteableName(), custom);
	} catch (NamedObjectNotFoundException ex) {
	logger.warn("Skipping unknown custom object with type {}", currentFieldName);
	parser.skipChildren();
	}

Since it is now possible to manually remove named customs from the on-disk metadata (via elasticsearch-node remove-customs), I think we could reasonably remove this lenience and make the node fail to start if it can't read its whole metadata.

NB to avoid having to do manual surgery on the cluster state would require uninstallable plugins to support removing their custom metadata whilst still installed. Today I think there are no known uninstallable plugins, nor is there even a stable API by which plugins can manipulate their parts of the cluster state. I'll raise this point with the plugins folks.

[DaveCTurner]

I looked into how disruptive it might be to remove this lenience and found at least one blocker, see #88983.