Perform more startup consistency checks before writing anything to disk

[DaveCTurner]

Today when a node starts up after an upgrade it might write upgraded versions of at least these separate structures to disk:

• the keystore
• the node metadata file
• (some portion of) the cluster metadata

None of these have forwards-compatible representations, and all of them are loaded, checked, and then rewritten independently. This can potentially get a node completely stuck in an upgrade:

• if the node metadata file is invalid (e.g. comes from a version that is too old to support an in-place upgrade) then we do not discover this until after upgrading the keystore to the latest version. This version of the node cannot start up due to the invalid node metadata file, but an attempt to downgrade to the previous working version will also fail because of the upgraded keystore.

• if the cluster metadata is invalid (e.g. contains an index from an unsupported version) then we do not discover this until after upgrading the keystore and the node metadata files to the latest versions. Again, this version of the node cannot start up due to the invalid cluster metadata, but an attempt to downgrade to the previous working version will also fail because of the upgraded keystore and node metadata files.

One common path into this kind of situation is by upgrading without first getting a clean bill of health from the upgrade assistant.

We can make this experience better by performing more consistency checks before writing anything to disk at startup, to avoid blocking a subsequent downgrade in cases where the upgrade is obviously infeasible.

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

I agree this is something important we should aim for, but there are some caveats we need to consider regarding the keystore. The keystore is loaded very early in startup, before we have even installed the security manager. This is important, as our security policy only allows reading from the config directory, not writing.

We could add this write permission, but that doesn't seem worth it, and would mean any plugin could overwrite the keystore.

The important thing about security manager installation is it happens before we have loaded any plugin code. Can the checks on node metadata and cluster metadata be done independent of any services being loaded?

[DaveCTurner]

It's going to be tricky to do much with the cluster metadata before loading plugins.

Do we need to write to the keystore at all during startup? Could we instead move the responsibility for upgrading an old-format keystore to the elasticsearch-keystore command?

That said, on reflection the keystore might not matter so much here anyway because it's something that a user can create again themselves. It could be enough to emit a more actionable error message telling them to delete and re-create it if it can't be loaded.

[rjernst]

Do we need to write to the keystore at all during startup?

If the keystore does not exist, we auto create it. This is because we always need the keystore.seed value, and do not want archive users to need additional setup before running Elasticsearch. We also do format upgrade as you mention, which is again important to not require additional setup on upgrade, and would not change the difficulty in downgrading.

It could be enough to emit a more actionable error message telling them to delete and re-create it if it can't be loaded.

I agree. I opened #46291

[DaveCTurner]

We made progress towards fixing this in #50907 which delays writing the node metadata file until after validating and upgrading the cluster metadata, but unfortunately that's not enough: #42489 also moves the contents of the data path around so as to make it incompatible with 7.x, and does so before looking at the cluster metadata.