Improve major version upgrades

[pebrc]

#231 introduces robust support for running different versions of the Elastic stack with the k8s operator but we don't have any support yet for the subtleties of migrating from one version to another.

Ideas

• report unsupported version upgrades via status + events
• implement upgrade checks when upgrading from version n to m
  Using either the Kibana upgrade assistent APIs and or deprecation API to identify issues where possible
• Prevent skipping a version and other version related invariants (e.g. 6.7-> 7.0) via a validating admission controller

[pebrc]

A rolling upgrade is only possible from 6.7. because 7.0 as of beta1 (?) no longer supports minimum master nodes and only 6.7 publishes m_m_n in cluster state (see elastic/elasticsearch#37701). That means in practical terms that a 7.0 node cannot join a 6.x cluster unless that cluster is >= 6.7

[pebrc]

Decided to pursue only #464 and record/express the inability to upgrade via a new status (something like precondition failed) and events (could not upgrade because ...)

[pebrc]

See #312 for details of PVC reuse which also applies to major version changes.

[pebrc]

Maybe we should not do any of the upgrade guidance mechanism mentioned above.

• Users can already use the Kibana upgrade assistant exactly as they would on a bare-metal installation
• The operator has no good way of rejecting an upgrade in a user friendly way
  • events and logs are sub-optimal
  • if we re-establish an admission control we would have a fairly user-friendly way of communicating problems before upgrading. But calling the deprecation API from a webhook seems like asking for trouble. If the cluster is unavailable the user has no means of fixing their cluster if we make that validation required. If it is made optional it might still run for a very long time. If we are leveraging our observer mechanism to decouple calls to the deprecation API from webhook requests, the information returned might be stale.
• Some of the deprecation warnings are misleading because they apply to configuration managed by the operator which will be auto-upgraded during the version upgrade. Example:

 {
      "level": "critical",
      "message": "Security realm settings structure changed",
      "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#include-realm-type-in-setting",
      "details": "these settings must be updated to the new format while the node is offline during the upgrade to 7.0 (nodes impacted: [major-upgrade-es-masters-2, major-upgrade-es-data-2, major-upgrade-es-data-0, major-upgrade-es-masters-0, major-upgrade-es-masters-1, major-upgrade-es-data-1])"
    }

We automatically generate the correct realm settings structure for v7 (unless the user has overridden these settings of course). Returning this message to the user would incorrectly communicate a call to action. Not sure if they could be filtered out.

So even if we decide not to implement calls to the deprecation API from the operator, we still need to document that certain deprecation warnings shown in the upgrade assistant and the deprecation API can be safely ignored on ECK

[anyasabo]

Also related to the doc issue about upgrades #1814

[pebrc]

We discussed this offline and consensus was that we are seeing this more as nice to have rather than as a crucial bit of functionality or even a precondition for productive use of ECK.

[pebrc]

Closing this for now given the last comments here, let's reopen if needed or better yet open a more focused issue if the need arises.