Elasticsearch service status when ignoring incompatible versions

[legrego]

The elasticsearch core service has three service status levels defined today:

critical

This happens when Kibana has connected to a cluster which has one or more nodes with incompatible versions.

degraded

This happens when Kibana has connected to a cluster which has one or more nodes with incompatible versions, and Kibana is configured with elasticsearch.ignoreIncompatibleVersions: true. This is a dev-only setting which allows Kibana to start up even if it's connected to an incompatible cluster.

available

This happens when Kibana has connected to a cluster with fully compatible nodes.

---

The Spaces plugin waits for elasticsearch to become available before attempting to create the default space:

kibana/x-pack/plugins/spaces/server/default_space/default_space_service.ts

Lines 74 to 82 in bf04235

	const isElasticsearchReady = status.elasticsearch.level === ServiceStatusLevels.available;
	const isSavedObjectsReady = status.savedObjects.level === ServiceStatusLevels.available;
	if (!isElasticsearchReady || !isSavedObjectsReady) {
	return of({
	level: ServiceStatusLevels.unavailable,
	summary: 'required core services are not ready',
	} as ServiceStatus);
	}

This works for pretty much everyone, but we run into problems when Kibana is configured with elasticsearch.ignoreIncompatibleVersions: true, since the elasticsearch service never reaches the available status when connected to an incompatible cluster.
The result is we never create the Default space, which leads to users getting the dreaded application privileges must refer to at least one resource error.

---

Given that, I have a couple of questions:

1. In general, should consumers attempt to work with services which are in a degraded state? It's hard to know why a service is degraded to know whether or not to attempt a particular action. For this specific service, and specific degradation reason, it makes sense for Spaces to attempt to create the default space, but that's relies on an implementation detail that this is the only reason the service would ever enter the degraded state.
2. If Kibana is configured to ignore incompatible versions, should we really be reporting a degraded state? Or would available be more appropriate? This is a genuine question, I'm not sure what the best approach here would be. I could make arguments for both sides.

cc @dgieselaar @elastic/kibana-security

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[joshdover]

In general, should consumers attempt to work with services which are in a degraded state? It's hard to know why a service is degraded to know whether or not to attempt a particular action. For this specific service, and specific degradation reason, it makes sense for Spaces to attempt to create the default space, but that's relies on an implementation detail that this is the only reason the service would ever enter the degraded state.

I consider how the status state behaves is part of the public API of the Core. Any behavior changes there would need to be considered a breaking change to the API, so if we were to make any changes in the future, we will need to consider how it is currently being used. Once/if Core ever has API stability guarantees, we'd need to defer changes to this status check until a major version upgrade.

If there is consensus on that premise, then I think Spaces can directly depend on how the degraded state behaves and the meta.warningNodes and meta.incompatibleNodes to determine whether or not it should perform some logic.

If Kibana is configured to ignore incompatible versions, should we really be reporting a degraded state? Or would available be more appropriate? This is a genuine question, I'm not sure what the best approach here would be. I could make arguments for both sides.

This is a good question, and I think it makes sense to not be in a degraded state if there are warning or incompatible nodes. This option is basically intended to be used as an escape hatch and I think it should be reflected in the status of the ES service. I don't think this introduces any significant risks since this option is only available in development

[bhavyarm]

I ran into this in a different scenario while testing Run Kibana with next ES (version +1) cc @LeeDr

Initial configuration:

1. Kibana 7.9.0 BC9 and ES 7.9.0 BC9 with two spaces defined and security with trial license enabled. Has data.

Configuration where I ran into trouble:

2. Kibana 7.9.0 BC9 from step one with a fresh snapshot of ES 7.10.0
   Kibana got stuck at this screen and with this message in console

Failed to load resource: the server responded with a status of 404 (Not Found)
:5601/33813/bundles/core/core.entry.js:84 Detected an unhandled Promise rejection.
Error: Not Found
:5601/33813/bundles/core/core.entry.js:34 Uncaught (in promise) Error: Not Found
    at Fetch._callee3$ (:5601/33813/bundles/core/core.entry.js:34)
    at l (:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368)
    at Generator._invoke (:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368)
    at Generator.forEach.e.<computed> [as next] (:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368)
    at fetch_asyncGeneratorStep (:5601/33813/bundles/core/core.entry.js:34)
    at _next (:5601/33813/bundles/core/core.entry.js:34)

3. Zoomed with @legrego and copied data & config directory from ES 7.9.0 BC9 into ES 7.10.0 snapshot and disabled security (rolling upgrade step) - Now Kibana is operational with ES (without security)