Remove HTTPS check for API Keys & Service Accounts

[tvernum]

This commit removes the checks that prevented the use of API Keys and
Service Account (Service Tokens) on nodes without HTTPS
(xpack.security.http.ssl.enabled)

As a consequence of removing this check, the API Key service is now
automatically enabled, but can be explicitly disabled with

xpack.security.authc.api_key.enabled: false

[tvernum]

@elastic/es-security I've marked this as a draft while we discuss exactly what behaviour we want here.

Questions:

• Do we want to include this in 7.16? (I've labelled as if we do, but it's open for discussion)
• Do we want to automatically turn on the API Key Service even if HTTPS is disabled? (I've assumed we do, but opinions welcome).

[bytebilly]

Do we want to include this in 7.16? (I've labelled as if we do, but it's open for discussion)

I'm in favor of doing that. Users that have to stay on 7.x will be able to fully leverage Kibana Alerting and other similar features even if their environment requires HTTPS off, for example service mesh setups.

Do we want to automatically turn on the API Key Service even if HTTPS is disabled? (I've assumed we do, but opinions welcome).

In 8.x I'm in favor of turning on by default, since security is also on by default.
I'm not against turning on by default in 7.x, but we could keep it off if we want to make it "less breaking".

[jkakavas]

Do we want to include this in 7.16?

I agree we should and I think that previous team discussions also came to the same conclusion

Do we want to automatically turn on the API Key Service even if HTTPS is disabled?

I think that for 8.x, since we are deciding to decouple API Key Service from HTTPS configuration, it wouldn't make sense to turn off API Key service if HTTPS is disabled. In 7.x though, this has the potential of being a "breaking change" for some of our users. If an administrator willingly wants API Key service to be disabled and now depends on it being disabled because TLS for HTTP is not enabled, will have API Key service enabled (unknowingly) on upgrade to 7.16. I can't think of a use case where this becomes breaking in itself, but consider the case where we ( or an application that uses elasticsearch ) has a bug where API keys can be created and leaked and can be used for data exfiltration.
It feels also consistent with what we did with enabling the file realm by default. We could have done it in 7.x too but this would be "breaking" in a similar way, and we did it only in 8.0

[jkakavas]

We also need to update

docs/reference/settings/security-settings.asciidoc#L174
x-pack/docs/en/security/authentication/service-accounts.asciidoc#L61

[ywangd]

I second on @jkakavas's comment. Also a few more docs to update:

• x-pack/docs/en/rest-api/security/create-api-keys.asciidoc#L33
• x-pack/docs/en/security/securing-communications/tls-http.asciidoc#L11
• x-pack/docs/en/security/securing-communications/security-basic-setup-https.asciidoc#L8

[tvernum]

If an administrator willingly wants API Key service to be disabled and now depends on it being disabled because TLS for HTTP is not enabled, will have API Key service enabled (unknowingly) on upgrade to 7.16.

My counter point would be that if we follow this line of reasoning, then the first time something like API Key Service is introduced it should be disabled by default because we didn't get administrators to opt-in to it.
Moving from disabled by default to enabled by default shouldn't be any more of a risk or breaking change than moving from does not exist to enabled by default.

I appreciate the analogy to file realm though, so I'm happy to not change the default in 7.16, but I'm cautious about setting too much of a precedent here - we shouldn't be afraid to add features in a minor.

[jkakavas]

My counter point would be that if we follow this line of reasoning, then the first time something like API Key Service is introduced it should be disabled by default because we didn't get administrators to opt-in to it.
Moving from disabled by default to enabled by default shouldn't be any more of a risk or breaking change than moving from does not exist to enabled by default.

but I'm cautious about setting too much of a precedent here - we shouldn't be afraid to add features in a minor.

True, I see your point.

[jkakavas]

LGTM

[ywangd]

LGTM

I appreciate the analogy to file realm though, so I'm happy to not change the default in 7.16

Do you plan to change the default when backporting? If so, all relevant docs/javadocs need to be changed as well. Do we also want to give users heads up in 7.16 docs about the default will be changed again in 8.0?

[ywangd]

Just a thought crossed my mind: Should API keys be always enabled in 8.0? That is, they cannot be disabled similar to service accounts? It's often used as an alternative to avoid directly serialising user credentials. It is also long lived. So it is potentially useful to any long running tasks.

I am not suggesting to make the change for this PR. But I think it is worth discussing before finalising 8.0.

[tvernum]

Do you plan to change the default when backporting? If so, all relevant docs/javadocs need to be changed as well.

Yes, the backport will require a few tweaks. Does anyone specifically want to review that PR before it merges?

Do we also want to give users heads up in 7.16 docs about the default will be changed again in 8.0?

I'm not sure - writing docs that talk about the future is risky. Probably we can get away with "A future release will ... "

Should API keys be always enabled in 8.0?

I think we probably want to work towards them always being enabled. I don't know whether we want to do that in 8.0, or just deprecating the setting and remove it in a future release. We should discuss.

[tvernum]

@elasticmachine update branch

[bytebilly]

Should API keys be always enabled in 8.0?

I did some telemetry analysis and the number of clusters with API keys "explicitly" disabled (HTTPS on, API keys off) is irrelevant.

However, since removing the configuration setting is a breaking change, I suggest to deprecate it in 8.0 and remove later.

[t-indumathy]

Is this applicable for xpack.security.authc.token.enabled: true too ? Because when I try to use Istio and apply OIDC realm, it expects xpack.security.authc.token.enabled: true which makes xpack.security.http.ssl.enabled: true to be mandatory. Whereas the mesh layer takes care of mTLS within mesh.

[tvernum]

It does not apply to xpack.security.authc.token.enabled

The Elasticsearch token service tries (where possible) to conform to OAuth2 specs, which require https.
We might change that position in the future, but this PR did not.