Elasticsearch API keys requires HTTP TLS to be switched on and conflicts with Istio

[charith-elastic]

The Elasticsearch API Keys service requires TLS to be enabled on the HTTP interface (xpack.security.http.ssl.enabled: true). Features that are built on top of the API keys service such as Kibana alerting have a hard requirement on this configuration.

This poses a problem for users who want to run Stack applications in a service mesh. As the idea is to let the mesh handle transport security and deal with things such as mTLS, the common integration pattern is to disable TLS at the application level and offload that work to the mesh. In fact, when strict mTLS is enabled by the mesh control plane, having application-level TLS can cause things to break as the traffic gets encrypted twice.

Currently, I am not aware of a way to have Elasticsearch working in a mesh with TLS enabled. Attempting to do this results in TLS verification errors and clients are not able to communicate with the server over the mesh. Therefore, users who wish to deploy the Stack in a service mesh will have to choose between mesh-enforced strict TLS vs. API keys and alerting etc.

[shubhaat]

@bytebilly mentioned that the Elasticsearch team does not want to relax the requirement en-masse, and would like for us to investigate if there is a way to detect Elasticsearch is running in a mesh.

I see having an technical answer on how to detect a service mesh as a strong requirement to progress, and the ECK team is probably the best fit for that task.

[kurbar]

This seems to be the same case when trying to configure a SAML realm which requires xpack.security.authc.token.enabled: true which also requires HTTP TLS to be enabled.

[bytebilly]

This seems to be the same case when trying to configure a SAML realm which requires xpack.security.authc.token.enabled: true which also requires HTTP TLS to be enabled.

Yes, same scenario as API keys. The base concern is to send valid credentials (tokens) over insecure connections.

[lenalebt]

Okay, so - I'm currently in the situation that I want to do exactly that: Running Elasticsearch and AppSearch/Enterprise-Search in an Istio-enabled cluster with mTLS enabled. Naturally, I deactivated elastic HTTPS. AppSearch/Enterprise-Search however only wants to connect to elasticsearch with API keys, and I cannot activate them because elasticsearch does not let me do that:

bootstrap check failure [1] of [1]: HTTPS is required in order to use the API key service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the API key service using the [xpack.security.authc.api_key.enabled] setting

So, how should I configure AppSearch or Elasticsearch to get it running? I already do have another Kibana and Elasticsearch cluster running in the same cluster and they like each other just fine...

[pebrc]

This issue will be fixed by relaxing the TLS requirement in the API key service in the upcoming Elasticsearch releases 7.16.0/8.0.0 respectively elastic/elasticsearch#76801

[maggieghamry]

@pebrc is this PR elastic/elasticsearch#76801 designed to prevent the bootstrap errors
bootstrap check failure [1] of [1]: HTTPS is required in order to use the API key service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the API key service using the [xpack.security.authc.api_key.enabled] setting
in an upgraded in k8 (7.16.0) running Elasticsearch (no APM or Kibana) on an Istio-enabled cluster with TLS disabled in 7.16.0 while using service tokens?

For example:
xpack.security.authc.api_key.enabled: false
plus

kind: Elasticsearch
...
spec:
 version: 7.16.0
  http:
    tls:
      selfSignedCertificate:
        disabled: true

^ is this configuration now supported on k8s?