Stack Monitoring certificate validation on beats

[mens1205]

Bug Report

What did you do?
I setup Stack monitoring as described here: Stack Monitoring

Config looks like this:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic
spec:
  version: 8.3.2
[...]
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: elasticsearch
    logs:
      elasticsearchRefs:
      - name: elasticsearch
[...]
  http:
    tls:
      certificate:
        secretName: elasticsearch-certificate

What did you expect to see?

I expected that metricbeat and filebeat are able to connect to elasticsearch.

What did you see instead? Under which circumstances?

I received the following error message:

{"log.level":"error","@timestamp":"2022-07-28T13:26:36.239Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch-es-http.elastic.svc:9200)): Get \"https://elasticsearch-es-http.elastic.svc:9200\": x509: certificate is valid for REDACTED, REDACTED, not elasticsearch-es-http.elastic.svc","service.name":"metricbeat","ecs.version":"1.6.0"}

Due to the fact, that I'm providing my own certificate to elasticsearch which didn't include a SAN entry for elasticsearch-es-http.elastic.svc.

I think this is a quite common usecase. Is it possible to disable the certificate validation or set it to ssl.verification_mode: certificate for the beats?

Environment

• ECK version:

  2.3.0

• ES version:

  8.3.2

[pebrc]

This is unfortunately a limitation of the current implementation. The problem is the output configuration that Metricbeat and Filebeat use to know where to send the data to. It is generated based on the elasticsearchRefs and cannot be influenced by the user at the moment. So there is no way for you loosen the TLS verification mode.

I am not sure what the best way is to fix it. One way I could think of is to say whenever someone is using custom certificates for their Elasticsearch cluster that is the target of an stack monitoring configuration to switch to verification mode: certificate (we do it by default for Kibana already as I found out today)

@mens1205 until a fix is available I can think of a few workarounds that might be all somewhat unattractive:

1. change your architecture so that you do not configure Elasticsearch directly to terminate TLS for your REDACTED domain but use an Ingress instead and let the connection from Ingress to Elasticsearch be secured by internal certificates valid for the Kubernetes service domain e.g elasticsearch-es-http.elastic.svc or just use the self-signed certificates ECK issues by default.
2. treat this as an "external" monitoring setup https://www.elastic.co/guide/en/cloud-on-k8s/2.3/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.html which gives you more control over the URL the Beats will try to connect to. That would allow you let the monitoring go to REDACTED domain. The downside might be additional data transfer costs depending on your setup (e.g. if running on one of the big Cloud providers they will charge you for going through the external interface) Disclaimer: I have not tested this option myself

cc @thbkrkr and @naemono for awareness

[le0nard01]

@pebrc Doesn't it make sense to do the same verificantion mode: certificate for elasticsearch service too ? As has been done in Kibana as you mentioned.

[thbkrkr]

One way I could think of is to say whenever someone is using custom certificates for their Elasticsearch cluster that is the target of an stack monitoring configuration to switch to verification mode: certificate (we do it by default for Kibana already as I found out today)

This seems like a good compromise to me to solve this without asking anything from the users.

Another more involved idea I had is to propose a way to override the Beat config like it is possible for the Pod definition via the podTemplate. This is more powerful but has the downside of requiring users to have some knowledge of the existing Beat config, which I find unfortunate.

spec:
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: elasticsearch
        configTemplate:
          ouput:
            ssl:
              verification_mode: certificate

[mens1205]

I would prefere to set the default Beats output to verification_mode: certificate as it is set for Kibana to have a consistent behavior between products. Think it could be confusing for the user if Kibana connectivity to ES is working just with specifing the ElasticsearchRef but Beats/Monitoring require additional/different configuration.

[mens1205]

@mens1205 until a fix is available I can think of a few workarounds that might be all somewhat unattractive:

1. change your architecture so that you do not configure Elasticsearch directly to terminate TLS for your `REDACTED` domain but use an Ingress instead and let the connection from Ingress to Elasticsearch be secured by internal certificates valid for the Kubernetes service domain e.g `elasticsearch-es-http.elastic.svc` or just use the self-signed certificates ECK issues by default.

2. treat this as an "external" monitoring setup https://www.elastic.co/guide/en/cloud-on-k8s/2.3/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.html which gives you more control over the URL the Beats will try to connect to. That would allow you let the monitoring go to `REDACTED` domain. The downside might be additional data transfer costs depending on your setup (e.g. if running on one of the big Cloud providers they will charge you for going through the external interface) Disclaimer: I have not tested this option myself

@pebrc Thanks for your proposals. I implemented a variation of option 2. as a workaround for testing. So I can confirm that it is working. But indeed as you already mentioned the solution is not very attractive and won't go into production in our environment. So I'm looking forward to a fix for this issue.