Kibana cannot connect to ES when using self signed certs that are not created by the operator itself

[geekflyer]

We run a cluster internal CA (that is using cert-manager) which can issue certificates for all kind of applications in the cluster incl. elasticsearch. This cluster internal CA uses a self signed root certificate.

We use this Issuer to issue certificates to elasticsearch which works just fine.
E.g:
Remark: The code snippets below are pulumi code written in TypeScript, but I believe it's understandable enough how that would look in raw yaml

 {
    apiVersion: 'certmanager.k8s.io/v1alpha1',
    kind: 'Certificate',
    metadata: {
      name: CERT_NAME,
      namespace: dataNamespace.metadata.name
    },
    spec: {
      secretName: CERT_NAME,
      issuerRef: {
        name: 'cluster-internal-ca',
        kind: 'ClusterIssuer'
      },
      dnsNames: dataNamespace.metadata.name.apply(ns => [
        `${ES_NAME}-es-http.${ns}.es.local`,
        `${ES_NAME}-es-http`,
        `${ES_NAME}-es-http.${ns}.svc`,
        `${ES_NAME}-es-http.${ns}`
      ]),
      organization: [`ElasticSearch ${context.inferredEnvironment}`]
    }
  }

    apiVersion: 'elasticsearch.k8s.elastic.co/v1alpha1',
    kind: 'Elasticsearch',
    metadata: {
      name: ES_NAME,
      namespace: dataNamespace.metadata.name
    },
    spec: {
      version: '7.2.0',
      http: {
        tls: {
          certificate: {
            secretName: CERT_NAME
          }
        }
      },

However when creating a Kibana instance with

elasticsearchRef: {
        name: ES_NAME
     }

The kibana instance fails to get into healthy mode and eventually fails continuously with:

{"type":"log","@timestamp":"2019-08-07T03:55:51Z","tags":["warning","task_manager"],"pid":1,"message":"PollError No Living connections"}
{"type":"log","@timestamp":"2019-08-07T03:55:52Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://main-es-http.es-instances.svc:9200/"}

As far as I can see it kibana does not mount anywhere the ca.crt that is contained in the elasticsearch secret (named CERT_NAME above) and could be used to establish trust.

I also tried to setup a kibana without elasticsearchRef via the undocumented elasticsearch section of Kibana:

   elasticsearch: {
        certificateAuthorities: {
          secretName: CERT_NAME
        },
        url: 'https://main-es-http:9200/',
        auth: {
          inline: { username: 'elastic', password: 'elastic-user-password' }
        }
      },

but that fails with the same error.

NOTE: The ElasticSearch instance works fine. I can connect to it from other application or via port-forward as long as I import the ca.crt from our cluster internal CA (that root cert is also contained in the secret passed to the elasticsearch instance).

Last but not least I believe the spec.elasticsearch.auth.secret is defined incorrectly in the CRD since it expects an object instead of a string / name.
Also it should probably be called secretName to be consistent with other places.

[geekflyer]

I got it kinda working after a bunch of try and error and going through the source code now.

What works was using this spec in Kibana:

  elasticsearch: {
        certificateAuthorities: {
          secretName: clusterInternalCaCertSecretName
        },
        url: `https://main-es-http:9200/`,
        auth: {
          secret: { name: 'main-es-elastic-user', key: 'elastic' }
        }
      }

The magic sauce was basically that the spec.elasticsearch.certificateAuthorities.secretName needs to point to a secret which contains a key tls.crt with the public key of elasticsearch or a key up in the chain which was used to create the elasticsearch cert (i.e. the root cert).

Unfortunately it's not possible to only set spec.elasticsearch.certificateAuthorities.secretName an use the remaining config auto-inferred from the elasticSearchRef. So when one wants to specify a custom CA one has to also set all the other parameters like url and auth which is kind of bothersome. It should be possible to set the ca cert without setting all the other stuff manually.

Btw a few more other oddities:

• When one sets both elasticsearch and elasticSearchRef, no warning or error is being thrown but the elasticsearch config is simply ignored.

Could you please make this a bit more intuitive and update the docs to include info about those settings?

[geekflyer]

another thing that's missing from the docs is:
What altNames do custom certs have to have for ES and Kibana to work? I kind of reverse-engineered this but it would be good to have it in the docs as well.

[pebrc]

I think the problem here is that we implicitly assume that the custom certificate you configure in Elasticsearch is also a CA or uses a public CA, which is of course a problematic assumption we need to reconsider.

One workaround is to include the whole certificate chain in the tls.crt but it is my understanding that cert-manager does not currently support that and that it could be considered problematic.

As far as I can see it kibana does not mount anywhere the ca.crt that is contained in the elasticsearch secret (named CERT_NAME above) and could be used to establish trust.

What happens here is that we create a $KIBANA_NAME-kb-es-ca which contains the tls.crt you configured in Elasticsearch and this secret is then mounted into the the Kibana container. Of course that does not help you because Kibana cannot verify the issuer of that certificate.

I also tried to setup a kibana without elasticsearchRef via the undocumented elasticsearch section of Kibana:

The reason this section is undocumented is that is meant for internal use by the operator only and the restrictions apply that you have to specify everything in that section and the operator might also overwrite what you specified at any time.

The supported workaround for your issue is to use standard Kibana configuration in combination with a pod template. Example

apiVersion: kibana.k8s.elastic.co/v1alpha1
kind: Kibana
metadata:
  name: KIBANA_NAME
spec:
  version: 7.2.0
  nodeCount: 1
  config:
    elasticsearch.ssl.certificateAuthorities: /mnt/usr/ca.crt
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
        - name: certs
          mountPath: /mnt/usr
      volumes:
      - name: certs
        secret:
          secretName: clusterInternalCaCertSecretName
  elasticsearchRef:
    name: ES_NAME

another thing that's missing from the docs is:
What altNames do custom certs have to have for ES and Kibana to work? I kind of reverse-engineered this but it would be good to have it in the docs as well.

Absolutely, I will create a doc issue for that.

[geekflyer]

I think the problem here is that we implicitly assume that the custom certificate you configure in Elasticsearch is also a CA or uses a public CA...

Yeah well, no public would issue certs for kubernetes' cluster internal DNS names like *.es-instances.svc (.svc and .svc.cluster.local are not even ICANN approved TLDs), so I would say it is very rare that someone passes a trusted custom cert to elaticsearch directly.
Of course that could be somewhat circumvented by tampering with kube dns or routing all traffic from kibana to ES via a public IP / domain, but that seems like a lot of work as well. I think most people will want to expose their elasticsearch instance anyways just VPC or cluster-internal (unlike Kibana which may as well be exposed to the Internet).

Thanks for your workaround suggestion, I'll try that out.

[pebrc]

We could inspect the user provided certificate secret to see if it contains a ca.crt as well and propagate that through the association controller to Kibana. That would cover the cert-manager case where the ca.crt is included in the same secret.

[geekflyer]

yeah that'd be great!

[jennifer-klemisch-seagen]

I think the problem here is that we implicitly assume that the custom certificate you configure in Elasticsearch is also a CA or uses a public CA, which is of course a problematic assumption we need to reconsider.

One workaround is to include the whole certificate chain in the tls.crt but it is my understanding that cert-manager does not currently support that and that it could be considered problematic.

As far as I can see it kibana does not mount anywhere the ca.crt that is contained in the elasticsearch secret (named CERT_NAME above) and could be used to establish trust.

What happens here is that we create a $KIBANA_NAME-kb-es-ca which contains the tls.crt you configured in Elasticsearch and this secret is then mounted into the the Kibana container. Of course that does not help you because Kibana cannot verify the issuer of that certificate.

I also tried to setup a kibana without elasticsearchRef via the undocumented elasticsearch section of Kibana:

The reason this section is undocumented is that is meant for internal use by the operator only and the restrictions apply that you have to specify everything in that section and the operator might also overwrite what you specified at any time.

The supported workaround for your issue is to use standard Kibana configuration in combination with a pod template. Example

apiVersion: kibana.k8s.elastic.co/v1alpha1
kind: Kibana
metadata:
  name: KIBANA_NAME
spec:
  version: 7.2.0
  nodeCount: 1
  config:
    elasticsearch.ssl.certificateAuthorities: /mnt/usr/ca.crt
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
        - name: certs
          mountPath: /mnt/usr
      volumes:
      - name: certs
        secret:
          secretName: clusterInternalCaCertSecretName
  elasticsearchRef:
    name: ES_NAME

another thing that's missing from the docs is:
What altNames do custom certs have to have for ES and Kibana to work? I kind of reverse-engineered this but it would be good to have it in the docs as well.

Absolutely, I will create a doc issue for that.

Did Docs for the altNames get created?

Several years later this is still an issue and confusing. I'd expect to be able to add a custom CA Cert, to the Http.tls. and give it multiple names (enterprise-search, kibana, dns & ips). Then when going to the kibana or ent external urls it should see the cert and say it's secure. This is not the case.