Elasticsearch X-Pack valid ssl certificate not trusted by client because ca chain not provided by server.

[csahli]

Describe the feature:
Elasticsearch x-pack security settings

Elasticsearch version (bin/elasticsearch --version):
./elasticsearch --version
Version: 6.2.0, Build: 37cdac1/2018-02-01T17:31:12.527918Z, JVM: 1.8.0_171

Plugins installed: []
./elasticsearch-plugin list

repository-s3
x-pack
	x-pack-core
	x-pack-deprecation
	x-pack-graph
	x-pack-logstash
	x-pack-ml
	x-pack-monitoring
	x-pack-security
	x-pack-upgrade
	x-pack-watcher

JVM version (java -version):
java -version

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

OS version (uname -a if on a Unix-like system):
uname -a
Linux node1.elasticsearch.paris.sasstacloud.sascloud.io 3.10.0-862.6.3.el7.x86_64 #1 SMP Fri Jun 15 17:57:37 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Valid SSL certificates provided by elasticseach nodes not trusted.
Steps to reproduce:

1. Install and configure x-pack security to enable http transport over ssl.
2. Use a valid ssl certificate signed by a trusted CA. Let's Encrypt this my case.
3. Perform any https request on elasticsearch api

Here is the elasticsearch x-pack configuration we set:

xpack.security.enabled: true
xpack.ssl.verification_mode: none
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.http.ssl.verification_mode: certificate 
xpack.security.http.ssl.key: /etc/elasticsearch/tls/node.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/tls/node.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/tls/chain.cer" ] 
xpack.security.transport.ssl.key: /etc/elasticsearch/tls/node.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/tls/node.cer 
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/tls/chain.cer" ]

Then preform an https request on elasticsearch

sh-4.2# curl -v https://elasticsearch.paris.sasstacloud.sascloud.io:9200
* About to connect() to elasticsearch.paris.sasstacloud.sascloud.io port 9200 (#0)
*   Trying 10.145.30.94...
* Connected to elasticsearch.paris.sasstacloud.sascloud.io (10.145.30.94) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=elasticsearch.paris.sasstacloud.sascloud.io
*       start date: Jun 29 13:50:01 2018 GMT
*       expire date: Sep 27 13:50:01 2018 GMT
*       common name: elasticsearch.paris.sasstacloud.sascloud.io
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
*** NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.
* Closing connection 0**
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

But with any other web browser performing the same https request, the ssl certificate is trusted and valid:

Here is the output of a request done using the openssl s_client:

sh-4.2# openssl s_client -showcerts -host elasticsearch.paris.sasstacloud.sascloud.io -port 9200
CONNECTED(00000003)
**depth=0 CN = elasticsearch.paris.sasstacloud.sascloud.io
verify error:num=20:unable to get local issuer certificate**
verify return:1
depth=0 CN = elasticsearch.paris.sasstacloud.sascloud.io
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=elasticsearch.paris.sasstacloud.sascloud.io
  i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIGQDCCBSigAwIBAgISA5tWGfplAZn411Je7YlKU1+lMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODA2MjkxMzUwMDFaFw0x
ODA5MjcxMzUwMDFaMDYxNDAyBgNVBAMTK2VsYXN0aWNzZWFyY2gucGFyaXMuc2Fz
c3RhY2xvdWQuc2FzY2xvdWQuaW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDB8VvUxHHiuJ13UO+OEVrdLW4nNmBrnA7eOm9TM46Z5EBzHyV2WO9x3F9k
cFSQ4BQ4jmig0JT3XU7hmqZXg5bQN2fL4/px1GtEb3O+/YgjVk2J0N9RC56iUqJS
TN9FmfJcAQPck1QA8OC2yVqB7SqnYJsg4wUUGMoZumtIaHvIZZ9XeTQzg8/Y5aP6
zjN5cuuVlPPXxFLfPJlK+WgwsIHUkKSvP5kA8ds4HLsc0NTRWGpHGMCwDGRFxYa3
sMn6LHzntdWgvcTcxHVUSJBKcS6gX8djLHOtLg81tWDSbaT973C34cP/uD8ATLQv
llBE5lN8GTQoIHcDNw9xOKJymIW9AgMBAAGjggMyMIIDLjAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFPd2Qva6TUxC2F2juR7PDlbsOu1sMB8GA1UdIwQYMBaAFKhKamME
fd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0
cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0
cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0Lm9yZy8wNgYDVR0RBC8wLYIrZWxh
c3RpY3NlYXJjaC5wYXJpcy5zYXNzdGFjbG91ZC5zYXNjbG91ZC5pbzCB/gYDVR0g
BIH2MIHzMAgGBmeBDAECATCB5gYLKwYBBAGC3xMBAQEwgdYwJgYIKwYBBQUHAgEW
Gmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIGrBggrBgEFBQcCAjCBngyBm1Ro
aXMgQ2VydGlmaWNhdGUgbWF5IG9ubHkgYmUgcmVsaWVkIHVwb24gYnkgUmVseWlu
ZyBQYXJ0aWVzIGFuZCBvbmx5IGluIGFjY29yZGFuY2Ugd2l0aCB0aGUgQ2VydGlm
aWNhdGUgUG9saWN5IGZvdW5kIGF0IGh0dHBzOi8vbGV0c2VuY3J5cHQub3JnL3Jl
cG9zaXRvcnkvMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHYAKTxRllTIOWW6qlD8
WAfUt2+/WHopctykwwz05UVH9HgAAAFkTAW9EgAABAMARzBFAiEAlnV+V2m9j/CT
bHEcFrcsobQyUALSsvkiBbnnlwYqDVoCIHK1gSs9m0nv/dOSH9j+HSi8K1/Oot4v
G3Fzvjpg7mqoAHUAb1N2rDHwMRnYmQCkURX/dxUcEdkCwQApBo2yCJo32RMAAAFk
TAW/kwAABAMARjBEAiEAoRwpjrISarna6VR/sAIyTGblLMgrdtXecHnoBDCI9kQC
HxVeFSFyrMYlNW7P8Nv0B3cnp17uBEdNnK9q389xFGIwDQYJKoZIhvcNAQELBQAD
ggEBADjLPWWvCEdRUkl59OcMb+EucvRwYlKrSjWkBbVH98m/z+R3ipI5/ufnCafj
Ho4wGwTQ1P2j6r/k5sC3TK5FjgmY32Cj7sZvBoGLq2i0wOrUhwG2kGohXCguypgm
wgLqgMxpe1qzYAXncunS00vWj+a1nhEgsJ+ZE6EqdsbzsSKBLtwCyqmgVwxSh5g6
iGTPg8C8iUT485+t83WKlt6APlBCU6SAE5knXBfhDv36UW1IBkGbTBqqarHieDrA
M58FKSSabyggyAZX/QB3LBzKFdI9bw26lf3GHqcdYuO6/IrIkoi6dRX2OMegy2Np
tZ5uxITgfgC1gfwj5qcBNvLFV9k=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=elasticsearch.paris.sasstacloud.sascloud.io
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2144 bytes and written 471 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
   Protocol  : TLSv1.2
   Cipher    : ECDHE-RSA-AES256-SHA384
   Session-ID: 5B36579BE33DD71E277BD8D376C507CC9136B7A80C98FE05DEC3C6D34B9EB939
   Session-ID-ctx:
   Master-Key: 89962CACF8AE173D264A169C31ED47BA1DE00034D11173CEE4B8BAEF4C6A07E8C847F42F5F2641B856B8ABA13FED86DF
   Key-Arg   : None
   Krb5 Principal: None
   PSK identity: None
   PSK identity hint: None
   Start Time: 1530288029
   Timeout   : 300 (sec)
   Verify return code: 21 (unable to verify the first certificate)

The problem seems to be due to a bad behavior of elastaicsearch which doesn't send the entire certificate chain during the handshake making the client unable to validate the intermediate certificates.

To workaround the problem we applied the following configuration:

xpack.security.enabled: true
xpack.ssl.verification_mode: none
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.http.ssl.verification_mode: certificate 
xpack.security.http.ssl.key: /etc/elasticsearch/tls/node.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/tls/chain.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/tls/chain.cer" ] 
xpack.security.transport.ssl.key: /etc/elasticsearch/tls/node.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/tls/node.cer 
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/tls/chain.cer" ]

replaced:
xpack.security.http.ssl.certificate: /etc/elasticsearch/tls/node.cer
by:
xpack.security.http.ssl.certificate: /etc/elasticsearch/tls/chain.cer

We basically defined the full ssl chain ca as the ssl certificate. This make elasticsearch sending the full ca chain including the ssl certificate to the client during the ssl handshake.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

The problem seems to be due to a bad behavior of elastaicsearch which doesn't send the entire certificate chain during the handshake making the client unable to validate the intermediate certificates.

Is it really an ES problem? Because I understand that you only configure a node.cer instead of chain.cer.
The file contents of the setting certificate is sent as is during handshake. The certificate_authorities setting is used only to validate authenticated clients (browser certificate) during handshake.

[albertzaharovits]

@csahli I am going to speculatively close this by pointing to the manual:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/configuring-tls.html#tls-http

Please do reopen if it appears that I'm mistaking.

[csahli]

Thank you for your support but I am not able to reopen a closed issue. I am afraid I didn't explain correctly the issue. According to the manual which I followed to configure x-pack, you can provide the ssl certificate using 2 distinct formats: pkcs12 or pem.
I am using pem format as defined in the manual.

Extract from the manual:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /home/es/config/x-pack/node01.key 
xpack.security.http.ssl.certificate: /home/es/config/x-pack/node01.crt 
xpack.security.http.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ] 

xpack.security.http.ssl.key: The full path to the node key file.
xpack.security.http.ssl.certificate: The full path to the node certificate.
xpack.security.http.ssl.certificate_authorities: An array of paths to the CA certificates that should be trusted.

The ssl client certificate is a file containing a public key generated by a client using its private key and signed by a CA. The client certificate is not suppose to contain the CA Chain. Providing the CA chain instead of the client certificate is unexpected behavior. Elasticsearch should instead discover the Intermediate CA using the CA cert or the CA chain and send the chain during the ssl handshake.

[csahli]

I review your answer and I think you may be misunderstood my issue. To workaround the issue we used the CA Full chain as value of the xpack.security.http.ssl.certificate parameter. According to the configuration manual you shared, this parameter is supposed to be used to set the SSL cert not the Full CA Chain. If this is an expected behavior, you should update the manual to reflect that.

xpack.security.http.ssl.certificate: The full path to the node certificate.
should then be
xpack.security.http.ssl.certificate: The full path to the full ca chain.

[albertzaharovits]

@csahli,
Thanks for the detailed diagnosis. For future such issues, it is better to reach out to https://discuss.elastic.co/c/elasticsearch . It has higher reach, so more people can chime in, and there is a good chance a similar issue came around before.

I think I understand the issue, but I jumped the gun without a proper detailed answer, apologies.

Your last comment spells the problem. Indeed the xpack.security.http.ssl.certificate setting should contain a chain.

---

For completeness:

Elasticsearch should instead discover the Intermediate CA using the CA cert or the CA chain and send the chain during the ssl handshake.

Elasticsearch does not, and it would be shoddy, to "generate" chains. It simply forwards the contents of the xpack.security.http.ssl.certificate file.

The browser, you say, validates the certificate, however curl trips. Following is an excerpt of the curl failure:

More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.

Elasticsearch sends the same certificate (chain) to both the browser and curl. A TLS party does not "construct" chains for the other to trust. It sends what it got, and hopes the other will trust it. The browser trusts it, but curl does not. From the excerpt, I would try to add the Let's encrypt CA with the --cacert option. There are other ways as well (curl follows some env vars to the system level truststore).

---

I will open a docs PR about the *.certificate settings to be more on the lines of:
https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate , ie Specifies a file with the certificate in the PEM format for the given virtual server. If intermediate certificates should be specified in addition to a primary certificate, they should be specified in the same file in the following order: the primary certificate comes first, then the intermediate certificates.
at least for https.

Does this sound fair to you, @csahli ?

[csahli]

@albertzaharovits Thank you for you reactivity. I am totally ok with your proposal of updating the documentation to reflect nginx approach.

There is another approach which is clearly more expensive ... It would consist in adding a new parameter
xpack.security.http.ssl.certificate_chain for CA chain or CA bundle and changing the parameter xpack.security.http.ssl.certificate_authorities to xpack.security.http.ssl.trusted_certificate_authorities

However openssl terminology is clear: ssl cert, ssl bundle and ssl chain are 3 distinct things with distincts roles. And for server side encryption, this is the server role to provide the chain of trust. No matter if the server has to discover or generate it. If the chain is not provided by the server and the client refuses the ssl handshake, this is a server issue not a client issue. Nginx added the comment you mentioned on their documentation because they understand they don't respect openssl terminology.

Thank you for your kind support and for your time

[albertzaharovits]

Thanks for these suggestions @csahli , much appreciated.
@tvernum is more versed on matters as such. Sure he'll enjoy this banter 😄

[tvernum]

However openssl terminology is clear

but we're not using openssl, and openssl doesn't define the TLS standards (thankfully).

Your point about the documentation being unclear is fair. Getting TLS docs correct is a difficult balancing act - most users just want step by step instructions and don't really want to have to get into details that don't concern them. We intentionally leave out some finer details from those docs because if we put everything in then the docs would be so long that it would scare people off. I agree that they're not quite right here, and we'll work out what we can do to overcome that, but I hope you understand that what you were reading is a guide to setting up TLS, not a full reference, so it's never going to be able to answer every scenario.

The settings reference actually says:

xpack.security.http.ssl.certificate
Path to a PEM encoded file containing the certificate (or certificate chain) that will be presented when requested.

So in summary:

• The current behaviour is intentional
• It is possible to do everything you need to do within the current behaviour
• The documentation doesn't make that clear enough, and we'll try and improve it.
• If we were designing the settings from scratch then we might name them differently, but any change we made here would break existing configurations and I don't think there is sufficient value to justify that level of pain for existing users.

[wchrisdean]

[doc issue triage]

[ceefour]

@csahli THANK YOU!!! I've been confused in the last few hours and you saved me!

I review your answer and I think you may be misunderstood my issue. To workaround the issue we used the CA Full chain as value of the xpack.security.http.ssl.certificate parameter. According to the configuration manual you shared, this parameter is supposed to be used to set the SSL cert not the Full CA Chain. If this is an expected behavior, you should update the manual to reflect that.

xpack.security.http.ssl.certificate: The full path to the node certificate.
should then be
xpack.security.http.ssl.certificate: The full path to the full ca chain.

Update: Although curl works, Kibana still won't connect to my elasticsearch node:

kib01    | {"type":"log","@timestamp":"2020-03-26T20:13:24Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://es01-SOMEDOMAIN:9243/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to get issuer certificate"}

Workaround:

in elastic-docker-tls.yml:

services:
  kib01:
    environment:
      ELASTICSEARCH_SSL_VERIFICATIONMODE: none

[eedugon]

@csahli , @ceefour , let me add a few quick comments here:

I think there's no need to add the entire / full chain (including the root CA) into the server certificate setting. The best practice here (in my view) is to set the certificate with a chain including your server certificate + all intermediate CAs you might have but not the root CA, as that one is actually useless at this level. I usually call this cert chain the "server cert chain".

With that server chain, curl command should work fine (assuming the root CA is a public CA and part of the OS list of installed CAs), and if it doesn't work providing the ca-chain in the --cacert parameter should make it working (see what to trust section below).

Setting the server certificate with just the server certificate is also possible, but then you need to make sure that your clients know all the intermediate and root CAs. Some smart clients like web browsers are able to fetch directly the intermediate CAs and do the process of validation smoother, but some other clients like curl will never do that. I always try to make a setup for all clients to work without needing to provide the intermediate CAs, that's the reason of my proposal.

But as Tim mentioned, there isn't any standard defined for this.

On the other side, on the what to trust side (truststore), we could provide the client with only the root CA (which might work if the server is providing the intermediate CAs together with its own certificate) or giving the client a cert chain including the intermediate CAs + rootCA. I like more this second approach. I think it's safer and I always try to implement it when possible. I call this chain the ca-chain (with all CAs needed to trust the certificate).

So, for the kibana configuration shared by @ceefour , the following should work too.

elasticsearch.ssl.certificateAuthorities: ["/path/to/ca-chain.pem"]
elasticsearch.ssl.verificationMode: "certificate"

In summary:

• When intermediate CAs are in the game, configuring only the server cert at server certificate side and only the rootCA at what to trust side will tend to fail in many scenarios. Hence is better to add a server-cert-chain.pem configured at server side (but not the full chain).

All this is not really related with Elasticsearch, is more generic SSL stuff, so I don't know if we should include this in the docs. Probably not, or maybe a very short comment about it (add any intermediate CA together with the server certificate in the certificate setting).

If you continue having issues I guess we should move this to https://discuss.elastic.co/c/elasticsearch

[sandeepkanabar]

Thanks for such a crystal clear comment @eedugon !

[sandeepkanabar]

Thanks for such a crystal clear comment @eedugon !

[anthonylavado]

@csahli Thank you from 2021. You just saved me a whole lot of effort. I've been poring over the documentation and couldn't understand why this wasn't working right. This made it crystal clear.

[zffocussss]

The problem seems to be due to a bad behavior of elastaicsearch which doesn't send the entire certificate chain during the handshake making the client unable to validate the intermediate certificates.

Is it really an ES problem? Because I understand that you only configure a node.cer instead of chain.cer. The file contents of the setting certificate is sent as is during handshake. The certificate_authorities setting is used only to validate authenticated clients (browser certificate) during handshake.

I do not think it is a mutualTLS handshake, why does es validate the client with its own ca certificate?
a common setting in a server side ssl configurate is a cerfiticate and a private key.you can see many examples in nginx conf

[harwinds]

Thanks a ton @eedugon from 2023. In our case we have our own Private CAs, and we haven't installed any CA (interemdiate or Root) cert to the host truststore, and providing both the Server + Intermediate + Root to the setting "xpack.security.http.ssl.certificate" worked like a charm.