SSL configuration does not appear to work.

[srbry]

When configuring the certs etc to enable SSL. It appears that SSL doesn't get properly enabled. After confirming certs/ ciphers etc are all valid using openssl we are still getting handshake errors.

It sounds like we need to configure auth_mechanisms to "External" to allow SSL auth, but that option doesn't seem to be present in this release?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[srbry]

the ssl config we are trying to use:

export SSL_KEY=""
export LOAD_DEFINITIONS=""



export SSL_SUPPORTED_TLS_VERSIONS="['tlsv1.2','tlsv1.1']"
export SSL_SUPPORTED_TLS_CIPHERS=",{ciphers,[\"ECDHE-ECDSA-AES256-GCM-SHA384\",\"ECDHE-RSA-AES256-GCM-SHA384\",\"ECDHE-ECDSA-AES256-SHA384\",\"ECDHE-RSA-AES256-SHA384\",\"ECDHE-ECDSA-DES-CBC3-SHA\",\"ECDH-ECDSA-AES256-GCM-SHA384\",\"ECDH-RSA-AES256-GCM-SHA384\",\"ECDH-ECDSA-AES256-SHA384\",\"ECDH-RSA-AES256-SHA384\",\"DHE-DSS-AES256-GCM-SHA384\",\"DHE-DSS-AES256-SHA256\",\"AES256-GCM-SHA384\"]}"
export SSL_VERIFY="false"
export SSL_VERIFICATION_DEPTH="5"
export SSL_FAIL_IF_NO_PEER_CERT="false"


  export SSL_KEY="-----BEGIN RSA PRIVATE KEY-----
cert_contents_go_here
-----END RSA PRIVATE KEY-----
"


export CLUSTER_PARTITION_HANDLING="pause_minority"
export DISK_ALARM_THRESHOLD="{mem_relative,0.4}"

[michaelklishin]

Consider going through the TLS troubleshooting guide and posting actual log/error messages from both server and client ends.

There can be perfectly legitimate reasons for TLS connections to fail even if everything is set up correctly as far as RabbitMQ is concerned: e.g. client certificate(s) are not trusted on the node.

[srbry]

Hi @michaelklishin thanks for responding. I had been going through the troubleshooting guide earlier. Everything was working as expected, SSL listeners being started etcs, however I cannot connect via SSL with the RabbitMQ nodes (on AMQP+SSL). All the SSL errors were relating to SSL handshakes. When I get access to the logs ill post some extra detail in here.

Has the latest release (v241) been tested with AMQP+SSL? I can't see anything obvious thats wrong in our config and it would very helpful to know if anyone else has it working so we can rule it out being a bug.

[michaelklishin]

@srbry you'd get a much more informed response if you post actual log entries from both sides plus openssl s_client output as demonstrated in the TLS troubleshooting guide. "Related to SSL handshakes" is not enough. Engineers need facts and cold hard data to work with.

To rule out any possible version-specific differences in behavior, simply deploy an older version and compare.

[srbry]

@michaelklishin I fully intend to get the data. I haven't got access to the system to re-run the tests right now so I was simply trying to establish if there were any known issues (or common misconfigurations)/ working deployments on the latest version.

[srbry]

@michaelklishin here are some more useful details.

Startup logs

=INFO REPORT==== 11-Apr-2018::07:56:00 ===
Starting RabbitMQ 3.6.15 on Erlang 19.3.6.4
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 11-Apr-2018::07:56:00 ===
node           : rabbit@f7df42505e1428119ad3649d19fc9998
home dir       : /var/vcap/store/rabbitmq
config file(s) : /var/vcap/jobs/rabbitmq-server/bin/../etc/rabbitmq.config
cookie hash    : ciQiRJc0xMxRc8mGjDUP3Q==
log            : /var/vcap/sys/log/rabbitmq-server/rabbit@f7df42505e1428119ad3649d19fc9998.log
sasl log       : /var/vcap/sys/log/rabbitmq-server/rabbit@f7df42505e1428119ad3649d19fc9998-sasl.log
database dir   : /var/vcap/store/rabbitmq/mnesia/db

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Memory high watermark set to 1505 MiB (1578134732 bytes) of 3762 MiB (3945336832 bytes) total

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Enabling free disk space monitoring

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Disk free limit set to 1578MB

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Limiting to approx 299900 file handles (269908 sockets)

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
FHC read buffering:  OFF
FHC write buffering: ON

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Starting rabbit_node_monitor

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Management plugin: using rates mode 'basic'

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
started SSL Listener on 0.0.0.0:5671

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Management plugin started. Port: 15672

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Statistics database started.

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
...skipping...
Starting rabbit_node_monitor

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Management plugin: using rates mode 'basic'

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 11-Apr-2018::07:56:02 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
started SSL Listener on 0.0.0.0:5671

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Management plugin started. Port: 15672

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Statistics database started.

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
opening log file: "/var/vcap/sys/log/rabbitmq-server/management-ui/access.log.2018_04_11_07"

=INFO REPORT==== 11-Apr-2018::07:56:02 ===
Server startup complete; 7 plugins started.
 * rabbitmq_management
 * rabbitmq_auth_mechanism_ssl
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * cowboy
 * amqp_client
 * cowlib

=INFO REPORT==== 11-Apr-2018::07:56:06 ===
Deleting user 'guest'

=INFO REPORT==== 11-Apr-2018::07:56:07 ===
Deleting user 'rmq-mgmt-user'

=INFO REPORT==== 11-Apr-2018::07:56:09 ===
Creating user 'rmq-mgmt-user'

=INFO REPORT==== 11-Apr-2018::07:56:11 ===
Changing password for 'rmq-mgmt-user'

=INFO REPORT==== 11-Apr-2018::07:56:12 ===
Setting user tags for user 'rmq-mgmt-user' to [administrator]

=INFO REPORT==== 11-Apr-2018::07:56:16 ===
Setting permissions for 'rmq-mgmt-user' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 11-Apr-2018::07:56:17 ===
Setting permissions for 'rmq-mgmt-user' in 'cb7e8c40-a894-43d1-a1fe-b16b6aef53dd' to '.*', '.*', '.*'

=INFO REPORT==== 11-Apr-2018::07:56:19 ===
Creating user 'rmq-broker-admin'

=INFO REPORT==== 11-Apr-2018::07:56:21 ===
Changing password for 'rmq-broker-admin'

=INFO REPORT==== 11-Apr-2018::07:56:22 ===
Setting user tags for user 'rmq-broker-admin' to [administrator]

=INFO REPORT==== 11-Apr-2018::07:56:26 ===
Setting permissions for 'rmq-broker-admin' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 11-Apr-2018::07:56:27 ===
Setting permissions for 'rmq-broker-admin' in 'cb7e8c40-a894-43d1-a1fe-b16b6aef53dd' to '.*', '.*', '.*'

From the above I can see that we are starting SSL Listeners, I have enabled the rabbitmq_auth_mechanism_ssl plugin, not sure if I need that or any others to make this work.

SSL Test

Openssl

rmq/85b06195-6aa7-4b26-9276-6634b7ffc222:/var/vcap/sys/log/rabbitmq-server# openssl s_client -debug -connect localhost:5671 -cert /var/vcap/jobs/rabbitmq-server/etc/cert.pem -key /var/vcap/jobs/rabbitmq-server/etc/key.pem -CAfile /var/vcap/jobs/rabbitmq-server/etc/cacert.pem
CONNECTED(00000003)
write to 0x26a63f0 [0x26a6de0] (295 bytes => 295 (0x127))
0000 - 16 03 01 01 22 01 00 01-1e 03 03 a9 6d cc e0 f4   ....".......m...
0010 - da 20 93 41 bf 35 d9 39-08 8c 1d 6d 58 45 af b0   . .A.5.9...mXE..
0020 - 5b 4c 25 a0 e0 42 31 3c-3d 01 69 00 00 88 c0 30   [L%..B1<=.i....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b   .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a   .j.9.8.....2...*
0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 12   .&.......=.5....
0060 - c0 08 00 16 00 13 c0 0d-c0 03 00 0a c0 2f c0 2b   ............./.+
0070 - c0 27 c0 23 c0 13 c0 09-00 a2 00 9e 00 67 00 40   .'.#.........g.@
0080 - 00 33 00 32 00 9a 00 99-00 45 00 44 c0 31 c0 2d   .3.2.....E.D.1.-
0090 - c0 29 c0 25 c0 0e c0 04-00 9c 00 3c 00 2f 00 96   .).%.......<./..
00a0 - 00 41 c0 11 c0 07 c0 0c-c0 02 00 05 00 04 00 15   .A..............
00b0 - 00 12 00 09 00 ff 01 00-00 6d 00 0b 00 04 03 00   .........m......
00c0 - 01 02 00 0a 00 34 00 32-00 0e 00 0d 00 19 00 0b   .....4.2........
00d0 - 00 0c 00 18 00 09 00 0a-00 16 00 17 00 08 00 06   ................
00e0 - 00 07 00 14 00 15 00 04-00 05 00 12 00 13 00 01   ................
00f0 - 00 02 00 03 00 0f 00 10-00 11 00 23 00 00 00 0d   ...........#....
0100 - 00 20 00 1e 06 01 06 02-06 03 05 01 05 02 05 03   . ..............
0110 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02   ................
0120 - 02 03 00 0f 00 01 01                              .......
read from 0x26a63f0 [0x26ac340] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28                              ......(
139715452597920:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:770:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 295 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

RabbitMQ logs

=ERROR REPORT==== 11-Apr-2018::08:04:29 ===
SSL: hello: tls_handshake.erl:127:Fatal error: handshake failure - malformed_handshake_data

Nmap

rmq/85b06195-6aa7-4b26-9276-6634b7ffc222:/var/vcap/sys/log/rabbitmq-server# nmap --script +ssl-enum-ciphers -p 5671 localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2018-04-11 08:08 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000039s latency).
Other addresses for localhost (not scanned): 127.0.0.1
PORT     STATE SERVICE
5671/tcp open  unknown
| ssl-enum-ciphers:
|   TLSv1.1: No supported ciphers found
|_  TLSv1.2: No supported ciphers found

Nmap done: 1 IP address (1 host up) scanned in 0.31 seconds

[srbry]

Just for extra info, when running:

/var/vcap/sys/log/rabbitmq-server# openssl s_server -accept 8443 -cert /var/vcap/jobs/rabbitmq-server/etc/cert.pem -key /var/vcap/jobs/rabbitmq-server/etc/key.pem -CAfile /var/vcap/jobs/rabbitmq-server/etc/cacert.pem

with

rmq/85b06195-6aa7-4b26-9276-6634b7ffc222:~# openssl s_client  -connect localhost:8443 -cert /var/vcap/jobs/rabbitmq-server/etc/cert.pem -key /var/vcap/jobs/rabbitmq-server/etc/key.pem -CAfile /var/vcap/jobs/rabbitmq-server/etc/cacert.pem

---
SSL handshake has read 2327 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5E02B91E15D4496DBEE24574D0B18C86D30AB4D6E20439CE01EBB5420B3EE070
    Session-ID-ctx:
    Master-Key: EC22A187A6CB19DDC108C0C2BD3D0F1D6F44B3DD3C8FF754BB0C66CF3812C0DDBE983775F61A77CCDE7E56364D9F298D
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 6f b6 8f a3 61 3c f7 97-e3 90 f4 ce ee da 60 cd   o...a<........`.
    0010 - 52 3b d9 f4 bc 18 ea 6e-24 3a 40 95 4f ec dd de   R;.....n$:@.O...
    0020 - 1b e6 9c 0a f6 c9 bd fe-60 57 4f f4 e2 73 96 26   ........`WO..s.&
    0030 - 66 dd 38 3a 3d 60 f8 a2-ec 33 4c 95 03 84 6d 2c   f.8:=`...3L...m,
    0040 - d8 42 03 14 72 f7 15 08-e9 1b 37 b7 82 3d e4 31   .B..r.....7..=.1
    0050 - 9b 14 ff 20 8f 40 2b 28-9d d4 f4 fd db 1a 0e 9f   ... .@+(........
    0060 - 60 4a ab 5e ab 54 9c 44-97 f8 c9 40 e8 68 10 ff   `J.^.T.D...@.h..
    0070 - 4b 5f 4c 8b 3d fa fd a2-6b fe 7d 4f 04 87 9f 9b   K_L.=...k.}O....
    0080 - bc 8e 4f 52 77 31 e3 d7-c5 0c 2d 02 b7 ef de a0   ..ORw1....-.....
    0090 - a0 f0 54 5b 53 f7 3a 86-63 ee e6 f6 84 a9 e2 36   ..T[S.:.c......6

    Start Time: 1523434351
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[michaelklishin]

error: handshake failure - malformed_handshake_data

is definitely something to investigate, even though it's not very specific. Specifically malformed_handshake_data is very rare to see and seems to come down to an available cipher suite mismatch.

I don't know what cipher suites this release may be forcing by default but it is something that the user can control.

I am surprised to see

routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

output by s_client. If it really does use SSLv3 then it won't be able to connect because RabbitMQ disables SSLv3 by default. s_client can be instructed to use TLSv1 or later via CLI flags.

But it can be a red herring because the output against s_server contains

New, TLSv1/SSLv3

but then the two negotiate to use

Protocol : TLSv1.2

There are more threads for other Erlang-based projects that hint at cipher suite unavailability/mismatch.

Consider trying with OTP 20.3 to compare.

[gerhard]

SSL definitely works in this release, my hunch is that it's a configuration issue. Will provide more details when a deployment with SSL completes successfully.

[michaelklishin]

@gerhard @srbry I'd recommend a couple of things to make sure there's at least some overlap in the environments discussed:

• Switch to OTP 20.3 (no reason not do it)
• Use tls-gen for certificate generation if possible

There are too many variables at play otherwise. Thank you for your help, @gerhard.

[srbry]

Thanks @gerhard and @michaelklishin. I will also keep digging, I have currently configured rabbitmq to use all the ciphers that show up from rabbitmqctl eval 'ssl:cipher_suites(openssl).' so not sure what else we can do from that point of view.

I will give the switch a go. We are currently letting bosh int generate our certs but I can look at tls-gen if thats likely to make a difference. I will let you know how I get on!

[srbry]

@michaelklishin @gerhard using tls-gen seems to have resolved the issue. It does raise another question about using interpolate/ credhub (I haven't actually tried credhub yet) for doing the cert generation.

[srbry]

@michaelklishin @gerhard I am not giving in on this one, its a bit of a strange one. After upgrading to OTP 20.3 I can now see the following message in the logs:

2018-04-11 18:59:41.708 [info] <0.270.0> SSL WARNING: Ignoring a CA cert as it could not be correctly decoded.

Its a bit strange considering they are read exactly the same using openssl.
Bosh generated:

openssl x509 -in <(bosh int rmq-var-store.yml --path /rabbitmq_server/ca) -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ff:11:c0:ea:25:37:af:3f:00:a3:9a:48:f5:1d:04:8b
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=USA, O=Cloud Foundry, CN=rabbitmq|CA
        Validity
            Not Before: Apr 10 12:55:20 2018 GMT
            Not After : Apr 10 12:55:20 2019 GMT
        Subject: C=USA, O=Cloud Foundry, CN=rabbitmq|CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:aa:92:dd:ad:4b:ce:df:fd:b9:fd:e3:48:ad:81:
                    ed:42:d7:0d:12:bc:13:e4:f5:62:6c:d4:ce:13:12:
                    2b:77:69:4c:56:2d:e1:7e:51:60:76:cd:2d:2b:7f:
                    31:70:1b:e9:fd:90:4f:65:d0:85:60:00:ff:a6:d7:
                    38:47:2e:18:e1:60:d8:ef:0f:59:e6:c4:37:8d:15:
                    54:e8:6d:cb:1a:06:47:a6:20:a7:ea:81:15:92:b2:
                    b6:71:49:26:93:00:a8:fc:6e:6e:76:eb:4d:d3:16:
                    a2:b7:ed:94:e5:e9:72:18:71:5f:f3:77:73:63:3e:
                    81:f7:69:7a:ba:24:56:3a:69:aa:9b:b8:2b:71:5f:
                    86:3d:93:c4:d8:59:f0:bc:1b:39:b9:74:11:d3:5b:
                    99:68:fa:1b:88:ca:86:55:cb:92:87:f1:ca:39:11:
                    cf:7f:e2:00:6b:45:e7:4d:42:e9:24:3f:bb:e1:7f:
                    6d:e9:ca:de:06:fa:a4:ea:dd:e4:9f:e9:2d:6a:85:
                    df:9c:fd:33:2b:08:9f:f3:10:af:9d:8d:07:b3:2f:
                    21:9d:4f:bf:16:c5:68:29:8e:86:ff:79:1c:46:c2:
                    75:62:be:08:03:70:7e:54:02:25:3a:ca:ad:b6:37:
                    db:82:f4:57:e9:7a:89:54:f2:f1:b2:4b:4f:9b:f0:
                    07:6f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         59:cb:16:24:0a:85:59:ce:95:18:07:fd:aa:c1:c0:f3:71:7d:
         e1:84:1d:2d:30:af:46:27:b8:fc:9c:68:f5:f3:5a:36:cf:f0:
         f2:5c:20:fb:6f:a1:c4:ff:71:f6:0f:51:b1:07:9b:dc:ca:0a:
         c3:dd:94:f7:4f:a7:0d:ed:4d:67:ef:0c:83:62:03:c5:f9:45:
         ac:23:29:62:b3:62:2b:af:b8:2c:e1:7e:71:b5:9b:78:cc:d5:
         29:d2:7f:e4:a4:75:ce:18:cf:8c:2b:64:b5:38:32:c2:cb:01:
         ff:02:f2:73:a7:a0:1f:a7:07:90:85:0a:66:70:0e:cd:41:22:
         7c:51:b4:9f:5e:a1:71:f0:de:70:0a:84:59:23:3f:9d:09:50:
         82:68:e1:c3:1c:0d:af:03:99:21:1c:77:c6:ee:54:3d:48:2d:
         58:a5:67:d2:22:eb:0e:a0:53:e9:09:28:e4:17:b1:d9:6f:b1:
         a0:3f:81:12:f3:48:d5:56:14:86:52:1e:c4:c5:cf:5a:68:38:
         af:ed:36:65:17:dd:18:99:84:03:26:1f:8d:40:fe:46:cc:4b:
         66:75:67:e9:81:91:82:ae:ee:85:b7:7a:9f:b0:6b:49:7e:66:
         68:42:a5:7b:6f:e4:88:0b:94:cd:43:94:a1:9c:bd:de:15:fd:
         a1:d6:e0:e8

tls-gen generated:

~ $ openssl x509 -in cacert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ae:08:9a:c7:86:e7:78:55
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=TLSGenSelfSignedtRoot|CA
        Validity
            Not Before: Apr 11 13:16:07 2018 GMT
            Not After : Apr 11 13:16:07 2019 GMT
        Subject: CN=TLSGenSelfSignedtRoot|CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d5:54:74:21:98:53:65:5c:6d:8d:84:6c:30:78:
                    cc:9d:76:6b:14:f2:44:ae:c9:ba:de:89:d6:19:19:
                    15:3b:84:bd:cd:7e:c6:41:0c:89:df:fb:0a:c5:01:
                    72:49:3b:d4:d7:a8:9e:f8:78:b3:77:a1:60:83:bf:
                    82:09:c1:30:86:b2:0f:7f:b8:7d:58:31:89:7b:82:
                    d8:64:e7:72:91:57:29:6b:7e:ae:46:c6:0e:33:12:
                    fa:64:bc:33:33:e7:43:0b:6b:ec:be:85:46:be:64:
                    82:a5:c3:40:92:e3:b6:d8:11:aa:26:c2:54:f7:72:
                    70:6f:c8:76:a8:a0:8d:f8:c8:be:4d:16:3c:01:4b:
                    5d:4e:c0:be:12:10:90:39:fa:59:9a:11:35:f4:55:
                    67:8d:0c:f1:84:a0:2c:67:0a:ff:1b:19:da:aa:d3:
                    c2:c3:08:e9:e4:73:a0:29:e8:0d:5b:bb:29:88:c9:
                    92:e0:6e:e1:27:25:69:0c:00:18:e1:9b:9f:86:37:
                    73:dd:08:8b:b5:51:94:63:76:19:4a:85:1f:9a:c6:
                    c1:1d:30:22:57:d7:ae:bd:92:a4:d3:03:72:35:86:
                    63:14:71:6a:75:1d:ef:a7:b6:9b:ea:57:25:03:f5:
                    91:7c:26:db:ab:7e:23:ff:e3:4e:d2:15:26:01:84:
                    17:cd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         40:eb:2c:dd:93:90:d8:19:f0:f1:d9:6e:1e:4a:f1:2f:c6:e5:
         38:1e:0b:1b:be:8c:cc:6f:c7:d5:b5:02:cb:7f:37:f6:7e:d9:
         28:c1:2c:8d:fb:57:bf:86:49:d3:52:9d:45:85:64:b8:9a:23:
         a5:5e:3f:31:fd:35:e9:6f:76:8a:5d:33:b4:1b:97:de:3d:2f:
         a8:cf:49:e0:73:f8:7a:ef:f2:71:bd:24:84:a9:c6:ec:2e:72:
         17:b8:b4:28:05:4f:11:6b:90:cb:f8:ea:4a:2a:7e:a1:fc:c0:
         d7:f1:06:d6:88:cd:26:4e:e7:15:2a:ef:13:62:46:8f:96:56:
         ab:a5:1e:45:a6:e5:c1:95:4c:99:e7:19:35:70:78:04:5e:ea:
         ea:bd:91:22:15:47:35:c6:f6:f3:ef:af:7f:a7:64:21:13:3f:
         78:5a:e3:a1:2c:f6:2b:2a:e4:b6:6e:95:09:00:51:49:68:53:
         d9:cb:2d:39:38:ca:e3:ae:18:0c:ed:fa:0f:18:64:1d:44:32:
         a8:04:79:99:b7:c0:2b:89:29:32:4d:77:70:5c:2e:5e:17:71:
         c8:c2:39:3b:f7:e7:84:29:a9:27:68:c7:a9:37:55:a6:0b:6b:
         b0:35:20:cf:ac:d8:cc:c6:31:2b:f6:1f:d1:d8:ec:a7:f3:fe:
         e2:00:e0:be

[srbry]

@gerhard @michaelklishin Another update. If you generate the certs using CredHub rather than just bosh interpolate the certs are readable so sounds like a bug between Erlang and the bosh interpolate process. Still not exactly sure why the certs are not compatible but using CredHub seems like a good solution anyway.

[mkuratczyk]

I'm closing this issue as it seems like it's been resolved. There are numerous deployments of this release with TLS enabled so it seems very specific to your environment. Also, credhub is probably the best option anyway.