Support amqps

[codefromthecrypt]

Currently, our RabbitMQ collector (and sender) doesn't configure or test for amqps. We should fix that as this impacts the ability to use in environments such as cloud foundry

Thx for figuring this out @reshmik

related: spring-projects/spring-boot#6401

cc @shakuzen @marcingrzejszczak

[shakuzen]

After significant time spent to get SSL configured on my local RabbitMQ instance, I have looked into this. The rabbit hole (pun intended) goes just a bit deeper than I originally expected.

Status quo

Current support

Our current support assumes (or has the constraint) that what is passed to RABBIT_ADDRESSES is a comma-separated list of RabbitMQ Addresses (host:port pairs). The intention here was to easily support clustered RabbitMQ servers without a TCP load balancer. However, this does not allow specifying scheme, which is a problem here when the scheme is amqps instead of amqp. In retrospect, the constraint on RABBIT_ADDRESSES may have not been the best choice.

Cloud providers

Depending on how the connection properties are available from cloud providers, the viable solutions may change. It's a shame I didn't investigate this more before.
@reshmik @marcingrzejszczak
Can you confirm what is available to you from the RabbitMQ cloud providers you are using? Is it just one URI string like the following (with one host only, I assume)?

ampqs://user:pass@host[:port]/virtualHost

Or are the individual parts (username/password/host etc.) also provided as separate properties to some degree?

I'll try to do my due diligence on this tomorrow so we can best support what most Zipkin users may end up using. I'm out of time to look this up today, though.

---

Without a single property provided by the RabbitMQ Java client that can parse the RabbitMQ URI spec for multiple hosts, it's not easy to support both of these options. To be fair, the URI spec doesn't define multiple hosts as valid.

Quickest solution?

The flag for using TLS can be set separately, ConnectionFactory#useSslProtocol, if we want to expose that as a new property.
However, note that this spits out the following error because peer validation is not being used without configuring the trust manager (see Key Managers, Trust Managers and Stores). Though no one in the linked Spring Boot issue seemed to be asking for this part to be configurable.

2017-11-29 22:58:03.680  WARN 64970 --- [           main] c.r.client.TrustEverythingTrustManager   : This trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but prone to man-in-the-middle attacks. Please see http://www.rabbitmq.com/ssl.html#validating-cerficates to learn more about peer certificate validation.

With a flag exposed for that as say RABBIT_SSL_ENABLED, used in combination with passing just the host:port to RABBIT_ADDRESSES, I believe things will work (warning log above withstanding).

This solution feels like it avoids fixing the limitations with our current configuration options, which may be fine if it works for the vast majority of users, but I fear it may not.

Parse all the URIs

A more robust solution compared to above might be the following, which tries to define a version of the RabbitMQ URI spec that handles multiple hosts.

I think ideally we would be able to take a string that optionally contains a scheme, username, password, virtual host, port, and requires at the least one host but has potentially multiple hosts. Then, being able to pass that to the RabbitMQ client and let it parse it would be a cherry on top. Until that happens, we'll have to parse it ourselves.

There are subjective decisions that need to be made for what is allowed in the string, though. We could take inspiration from the MongoDB client's format.

The format of the URI is:

mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]

Which would make ours something like:

[(amqps://|amqp://)][username:password@]host1[:port1][,host2[:port2],...[hostN[:portN]]][/virtualHost]

• scheme is optionally specified as amqps:// or amqp://; absence implies amqp://
• username and password are optionally specified following the scheme; guest/guest are the default if absent
• host1 is required
• port2 is optionally specified; it is 5672 by default or when the scheme is amqp; when the scheme is amqps it defaults to 5671
• a comma-separated list of host:port can optionally follow host1
• /virtualHost can be optionally specified at the end of the string, defaulting to / in its absence

We could also consider parsing query parameters at the end. This could provide a way for users to pass the necessary files for full TLS configuration with a trust manager.

Reasonable (IMO) limitations the above implies are that all hosts would be connected to using the same scheme, authentication, and virtual host.

---

Sorry for the length of the above. I'm glad to hear other solutions if anyone has them, or feedback on anything above.

[reshmik]

Hello,

Please see the format of rabbitmq(amqps) provided in PCF. We tried to parse and hardcode the values such as addresses, vhost, user, password and it keeps saying "connection refused".

"p-rabbitmq": [
{
"credentials": {
"http_api_uris": [
"https://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@pivotal-rabbitmq-management.run.pivotal.io/api/"
],
"ssl": true,
"dashboard_url": "https://pivotal-rabbitmq-management.run.pivotal.io/#/login/574fb21f-1f2c-45cc-ba98-79a1a1601d16/rrsvmq4o2vmfo8v1qgkj3576ns",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"protocols": {
"management+ssl": {
"path": "/api/",
"ssl": false,
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"port": 15672,
"host": "10.10.38.8",
"uri": "http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:15672/api/",
"uris": [
"http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:15672/api/",
"http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:15672/api/"
]
},
"amqp+ssl": {
"vhost": "721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 5671,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": true,
"uri": "amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:5671/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"uris": [
"amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:5671/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:5671/721c2f68-c5da-45a3-9810-4dc80f70ecd7"
]
},
"management": {
"vhost": "721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 15672,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": false,
"uri": "http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:15672/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"uris": [
"http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:15672/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"http://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:15672/721c2f68-c5da-45a3-9810-4dc80f70ecd7"
]
},
"mqtt": {
"username": "721c2f68-c5da-45a3-9810-4dc80f70ecd7:574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 1883,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": false,
"uri": "mqtt://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:1883",
"uris": [
"mqtt://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:1883",
"mqtt://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:1883"
]
},
"mqtt+ssl": {
"username": "721c2f68-c5da-45a3-9810-4dc80f70ecd7:574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 8883,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": true,
"uri": "mqtt+ssl://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:8883",
"uris": [
"mqtt+ssl://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:8883",
"mqtt+ssl://721c2f68-c5da-45a3-9810-4dc80f70ecd7%3A574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:8883"
]
},
"stomp": {
"vhost": "721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 61613,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": false,
"uri": "stomp://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:61613",
"uris": [
"stomp://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:61613",
"stomp://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:61613"
]
},
"stomp+ssl": {
"vhost": "721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"password": "rrsvmq4o2vmfo8v1qgkj3576ns",
"port": 61614,
"host": "10.10.38.8",
"hosts": [
"10.10.38.8",
"10.10.102.8"
],
"ssl": true,
"uri": "stomp+ssl://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:61614",
"uris": [
"stomp+ssl://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8:61614",
"stomp+ssl://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8:61614"
]
}
},
"username": "574fb21f-1f2c-45cc-ba98-79a1a1601d16",
"hostname": "10.10.38.8",
"hostnames": [
"10.10.38.8",
"10.10.102.8"
],
"vhost": "721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"http_api_uri": "https://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@pivotal-rabbitmq-management.run.pivotal.io/api/",
"uri": "amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"uris": [
"amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.38.8/721c2f68-c5da-45a3-9810-4dc80f70ecd7",
"amqps://574fb21f-1f2c-45cc-ba98-79a1a1601d16:rrsvmq4o2vmfo8v1qgkj3576ns@10.10.102.8/721c2f68-c5da-45a3-9810-4dc80f70ecd7"
]
},
"syslog_drain_url": null,
"volume_mounts": [],
"label": "p-rabbitmq",
"provider": null,
"plan": "standard",
"name": "rabbitmq-stream",
"tags": [
"rabbitmq",
"messaging",
"message-queue",
"amqp",
"stomp",
"mqtt",
"pivotal"
]
}
]
}

[codefromthecrypt]

@shakuzen I'd go for RABBIT_SSL_ENABLED if it works for PCF as it can defer the more difficult options to later. It is hard to foresee configuration all providers might need and in general we try to make sure we know concretely what we support. Do you have a PCF account? @reshmik do you have one that can be borrowed?

[reshmik]

Yes most definitely, I have created a new space and given you space manager, dev permissions. You can deploy code and add services from marketplace there. I can assist with additional testing time permitting this week(since we will be at Spring One).@shakuzen : please give me your best email so I can add you as well.

[shakuzen]

@reshmik I'm at SpringOne this week also, so we could all gather for a quick hacking and testing session. I also have my local setup with amqps now for testing. I have another question now: are you making a custom Zipkin server so you can extract the properties from vcap services and set them for the corresponding Zipkin properties, or how are you getting them set?

[gerard-br-dge]

Hi I'm raising this question rather than create a new feature request since it looks like it is in the same area. Apologies if this is the wrong place.

I am trying to integrate zipkin into a RabbitMQ configured with Mutual TLS. The above describes some updated support where zipkin uses the dummy 'trust everything' TrustManager, which is fine. My problem is that the current approach, I think, is hard coded to set up a dummy/empty KeyStore.

This means that our RabbitMQ rejects the connection because it doesn't see an acceptable client certificate coming from Zipkin client.

Is it possible to update the ZipkinRabbitMQCollectorProperties or equivalent to optionally create an SSLContext that uses the dummy trust manager, but allows a 'real' KeyStore to be configured? and then call connectionFactory.useSslProtocol, specifying the SSLContext just created? Or some other mechaism to allow a KeyStore to be injected?

I tried setting JAVA_OPTS to default to a keystore file, but it is just ignored by the hard-coded logic

If there is another approach or workaround already in use I'd be grateful to hear about it