Feature Request: Log TLS Session Keys

[BHillam-digEcor]

When developing applications interacting with mosquitto it can be useful to get a Wireshark capture of the traffic, but when TLS is used in conjunction with an ephemeral cipher suite, Wireshark cannot decrypt the session unless the session keys are logged to a file. Chrome and Firefox have the option of doing this for HTTP (if the environment variable SSLKEYLOGFILE is defined, they will log to that location). See the Wireshark documentation on Using the (Pre)-Master-Secret.

Mosquitto uses OpenSSL, which should provide a mechanism for retrieving the necessary information, in order to log it to a file. This request is to implement a similar feature to that provided by Chrome & Firefox when the mosquitto broker has been compiled with TLS support enabled, enabled by a config file setting.

[toast-uz]

Did you try current logging features? You can get a lot of information from the debug logs. Full browsers like Chrome and Firefox are one of the richest application in the world. Comparing Mosquitto with them seems too rough.

[cjmaynard]

For our application, we are not encrypting the MQTT traffic yet, but we soon will, so this topic interests me quite a bit. I don't think the intention was to directly compare mosquitto with Chrome or Firefox, but certainly they could serve as some inspiration with regard to key logging, no?

How do folks deal with troubleshooting encrypted MQTT traffic today? Without being able to decrypt the traffic, I imagine the answer is probably "not very well". In any case, if mosquitto could follow the example of what Chrome and/or Firefox provide, it would probably make it a lot easier (or possible) to support decryption of the MQTT traffic in Wireshark.

That said, being that we haven't begun to encrypt the traffic, I am personally unfamiliar with what information might be provided in the current log files, so maybe there is already a mechanism provided whereby the session keys are logged somewhere? If there is, then perhaps it's sufficient, but if not, then please consider implementing this feature request - if not in the same manner by which Chrome and Firefox provide it - then by whatever means is feasible.

[karlp]

You test the encryption by itself, and then it just works, and you don't need to decrypt wireshark. Surely you wouldn't be trying to debug your application layer at the same time as the security transport?

[toast-uz]

@karlp is right.

@BryceHillam @cjmaynard, I guess the good way is to improve current debug logs whether encrypted or not. We need not decrypt encrypted MQTT traffic for troubleshooting except for TLS negotiations which is depend on OpenSSL logs. Therefore, I want to know the specific situation of your issue.

[cjmaynard]

We don't have the luxury of always being able to remotely access all our devices and read logs from them or capture locally at the remote device before the payload is encrypted. Sometimes the only way to tell what's going on is by looking at the packets, and Wireshark provides excellent filtering capabilities to be able to focus in on a small subset of packets of interest from the enormous flood of packets some of us have to deal with. Application developers often blame the network guys for problems, and sometimes the only way to exonerate the network stack is to show them the packets. There are sometimes other ways of course, but as they say in packet analysis, "The packets don't lie". I will always believe packets from a capture file over any application's debug log, for example.

Being able to decrypt the traffic has educational benefits as well, as it teaches new developers, Q/A personnel, etc. how things really work. There are of course many other benefits that come with being able to decrypt the traffic ... and yes, sometimes applications do break and it surely helps to be able to see the data in order to help debug them.

[berntr]

I agree with the original request from @BryceHillam, and also with @cjmaynard's comments - exactly our experience!

For an important project at our company it would be extremely helpful also. We can't afford to work unencrypted so we have to have TLS enabled all the time. When we're debugging our application code not being able to see what is being sent is a real pain!

And it does sound like a minor extension to the code base. We could probably do it ourselves, but are working on a tight deadline and we also don't really have anybody here that has the in-depth expertise that the current maintainers would have...

[BHillam-digEcor]

@toast-uz Our application is a microcontroller running MQTT + TLS with limited resources - a lot of things can go wrong in unexpected ways, and we need to see the decrypted application data in order to debug these issues (we use a different MQTT client on the embedded device, which is not a mature project, so there is some doubt whether or not this is behaving as expected when coupled with TLS).

Currently we log the session ID and keys on the embedded device to our UART debug port, but transferring these keys to the machine running the Wireshark capture for every debug session is tedious.

[toast-uz]

I have two questions:

1. How could not the current debug log of mosquitto help your issue? Basically, I guess the good way is to improve current debug logs. So I want to know it.
2. Any example of SSLKEYLOGFILE implementation on servers? I guess SSLKEYLOGFILE is a feature for clients which cannot get server secret keys. Because, the mechanism of SSLKEYLOGFILE seems to create log for each TLS session key exchange, and impacts significantly on server resources. Actually, current implementations seem only on browser clients.

[BHillam-digEcor]

@cjmaynard said it best - Wireshark is unparalleled in its ability to filter a large set of traffic down to quickly find the source of a problem. If we were to use the logs provided by Mosquitto, then lengthy manual sifting would be required.

I've not yet come across a server-side implementation of this feature. We do have access to the private keys of the server but would prefer to not use them since this risks copies of the keys being stored on developer machines and forgotten about.

The format of the file expected by Wireshark is not thoroughly documented by Wireshark (the standard seems to have been set by Mozilla and Wireshark are simply following their lead).

This is what I am currently doing, as described by this post, which works when combined with the server's private key
RSA Session-ID:<hex string> Master-Key:<hex string>
However, Chrome and Firefox both append to the log lines like this:
CLIENT_RANDOM <hex string>
I'm not sure if CLIENT_RANDOM is sufficient information for Wireshark to decrypt the TLS traffic.

Currently, the alternatives for achieving this are to use LD_PRELOAD (Linux only) to load a version of the OpenSSL binaries that exposes the required information, or attach a debugger to read them from memory.

[toast-uz]

I've never denied Wireshark has much features better than mosquitto's. I'd like to keep mosquitto lightweight and limit its new features. Therefore, don't explain how the proposed feature is great, but explain how common compared with the other reference (lightweight network server) projects.

[stig-bjorlykke]

Because mosquitto is a great tool for debugging it would be a really nice feature to be able to dump the SSLKEYLOGFILE for Wireshark analysis. Currently my debug log shows only 'sending CONNECT', which is not giving me any hints whats going on. And as @cjmaynard writes I'm not able to read the logs at the remote side.

[hablutzel1]

This is apparently already implemented as mentioned in #2266 (comment).

[ralight]

This is in the develop branch, as per the previous comment, and will be in 2.1.