mqtt-relay cannot connect to broker with TLS enabled and PEM certificates

[victoririzarx3d]

Hi @PatrickRitchie

I am having issues connecting the agent to a broker in my local network when using TLS with PEM certificates. I am testing with v6.4.1 on Ubuntu 20.04

My configuration for the mqtt-relay part of the agent looks like this

- mqtt-relay:
    QoS: 1
    allowUntrustedCertificates: true
    clientId: simulation
    currentInterval: 10000
    documentFormat: JSON
    port: 8883
    sampleInterval: 500
    server: 192.168.161.202
    tls:
      pem:
        certificateAuthority: /home/victor/venice_data/misc/certificates/gateway-SAF-Test-ID-2/GroupCACertificate.pem
        certificatePath: /home/victor/venice_data/misc/certificates/gateway-SAF-Test-ID-2/certificate.pem
        privateKeyPath: /home/victor/venice_data/misc/certificates/gateway-SAF-Test-ID-2/privateKey.pem
      verifyClientCertificate: false
    topicPrefix: MTConnect/Entity
    topicStructure: Entity

When the agent tries to connect, I get the following error on the logs

2024-06-18 03:19:10.4204|WARN|modules.mqtt-relay|MQTT Relay Connection Error : error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certifica>
2024-06-18 03:19:10.4204|WARN|modules.mqtt-relay|MQTT Relay Connection Error : Authentication failed, see inner exception.
2024-06-18 03:19:10.4204|INFO|modules.mqtt-relay|MQTT Relay Disconnected from External Broker (192.168.161.202:8883)

I am sure I can connect to the broker because using a python mqtt client with the same certificates and client ID I am able to connect without issues.

Looking into the code of the mqtt relay module I noticed that the part where we setup the certificates, the CA certificate is appended first and then we append the certificate .

MTConnect.NET/agent/Modules/MTConnect.NET-AgentModule-MqttRelay/Module.cs

Lines 127 to 140 in 0a09f5f

	// Set TLS Certificate
	if (_configuration.Tls != null)
	{
	var certificateResults = _configuration.Tls.GetCertificate();
	if (certificateResults.Success && certificateResults.Certificate != null)
	{
	var certificateAuthorityResults = _configuration.Tls.GetCertificateAuthority();
	var certificates = new List<X509Certificate2>();
	if (certificateAuthorityResults.Certificate != null)
	{
	certificates.Add(certificateAuthorityResults.Certificate);
	}
	certificates.Add(certificateResults.Certificate);

Further down in the code, however, the certificate validation handler is declared and that uses again the CA certificate for the validation.

MTConnect.NET/agent/Modules/MTConnect.NET-AgentModule-MqttRelay/Module.cs

Lines 150 to 171 in 0a09f5f

	#if NET5_0_OR_GREATER
	// Setup CA Certificate
	if (certificateAuthorityResults.Certificate != null)
	{
	tlsOptionsBuilder.WithCertificateValidationHandler((certContext) =>
	{
	var chain = new X509Chain();
	chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
	chain.ChainPolicy.RevocationFlag = X509RevocationFlag.ExcludeRoot;
	chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;
	chain.ChainPolicy.VerificationTime = DateTime.Now;
	chain.ChainPolicy.UrlRetrievalTimeout = new TimeSpan(0, 0, 0);
	chain.ChainPolicy.CustomTrustStore.Add(certificateAuthorityResults.Certificate);
	chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
	// convert provided X509Certificate to X509Certificate2
	var x5092 = new X509Certificate2(certContext.Certificate);
	return chain.Build(x5092);
	});
	}
	#endif

On the MQTTnet repo I cannot find exact references to how this setup is done but I imagine the CA certificate, being in that list first, will fail to validate on the broker side since it is not the "main" certificate.

As an experiment I removed the appending of the CA certificate from the certificate lists and now the relay is able to connect to the broker without an issue. Is this something that makes sense? I am not an expert on this so I might be doing something wrong.

Side notes

Exception logging

I noticed that when the agent was throwing exceptions I was seeing log lines with the following

2024-06-18 03:19:10.4204|WARN|modules.mqtt-relay|MQTT Relay Connection Error : Authentication failed, see inner exception.

When the exceptions are handled then only the last message is printed in the main connection loop, but it will not print any inner exceptions

MTConnect.NET/agent/Modules/MTConnect.NET-AgentModule-MqttRelay/Module.cs

Lines 217 to 231 in 0a09f5f

	catch (Exception ex)
	{
	Log(MTConnectLogLevel.Warning, $"MQTT Relay Connection Error : {ex.Message}");
	}
	finally
	{
	if (_documentServer != null) _documentServer.Stop();
	}
	Log(MTConnectLogLevel.Information, $"MQTT Relay Disconnected from External Broker ({_configuration.Server}:{_configuration.Port})");
	await Task.Delay(_configuration.ReconnectInterval, _stop.Token);
	}
	catch (TaskCanceledException) { }
	catch (Exception) { }

I have added some calls to exception.getBaseException() and checking for inner messages to aid on the debugging, something like

Log(MTConnectLogLevel.Warning, $"MQTT Relay Connection Error : {ex.GetBaseException().Message}")

Maybe is worth to add some of this inner exception logging or logging the full traceback?

VerifyClientCertificate option

This option allows untrusted certificates to be handled, and that is setup as expected. There are a couple of options that might be relevant to set when this parameter is set in the configuration, WithIgnoreCertificateRevocationErrors() and WithIgnoreCertificateChainErrors(). Would this be relevant?

[PatrickRitchie]

Thanks for the information. This looks like it may be related to Issue #57 as well. I am working on getting an ubuntu environment setup to test these issues and it looks like you may be correct that the order of the certificates may be the issue.

If you are interested, you can make a Pull Request with the logging updates. Otherwise I will add what you have in the comments.

[victoririzarx3d]

Perfect, thanks for your help!

Yes, I can create a PR with the logging updates.

[PatrickRitchie]

After some testing, using AWS IoT as a broker, I found that adding a CA cert in Windows is ignored and can connect but when run on Ubuntu I got the same error you saw. When I removed the CA cert, I was able to connect on both Windows and Ubuntu as you described as well.

I found a related Issue on the MQTTnet project that seems to point to the CA cert not being installed dotnet/MQTTnet#757.

I tried installing it on my instance of Ubuntu but still got the same errors so I'm not confident that is the entire solution yet.

I will keep this issue open and see if I can find anything else that will allow a CA cert to be added to the chain correctly.

[victoririzarx3d]

I think I forgot to mention but I am also working with an AWS IoT broker that is running on a windows machine on my local network.

Have you tried to , instead of not appending the ca cert to the certificates list, appending it after the main certificate? Initially that is what I did and that worked also.

When I was doing my testing I also bumped with suggestions similar to the one you pointed out on MQTTnet thread. In particular this part where they set the CertificateValidationHandler to return true always, and only appending the client certificate. I guess that is equivalent to allowing untrusted certificates?

[PatrickRitchie]

I did move the order around while testing but I may go back and try again. I also found a few more examples on the MQTTnet repo https://github.com/dotnet/MQTTnet/blob/e18a91a4b59bc312aa9acf9be575401d07b793e4/Samples/Client/Client_Connection_Samples.cs#L440

Yes I see most of the examples where CertificateValidationHandler just returns true with a note to "not use this for production" but not really any examples of how to properly validate the cert. I did find a built in method https://github.com/dotnet/MQTTnet/blob/master/Source/MQTTnet/Client/Options/MqttClientDefaultCertificateValidationHandler.cs but I was still getting similar errors when using it.

[victoririzarx3d]

So it seems, from the examples, that the main certificate is imported with importFromPem, which I guess is equivalent to appending it to the certificate list on the agent, and then uses explicitely the CA Cert for the trust chain. Why would one want to have a list of certificates instead of only one? Are there cases where we have more than one certificate and a CA+privateKey?