v6.4.0: Cannot connect to local Greengrass broker (moquette)

[jaxer]

Here is an agent config:

devices: devices

modules:
    - mqtt-relay:
          server: host.docker.internal
          port: 8883
          clientId: alex-printer-8
          certificateAuthority: certificates/certificateAuthority.pem
          pemCertificate: certificates/certificate.pem
          pemPrivateKey: certificates/privateKey.pem
#          Uncommenting this does not change much
#          allowUntrustedCertificates: true

observationBufferSize: 150000
assetBufferSize: 1000

# Sets whether the Agent buffers are durable and retain state after restart
durable: true

defaultVersion: 2.2

Starting agent with docker compose up compose config:

version: "3.8"

services:
    agent:
        image: trakhound/mtconnect.net-agent:6.0.9-beta-alpine-3.18-amd64
        volumes:
            - ./agent.config.yaml:/app/agent.config.yaml
            - ./devices:/app/devices
            - ./certificates:/app/certificates

Output on console:

⋊> docker compose up                                                                                                                   
[+] Running 1/0
 ⠿ Container mtconnectnet-agent-1  Created                                                                                                                        0.0s
Attaching to mtconnectnet-agent-1
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | Copyright 2023 TrakHound Inc., All Rights Reserved
mtconnectnet-agent-1  | MTConnect.NET Agent : Version 6.0.9.0
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | This application is licensed under the MIT License (https://choosealicense.com/licenses/mit/)
mtconnectnet-agent-1  | Source code available at Github.com (https://github.com/TrakHound/MTConnect.NET)
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | 2024-03-05 11:50:03.2538|INFO|application|Configuration File Read Successfully from: /app/agent.config.yaml
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3169|INFO|agent|Loading Observations from File Buffer...
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3461|INFO|agent|15 Observations Loaded from File Buffer in (0.025s)
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3476|INFO|agent|Loading Assets from File Buffer...
mtconnectnet-agent-1  | 2024-03-05 11:50:03.6038|INFO|application|Module Loaded : MQTT Relay
mtconnectnet-agent-1  | 2024-03-05 11:50:03.7549|INFO|agent|Device (M12346) Read From File : /app/devices/test.xml
mtconnectnet-agent-1  | 2024-03-05 11:50:04.1271|WARN|modules.mqtt-relay|MQTT Relay Connection Error : Authentication failed, see inner exception.
mtconnectnet-agent-1  | 2024-03-05 11:50:04.1406|INFO|modules.mqtt-relay|MQTT Relay Disconnected from External Broker (host.docker.internal:8883)
^CGracefully stopping... (press Ctrl+C again to force)

Logs from greengrass:

2024-03-05T11:50:04.109Z [ERROR] (nioEventLoopGroup-5-4) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.moquette.broker.metrics.BytesMetricsHandler.channelRead(BytesMetricsHandler.java:51)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:294)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1548)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1394)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDe
coder.java:449)
	... 25 more

2024-03-05T11:50:04.110Z [INFO] (nioEventLoopGroup-5-4) io.moquette.b
roker.metrics.MQTTMessageLogger: Channel Inactive. {}

I am guessing it's something about MQTT library configuration, as I can connect to same broker with same certificates using MQTT Explorer:

 

In which case, Greengrass logs look like:

2024-03-05T11:56:37.269Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2024-03-05T11:56:37.877Z [INFO] (nioEventLoopGroup-5-1) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=alex-printer-8, sessionId=2bfac112-3e14-4e25-9996-eeb5802dca9b}
2024-03-05T11:56:37.893Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:56:37.894Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=$SYS/#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:56:37.895Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=$aws/#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:57:20.889Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B DISCONNECT <alex-printer-8>. {}
2024-03-05T11:57:20.891Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: Channel Inactive. {}

[jaxer]

I think there is not much about greengrass configuration, I am using mostly default settings.

Please let me know if I can provide more info or can try it out on some test build of a docker image.

[PatrickRitchie]

Thanks for the information. Are you able to connect using MQTT Explorer from inside a Docker container? I'm wondering if that might be causing an issue. Have you installed the certificate in the Docker container? I'm not an expert on certificates but I get similar errors if I don't have the client certificate installed when running on Windows.

[jaxer]

Thanks for quick replys. Thats an interesting tought. I am also not an expert on certificates.
I think AWS root certificates should be trusted by modern systems by default. But I still tried to do somthing similar to this (took AWS IoT root CA for that) with no noticable changes.

Here is what I tried after that (with same setup as in description):

docker compose run --entrypoint sh agent
apk add mosquitto-clients
mosquitto_sub -t "#" -h host.docker.internal -p 8883 --cafile certificates/certificateAuthority.pem --cert certificates/certificate.pem --key certificates/privateKey.pem -i alex-printer-8

That connected sucessfully and greengrass logs are:

2024-03-06T10:59:50.726Z [INFO] (nioEventLoopGroup-5-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2024-03-06T10:59:51.308Z [INFO] (nioEventLoopGroup-5-3) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=alex-printer-8, sessionId=20447946-fe71-49cf-8069-9732dcb015f1}
2024-03-06T10:59:51.311Z [INFO] (nioEventLoopGroup-5-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}

Running dotnet agent.dll right after that on same instance of container produces exactly same result as in description.
So I am guessing that it's not related to system certificates chain.

[PatrickRitchie]

I just created a new release (v6.2.0) that may fix some of the issues you were having. I made a number of changes to configuring TLS for both HTTP and MQTT. Note that the configuration file format for setting certificate paths has changed to the below:

- mqtt-relay:
    server: localhost
    port: 8883
    clientId: mtconnect-test # Set the ClientId to the AWS Thing ID
    tls:
      pem:
        certificateAuthority: certs/AmazonRootCA1.pem
        certificatePath: certs/2316549874654321654984984158961634984794-certificate.pem.crt
        privateKeyPath: certs/2316549874654321654984984158961634984794-private.pem.key
    documentFormat: json-cppagent
    currentInterval: 5000
    sampleInterval: 500

There may be some work left to do so let me know if you are still having issues with it.

[jaxer]

Sorry for taking it so long to re-test.

Unfortunately the issue still remains for me.
Same exception with latest 6.4.0:

2024-05-14T15:30:06.497Z [ERROR] (nioEventLoopGroup-3-25) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.moquette.broker.metrics.BytesMetricsHandler.channelRead(BytesMetricsHandler.java:51)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:294)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	... 25 more

I could help with reproducing the issue by putting Greengrass gateway into public and giving you temporary certificates to connect with. Let me know if that would help. Our instance of Greegrass gateway is nothing special, it runs default Moquette broker and does not do any magic with root certs (using default AWS root certs).

Would be great to have this fixed as that will allow us to use MTConnect.NET agent in emulators (linux/docker based) simplifying MTConnect integration.

I think issue only present on non-windows platforms.