MQTT gateway doesn't find server cert/key on first startup with no local images pulled yet

[ppatierno]

While deploying EnMasse on OpenShift with images that aren't already on the local PC (so a long pull from Docker Hub is needed), the MQTT gateway fails its first start with following error :

10:35:53.793 [vert.x-eventloop-thread-0] INFO enmasse.mqtt.MqttGateway - Starting MQTT gateway verticle...
10:35:53.795 [vert.x-eventloop-thread-0] INFO enmasse.mqtt.MqttGateway - SSL/TLS support enabled key /etc/mqtt-gateway/ssl/server-key.pem cert /etc/mqtt-gateway/ssl/server-cert.pem
10:35:53.938 [vert.x-eventloop-thread-1] DEBUG enmasse.mqtt.Application - Failed to deploy verticle instance 1
io.vertx.core.VertxException: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:304)
at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:446)
at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:466)
at io.vertx.core.net.impl.NetServerBase.listen(NetServerBase.java:134)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:88)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:75)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:80)
at enmasse.mqtt.MqttGateway.bindMqttServer(MqttGateway.java:168)
at enmasse.mqtt.MqttGateway.start(MqttGateway.java:217)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:710)
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:702)
at io.vertx.core.file.impl.FileSystemImpl.readFileBlocking(FileSystemImpl.java:285)
at io.vertx.core.net.impl.KeyStoreHelper.create(KeyStoreHelper.java:96)
at io.vertx.core.net.KeyCertOptions.getKeyManagerFactory(KeyCertOptions.java:47)
at io.vertx.core.net.impl.SSLHelper.getKeyMgrFactory(SSLHelper.java:309)
at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:262)
... 15 common frames omitted
Caused by: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:706)
... 21 common frames omitted
10:35:53.939 [vert.x-eventloop-thread-1] ERROR enmasse.mqtt.Application - Could not start MQTT gateway
io.vertx.core.VertxException: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:304)
at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:446)
at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:466)
at io.vertx.core.net.impl.NetServerBase.listen(NetServerBase.java:134)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:88)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:75)
at io.vertx.mqtt.impl.MqttServerImpl.listen(MqttServerImpl.java:80)
at enmasse.mqtt.MqttGateway.bindMqttServer(MqttGateway.java:168)
at enmasse.mqtt.MqttGateway.start(MqttGateway.java:217)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:710)
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:702)
at io.vertx.core.file.impl.FileSystemImpl.readFileBlocking(FileSystemImpl.java:285)
at io.vertx.core.net.impl.KeyStoreHelper.create(KeyStoreHelper.java:96)
at io.vertx.core.net.KeyCertOptions.getKeyManagerFactory(KeyCertOptions.java:47)
at io.vertx.core.net.impl.SSLHelper.getKeyMgrFactory(SSLHelper.java:309)
at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:262)
... 15 common frames omitted
Caused by: java.nio.file.NoSuchFileException: /etc/mqtt-gateway/ssl/server-key.pem
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at io.vertx.core.file.impl.FileSystemImpl$12.perform(FileSystemImpl.java:706)

Maybe something related to the MQTT secret (with certificates) not yet created ?

[lulf]

Thea should be available at the secret creation time. This is likely a bug in the address controller.

[lulf]

@ppatierno I see the same error message. When I do 'oc rsh -c mqtt-gateway-tls ' and 'cat /etc/mqtt-gateway/ssl/server-key.pem', the key is there.

I also found the reason why this is the case. In address controller, we create the secrets in one place, and populate them in another! I don't think this is optional practice, so I will create a task for fixing that.

From what I can see, the error message is transient, in that it will restart until it works.

[ppatierno]

@lulf sorry I didn't mention that ... yes the error is transient and after some pod restarts it works

[lulf]

This has been fixed in #239, so closing issue. The mqtt-gateway seems to not get this error now.