Skip to content
This repository has been archived by the owner on May 7, 2020. It is now read-only.

MqttEmbeddedBroker IO extension #3910

Merged
merged 1 commit into from
Apr 27, 2018

Conversation

davidgraeff
Copy link
Contributor

@davidgraeff davidgraeff commented Jul 30, 2017

This is an extension to PR #3839 (Mqtt Broker configuration).

Abstract

Mqtt can be seen as an important IoT protocol next to CoAP but with different goals and a different architecture. CoAP is supported within ESH with the Californium library and at the moment bindings just use the raw library without any transport abstraction. Which is fine for CoAP. Mqtt on the other side needs a central additional component, the Mqtt broker. This binding lowers the entry barrier into the Mqtt world and allows to start a secure, correctly configured Mqtt server with a button click.

Features

This bundle contains an embedded broker (Moquette) and will create a MqttBrokerConnection instance shared with the MqttService so that other bindings, especially the MQTT Broker Binding can use the Broker.

Plain tcp and also secure ssl connection (with #4105) are possible for external clients, depending on the configuration.

The service is configured via ConfigAdmin but implements ConfigurableService to be configurable via PaperUI as well.

3rd party licenses

The Moquette server is licensed as EPL 1.0 / Apache 2.0, the required jetty tcp server is licensed as Apache 2.0.

@davidgraeff davidgraeff changed the title MqttEmbeddedBroker binding [Feedback required] MqttEmbeddedBroker binding Jul 30, 2017
@kaikreuzer
Copy link
Contributor

Hi @davidgraeff!

This is a feature I was hoping for since a long while 👍
For all MQTT-enabled devices, it is really nice when ESH comes with an embedded broker instead of requiring a separate instance somewhere.

So regarding your discussion point: I would clearly say that this can go into ESH, especially as Moquette once almost managed to become an Eclipse project as well.

I wouldn't call this a "binding" through (this is where you have currently put it). I'd think the better place would be https://github.com/eclipse/smarthome/tree/master/extensions/io, i.e. calling it "some I/O related extension".

@davidgraeff davidgraeff force-pushed the mqtt_embedded branch 2 times, most recently from 259b1d5 to 684644a Compare August 15, 2017 14:36
@davidgraeff davidgraeff changed the title [Feedback required] MqttEmbeddedBroker binding MqttEmbeddedBroker IO extension Aug 16, 2017
@davidgraeff davidgraeff mentioned this pull request Aug 22, 2017
@davidgraeff
Copy link
Contributor Author

@kaikreuzer Just to be on the same side here:

  • The embedded broker will use ConfigurableServices so that it can be enabled/disabled via the GUI. It will add its connection to the MqttService.
  • The handler and discovery part are removed.

@kaikreuzer
Copy link
Contributor

Sound good 👍

@sjsf
Copy link
Contributor

sjsf commented Oct 4, 2017

I might be a little late to the party, but hopefully you don't mind asking this question anyway: Why exactly do you all think it makes sense to have an embedded MQTT server within ESH? I mean, there are brilliant, easy-to-setup standalone brokers out there, working nicely as a standalone service. Isn't the approach of embedding it pretty much against the micro service idea of doing one thing (and doing it good) as opposed to the old JEE world of putting everything into the same process?
Don't get me wrong, it certainly makes sense to have good MQTT support within ESH, but ESH becoming also a MQTT server goes a little too far, IMHO. Hence I'm trying to understand the reasoning for this proposal...

@davidgraeff
Copy link
Contributor Author

I agree that an Mqtt server should not be part of the core and nothing should rely on this and it should not run in the same process. This is why this bundle is as independent as possible and the MQTT server runs in its own process.

Because Mqtt is a key protocol and only works with a server, it might be a good idea to provide a out of the box working a good solution here. Mqtt can be used for ESH instance sycronisation for example. I see this bundle as extension. like any other extensions, it is discussable if it should be in the core repository.

@kaikreuzer
Copy link
Contributor

there are brilliant, easy-to-setup standalone brokers out there, working nicely as a standalone service.

But that is the problem: These are external services that you need to setup and configure accordingly. You cannot easily build integrations using MQTT as you are always just a client, but not the server. Like supporting sensors that use MQTT and that need to be told where to connect to (=the ESH instance).

Isn't the approach of embedding it pretty much against the micro service idea

Microservices in an OSGi sense are services WITHIN the same JVM - and that's the approach here.
We are definitely not doing micro-services on a process level, otherwise we should also consider moving out the HTTP server, the event admin and having all bindings as separate processes.

Note that the feature discussed here is merely to wrap an existing broker, not to implement any on our own. And as @davidgraeff says, it is a fully optional component that nobody is required to use - but imho it can be tremendously helpful for many.

Copy link
Contributor

@kaikreuzer kaikreuzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @davidgraeff,

What is the status of this PR? Are you waiting for any discussions/decisions? Imho, this PR should be pretty unrelated to most of the rest and mergable on its own, do you agree?

*
* @author David Graeff - Initial contribution
*/
@Component(immediate = true, configurationPid = { "mqttembeddedbroker" }, property = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pid should always be prefixed by "org.eclipse.smarthome."

<artifactId>org.eclipse.smarthome.io.mqttembeddedbroker</artifactId>
<version>0.9.0-SNAPSHOT</version>

<name>MqttBroker Binding</name>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a binding.
Change to "Embedded MQTT Broker"

@@ -0,0 +1,10 @@
package org.eclipse.smarthome.io.mqttembeddedbroker.internal;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license header missing

@@ -0,0 +1,10 @@
package org.eclipse.smarthome.io.mqttembeddedbroker.internal;

public class ServiceConfiguration {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

javadoc missing

@@ -0,0 +1,32 @@
# MqttEmbeddedBroker Binding
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a binding.
Change it to "Embedded MQTT Broker"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidgraeff ping :-)

* __secure__: If set, hosts a secure SSL connection on port 8883 or otherwise a non secure connection on port 1883 (if not overwritten by the port parameter).
* __persistence_file__: The file were messages are stored for a persistence feature. Can be empty. Defaults to _moquette.msgs_.

## Channels
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove


There are no channels available. The status message and possible reasons for a failed connection can be accessed through thing properties. The broker connection is successful if the thing is ONLINE.

## Full Example
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Fragment-Host: org.eclipse.smarthome.io.mqttembeddedbroker
Import-Package: org.eclipse.smarthome.core.common.registry,
org.eclipse.smarthome.core.events,
org.eclipse.smarthome.core.thing,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be no dependency on org.eclipse.smarthome.core.thing

org.eclipse.smarthome.config.core,
org.eclipse.smarthome.config.discovery,
org.eclipse.smarthome.core.library.types,
org.eclipse.smarthome.core.thing,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be no dependency on org.eclipse.smarthome.core.thing.*

org.eclipse.jdt.annotation;resolution:=optional,
org.osgi.service.component.annotations;resolution:=optional,
org.eclipse.smarthome.config.core,
org.eclipse.smarthome.config.discovery,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be no dependency on org.eclipse.smarthome.config.discovery.

@davidgraeff
Copy link
Contributor Author

@kaikreuzer I'm in Japan and cannot contribute code at the moment. All my PRs are on hold.

@kaikreuzer
Copy link
Contributor

I assume you are back, at least I have seen you recently ;-)
Any chance to incorporate the feedback? Would be cool to get that merged some day!

@kaikreuzer
Copy link
Contributor

@davidgraeff Friendly ping!

@kaikreuzer
Copy link
Contributor

@davidgraeff Another friendly ping.
In case you do not plan to finish this, please state so as well, so that we know that somebody else would have to take this over. But as it is almost done and just a few small changes are required for the merge, I very much hope that you can find the time for it.

@davidgraeff
Copy link
Contributor Author

Unfortunate timing that we didn't finish those PRs before I went for another traveling journey but I'm back end of the week.

@kaikreuzer
Copy link
Contributor

@davidgraeff Did you find any time for it?

@kaikreuzer
Copy link
Contributor

@davidgraeff Another friendly ping!

@kaikreuzer
Copy link
Contributor

@davidgraeff As you are active tonight: Any chance to finally address this?

@davidgraeff
Copy link
Contributor Author

It's a 12h/d work week atm, but yes I want this to be included as well. I just first addressed all OpenHab issues and PRs on my todo list.

@davidgraeff davidgraeff force-pushed the mqtt_embedded branch 3 times, most recently from 8787bd6 to 17e3fe1 Compare March 17, 2018 12:19
@davidgraeff
Copy link
Contributor Author

davidgraeff commented Mar 17, 2018

All issues addressed and license headers are updated. PR depends on #4173. Does not depend on any SSL service implementation yet. I have commented out the affected code parts, but I'd like to keep them until #4105 is resolved.

Copy link
Contributor

@htreu htreu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments inline.

@@ -0,0 +1,32 @@
# MqttEmbeddedBroker Binding
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidgraeff ping :-)

@@ -0,0 +1,32 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop about.html, replace by NOTICE file.

OSGI-INF/,\
ESH-INF/,\
lib/*.jar,\
about.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace with NOTICE

*/
public void stopBrokerStartDetection() {
if (schedule != null) {
schedule.cancel(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should schedule also be set to null here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't hurt, I've added that

* @author David Graeff - Initial contribution
*/
public class MqttEmbeddedBrokerMetrics implements InterceptHandler {
public interface BrokerMetricsListener {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please JavaDoc the interface and method.

*
*/
public interface MqttEmbeddedBrokerStartedListener {
public void mqttEmbeddedBrokerStarted(boolean timeout);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add JavaDoc for the method.

public Cluster getCluster() {
return null;
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing new-line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@davidgraeff
Copy link
Contributor Author

@htreu Done

Bundle-Name: MqttBroker Binding Tests
Bundle-SymbolicName: org.eclipse.smarthome.io.mqttembeddedbroker.test;singleton:=true
Bundle-Vendor: Eclipse.org/SmartHome
Bundle-Version: 0.9.0.qualifier
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nedds to be updated

Bundle-Name: Embedded Mqtt Broker
Bundle-SymbolicName: org.eclipse.smarthome.io.mqttembeddedbroker;singleton:=true
Bundle-Vendor: Eclipse.org/SmartHome
Bundle-Version: 0.9.0.qualifier
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nedds to be updated

org.eclipse.smarthome.io.transport.mqtt,
org.eclipse.smarthome.io.transport.mqtt.reconnect,
org.eclipse.smarthome.io.transport.mqtt.sslcontext,
org.osgi.framework;version="1.8.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version constraint needs to be removed.

org.eclipse.smarthome.io.transport.mqtt.sslcontext,
org.osgi.framework;version="1.8.0",
org.osgi.service.component,
org.osgi.service.component.annotations;resolution:=optional,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

org.osgi.service.component,
org.osgi.service.component.annotations;resolution:=optional,
org.slf4j
Comment: TODO Add org.eclipse.smarthome.io.net.security.api for secure connections
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be placed in the manifest if in production (merged)

@@ -20,6 +20,7 @@ Import-Package:
org.eclipse.smarthome.io.transport.mqtt.reconnect,
org.eclipse.smarthome.io.transport.mqtt.sslcontext,
org.osgi.framework,
org.osgi.service.cm;version="1.5.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove the version restriction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I teach Eclipse not to add these automatically? Would be awesome

Copy link
Contributor

@htreu htreu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only one last minute missing NOTICE file.
Otherwise this looks good to me. After approval we need to file a CQ since this exceeds 1k lines.
Also moquette and netty need CQs.

src/test/resources/
output.. = target/test-classes
bin.includes = META-INF/,\
.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, didn't notice before: NOTICE file is missing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@htreu
Copy link
Contributor

htreu commented Apr 6, 2018

Thanks @davidgraeff! I suggest to freeze this now and file the appropriate CQs. I will do so as soon as privileges are granted to my by eclipse.org.

@davidgraeff
Copy link
Contributor Author

All right.
@kaikreuzer need to change his "requested changes" flag to make it all green ^^

@htreu
Copy link
Contributor

htreu commented Apr 9, 2018

CQs created for

  • moquette-broker-0.10: CQ16090
  • netty-all-4.1.19.Final: CQ16091 (PB CQ15280)

@htreu
Copy link
Contributor

htreu commented Apr 9, 2018

@davidgraeff since netty-4.1.19 was the most recent version approved by the eclipse IP team please update the version used here to 4.1.19 (instead of 4.1.13). The netty change log lists a lot of bug fixes for this version bump.
Thanks.

@davidgraeff
Copy link
Contributor Author

Moquette 0.10 has some memory leak issues (https://github.com/andsel/moquette/issues/385). It should be upgraded as well as soon as 0.11 is out.

@htreu
Copy link
Contributor

htreu commented Apr 9, 2018

Thanks @davidgraeff. Lets first see what the eclipse IP team can make out of the 0.10 version.

FYI: netty CQ16091 is already approved.

@maggu2810
Copy link
Contributor

Moquette 0.10 has some memory leak issues (https://github.com/andsel/moquette/issues/385). It should be upgraded as well as soon as 0.11 is out.

Thanks for pointing that out. Adding something that is already known to eat memory sounds a little bit weird.
Do you know how much memory is leaked and under which situations (e.g. does it depend on uptime only or on the number of published messages, ...)?
Do we need some documentation that it should not be used in production?

@davidgraeff
Copy link
Contributor Author

@maggu2810 Netty is just used to convert the received/published messages so publishing/receiving causes memory leaks and not concurrent connections.

@htreu
Copy link
Contributor

htreu commented Apr 20, 2018

CQ16090 for moquette is still not approved but we have clearance to proceed with the checkin. I will start a last review before issuing a CQ for the actual code.

Copy link
Contributor

@htreu htreu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @davidgraeff I found some remaining issue. See below and inline. Thanks.

Please remove the lib/moquette-broker-src-0.10.zip since we have a binary-only CQ approved.

When I finally got this to run the following log entries appear:

08:34:17.891 WARN  i.m.s.DefaultMoquetteSslContextCreator[:54] - The keystore path is null or empty. The SSL context won't be initialized.
08:34:17.891 ERROR i.m.server.netty.NettyAcceptor[:164] - Can't initialize SSLHandler layer! Exiting, check your configuration of jks

which leads to a stopped moquette broker.

@@ -0,0 +1,20 @@
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MqttBroker Binding Tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one obsolete " " here

<bundle.namespace>org.eclipse.smarthome.io.mqttembeddedbroker</bundle.namespace>
</properties>


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty line

@@ -0,0 +1,144 @@
/**
* Copyright (c) 2010-2017 by the respective copyright holders.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please update all license headers to EPL 2
Use the command mvn license:format -pl extensions/io/org.eclipse.smarthome.io.mqttembeddedbroker.test from the project root.

public class MqttEmbeddedBrokerServiceTest {

// Create an accept all trust manager for a client SSLContext
private static class X509ExtendedTrustManagerEx extends X509ExtendedTrustManager {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this class seems to be unused right now, please add a @SuppressWarnings("unused") so the whole test project is free of warnings.

}
}

EmbeddedBrokerService subject;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be private

*
* @author David Graeff - Initial contribution
*/
@Component(immediate = true, service = EmbeddedBrokerService.class, configurationPid = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configurationPid must not be a String[]

public MqttBrokerConnection connection;

@Reference(cardinality = ReferenceCardinality.MANDATORY)
public void setMqttService(MqttService service) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also add the unbind method

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought, an unbind method is not required if ReferenceCardinality.MANDATORY is set, because the entire service is teared down if the mandatory referenced service died.

Bundle-Vendor: Eclipse.org/SmartHome
Bundle-Version: 0.10.0.qualifier
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Bundle-ClassPath: ., lib/moquette-broker-0.10.jar, lib/netty-all-4.1.13.Final.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its netty 4.1.19 now

.,\
OSGI-INF/,\
ESH-INF/,\
lib/*.jar,\
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the moquette source zip removed this can be changed to lib/


This bundle allows to configure and start an embedded MQTT Server (based on (Moquette)[https://github.com/andsel/moquette]) for an easy way to have an MQTT Server up and running with a click.

## Supported Things
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bundle now adds a service, please update the docs accordingly.

Copy link
Contributor

@htreu htreu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @davidgraeff. I will create the corresponding CQ. Please do not change this PR anymore.

This io/extension can add a Mqtt broker connection and start an embedded broker (Moquette).
It allows secure connection via a provided and configured certificate or creates
a self-signed certificate and key on the first start otherwise.

Signed-off-by: David Graeff <david.graeff@web.de>
@htreu
Copy link
Contributor

htreu commented Apr 23, 2018

FTR: all commits squashed and CQ16166 created.

edit: link removed.

@htreu htreu added the CQ label Apr 23, 2018
@htreu
Copy link
Contributor

htreu commented Apr 27, 2018

CQ16166 got approved. Ready to go.

@htreu htreu removed the CQ label Apr 27, 2018
@htreu htreu merged commit 592b52c into eclipse-archived:master Apr 27, 2018
@davidgraeff davidgraeff deleted the mqtt_embedded branch April 27, 2018 08:26
ermartens pushed a commit to ermartens/smarthome that referenced this pull request Jun 15, 2018
This io/extension can add a Mqtt broker connection and start an embedded broker (Moquette).
It allows secure connection via a provided and configured certificate or creates
a self-signed certificate and key on the first start otherwise.

Signed-off-by: David Graeff <david.graeff@web.de>
@htreu htreu added this to the 0.10.0 milestone Oct 30, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants