MQTT Transport Implementation #48

Venkat2811 · 2016-10-14T12:36:08Z

Fix for #54.

Added EdgeSystemIdentity and RemoteSystemIdentity classes.
Changed DccComms.send(message) to DccComms.send(message, mess_attr)

MQTT Transport Implementation :

TLS Implementation
Username/Password based authentication
QoS 0, 1, 2 support
Mqtt Device and Dcc comms
Connect-Disconnect-Timeout mechanism
Example with reference to Vaibhav's example.
Added MQTTMessagingAttributes
AWSIoT will be a separate PR.

* MQTT Transport Implementation : - TLS Implementation - Username/Password based authentication - QoS 0, 1, 2 support - Adding gateway identity layer for encapsulating gateway details * Removing typo loop_forever and adding check for qos_details * Adding default parameters * MQTT Transport Example - Example for streaming data from device to IoTCC data centre component using MQTT channel - Adding device comm for MQTT - Updates in sampleProp.conf for MqttChannel which needs to be subscribed * Updated MQTT Example - Print statements removed * - Modifying gateway name as edge system - Description for Paho network loop method in MQTT example

KohliDev · 2016-11-04T21:56:53Z

examples/iotcc_mqtt.py

+        edge_system_identity = Identity(config['cacert'], config['certfile'], config['keyfile'], config['mqtt_username'],
+                               config['mqtt_password'])
+
+        # Encapsulate TLS parameters


Refactoring change: Move the mqtt channel subscription into a separate method mqtt_subscribe() and call it from the main just for better understanding and clarity that calls are separate from Project ICE.

Unclear on this. Could you please explain ?

Should we add mqtt_subscribe() in mqtt device and DCC comms ?
2). What about publish() then ?

No need to change the implementation take out the mqtt specific code into a separate method. Let`s discuss.

KohliDev · 2016-11-04T21:58:09Z

examples/iotcc_mqtt.py

+
+
+    except RegistrationFailure:
+        print "Registration to IOTCC failed"


Format the Python code using PEP 8. Give a blank line at the end of all files.

KohliDev · 2016-11-04T22:01:32Z

examples/sampleProp.conf

+#### [MQTT] ####
+MqttChannel = "mqtt-channel"
+MqttSubChannel1 = "mqtt-subchannel1"
+BrokerIP = "127.0.0.1"


Keep it general
BrokerIP = mqtt-broker
BrokerPort = mqtt-broker-port

I'll suggest to make it a different config file as samplePropMqtt.conf rather then keeping all this parameters in the main sampleProp.conf file.

- made suggested changes

KohliDev

Ship It.

Some minor comments, please fix them.
I'll wait for one more approval before merging the changes as per the review process.

KohliDev · 2016-11-08T05:55:18Z

liota/lib/identity/edge_system_identity.py

+#  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF     #
+#  THE POSSIBILITY OF SUCH DAMAGE.                                            #
+# ----------------------------------------------------------------------------#
+


Add some comments about the class.

KohliDev · 2016-11-08T05:55:51Z

liota/lib/identity/tls_conf.py

+#  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF     #
+#  THE POSSIBILITY OF SUCH DAMAGE.                                            #
+# ----------------------------------------------------------------------------#
+


Add some comments about the class.

KohliDev · 2016-11-08T06:05:27Z

examples/mqtt/iotcc_mqtt.py

@@ -0,0 +1,178 @@
+# -*- coding: utf-8 -*-


Rename file to iotcc_simulated_mqtt.py

KohliDev · 2016-11-08T06:10:09Z

examples/mqtt/samplePropMqtt.conf

+cacert = "/etc/broker-name/ca_certificates/ca.crt"
+certfile = "/etc/certs/client.crt"
+keyfile = "/etc/certs/client.key"
+mqtt_username = ""


Please try to add example using username and password based mqtt approach as well.

This example covers both username-password and certificate based Mqtt approach. i.e Remote Mqtt broker expects certificate and password.
Missed to add dummy username and password in prop file. Will add it :)

User can chose to use either certificate based approach or username password authentication by simply switching flags. So, should we add separate examples for certificate based and username-password ?

- made suggested changes

pmasrani · 2016-11-15T06:02:08Z

examples/mqtt/iotcc_simulated_mqtt.py

+from liota.entities.metrics.metric import Metric
+from liota.dccs.dcc import RegistrationFailure
+
+import Queue


Import order seems incorrect from pep8 convention. https://www.python.org/dev/peps/pep-0008/
You can use autopep8 for formatting as per pep8 convention.

pmasrani · 2016-11-15T06:11:36Z

examples/mqtt/samplePropMqtt.conf

+BrokerIP = "mqtt-broker"
+BrokerPort = "mqtt-broker-port"
+keepalive = 60
+cacert = "/etc/broker-name/ca_certificates/ca.crt"


Should these be set in /etc/liota/conf/ or /etc/liota/mqtt/conf/ instead of /etc/ directly?

pmasrani · 2016-11-15T06:22:03Z

liota/dcc_comms/mqtt_dcc_comms.py

+    def subscribe(self, topic, qos):
+        self.mqtt_client.subscribe(topic, qos)
+
+    def send(self, message):


The reason for extracting out common DCCComms module is that all DCCComms should provide a common interface for common functionalities like connect, disconnect, send and receive. The developers should be able to change just the DCCComms declarations (or minimum code) and should be able to change the communication channel. If we provide completely independent methods for implementing publsih/subscribe for MQTT and send/receive for others, the commonality is lost. We should try to achieve the functionality from a common interface.

Agreed.

@KohliDev and I discussed mqtt topic generation from liota's entity hierarchy. Topics will be generated at Dcc.
Here are the possible topic structure for a metric in liota:

gw-uuid/device-uuid/metric-name

gw-uuid/metric-name

So, Dcc will pass all args required by mqtt.publish(topic, message, qos, retain) in a dict to send(message) in MqttDccComms. Is that okay ? By this publish functionality can be achieved from common interface.

pmasrani · 2016-11-15T06:23:34Z

liota/device_comms/mqtt_device_comms.py

+    def subscribe(self, topic, qos):
+        self.mqtt_client.subscribe(topic, qos)
+
+    def send(self, message):


pmasrani · 2016-11-15T06:32:00Z

liota/dcc_comms/mqtt_dcc_comms.py

+log = logging.getLogger(__name__)
+
+
+class MqttDccComms(DCCComms):


Is this used anywhere as of now? I guess we need to add an MQTT Broker as DCC and an example of how to use it. Should we do MQTT DCC, MQTT DCC Comms and MQTT as DCC example as part of a separate pull request. Here it is unused code.

It is part of the implementation and it`ll be leveraged in AWS example. I think @Venkat2811 is going to update that pull request.
Can we also add one separate basic example to publish data to ThingsWorx using MQTT DCC comms ?

For publishing we'll need a DCC. It could be either a generic MQTT Broker DCC or specific DCCs (AWS, ThingWorx etc.,) that leverages MqttDccComms. We've used Mqtt Transports and Mqtt broker previously in car parking example for thingworx. Connecting to AWS should also be possible.

So, a generic MQTTBroker could be implemented and we could provide examples to publish data to AWS and ThingWorx.

- Signature change `subscribe(topic, qos, callback)` - Added `loop_start` and `loop_stop` in transports - Username/Password based authentication - QoS 0, 1, 2 support - Adding gateway identity layer for encapsulating gateway details * Removing typo loop_forever and adding check for qos_details * Adding default parameters * MQTT Transport Example - Example for streaming data from device to IoTCC data centre component using MQTT channel - Adding device comm for MQTT - Updates in sampleProp.conf for MqttChannel which needs to be subscribed * Updated MQTT Example - Print statements removed * - Modifying gateway name as edge system - Description for Paho network loop method in MQTT example * Changes in example code: - Created function for MQTT connection setup - Seperate conf file for MQTT example - Code refactoring * Change in method name * - Renamed MQTT example - Added comments - Change in configuration file * Updated MQTT example configuration file * - Adding network loop methods inside MQTT transport - Subscribe method change : Adding callback as a parameter - Changes in MQTT example script according to new changes - Optimised import statements * Rolling back mistakenly placed extra line

mess_attr -> msg_attr

KohliDev · 2016-12-21T16:52:47Z

liota/lib/transports/mqtt.py

+            log.error("Connection error with result code : {0} : {1} ".
+                      format(str(self._connect_result_code), paho.connack_string(self._connect_result_code)))
+            self._paho_client.loop_stop()
+            sys.exit(0)  # successful termination


Remove sys.exit as we discussed just keep the error getting logged so that MQTT loopback works.

KohliDev · 2017-01-09T07:18:42Z

examples/mqtt/iotcc_simulated_mqtt.py

+
+import Queue
+
+import pint


Use linux_metrics, it is recently updated in the master branch.

It's instead of psutil right.. I'll update it

KohliDev · 2017-01-09T07:25:10Z

examples/mqtt/samplePropMqtt.conf

+BrokerIP = "mqtt-broker"
+BrokerPort = "mqtt-broker-port"
+keep_alive = 60
+ca_cert = "/etc/liota/mqtt/conf/ca.crt"


Rename to:

gw_ca_cert
gw_cert_file
gw_key_file

Will use edge_system_ instead of gw_

KohliDev · 2017-01-09T07:27:24Z

liota/lib/identity/edge_system_identity.py

+#  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF     #
+#  THE POSSIBILITY OF SUCH DAMAGE.                                            #
+# ----------------------------------------------------------------------------#
+


Add some comments about the class.

KohliDev · 2017-01-09T07:27:34Z

liota/lib/identity/tls_conf.py

+#  ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF     #
+#  THE POSSIBILITY OF SUCH DAMAGE.                                            #
+# ----------------------------------------------------------------------------#
+


Add some comments about the class.

KohliDev · 2017-01-09T07:27:53Z

requirements.txt

@@ -1,4 +1,5 @@
 websocket-client==0.37.0
 psutil==4.3.1


Update this file as per latest code check-in

KohliDev · 2017-01-09T07:29:32Z

examples/mqtt/iotcc_simulated_mqtt.py

@@ -0,0 +1,172 @@
+# -*- coding: utf-8 -*-


We have to add the README.md in the next pull request on the usage of MQTT.

Yea. README should describe usage using GenericMqttDcc and IoTCC cases

KohliDev · 2017-01-09T11:14:30Z

liota/lib/transports/mqtt.py

+            # Validate CA certificate path
+            if self.edge_system_identity.ca_cert:
+                if not(os.path.exists(self.edge_system_identity.ca_cert)):
+                    raise ValueError("Error : Wrong CA certificate path.")


Add log.error statements where-ever applicable (ValueError & TypeError), don`t just raise the ValueError on the console

KohliDev · 2017-01-09T11:15:01Z

liota/dcc_comms/mqtt_dcc_comms.py

+        elif isinstance(mqtt_msg_attr, MqttMessagingAttributes):
+            self.msg_attr = mqtt_msg_attr
+        else:
+            raise TypeError("mqtt_mess_attr should either be None or of type MqttMessagingAttributes")


Add log.error statement

KohliDev · 2017-01-09T11:15:32Z

liota/lib/identity/edge_system_identity.py

+
+        if cert_file and key_file:
+            if not ca_cert:
+                raise ValueError("CA certificate path is required, when certification based auth is used")


Add log.error statement

KohliDev · 2017-01-09T11:20:39Z

liota/lib/transports/mqtt.py

+        if self.tls_details:
+
+            # Validate CA certificate path
+            if self.edge_system_identity.ca_cert:


Root_ca_cert come from the server is different from the gw_ca_cert which is generated locally. Use DCC_Identity class to hold username, password & dcc_ca_cert(root_ca_cert). Edge_system_identity class should only hold client certificates.

Yea. Agreed. Will have separate classes DccIdentity & EdgeSystemIdentity

KohliDev · 2017-01-10T14:09:27Z

liota/lib/identity/identity.py

@@ -47,7 +47,7 @@ class DccIdentity:
    def __init__(self, root_ca_cert, username, password):

        """
-        :param root_ca_cert: Root CA certificate path
+        :param root_ca_cert: Root CA certificate path or Self-signed certificate path


Self-signed server certificate path

KohliDev

All the commits in this pull request should be merged and committed as one single commit. It will be easy to track in future this specific pull request.

Venkat2811 changed the title ~~MQTT Transport Implementation : (#12)~~ MQTT Transport Implementation Oct 14, 2016

KohliDev reviewed Nov 4, 2016

View reviewed changes

Mqtt

f145bab

- made suggested changes

KohliDev approved these changes Nov 8, 2016

View reviewed changes

Mqtt (#22)

9bec783

- made suggested changes

pmasrani reviewed Nov 15, 2016

View reviewed changes

Ashish Panchal and others added 20 commits December 7, 2016 20:18

added 'mess_attr' to send(self, message)

646025e

MqttMessagingAttributes and other changes

2cb3746

has_attr() check for 'mess_attr'

5f0da72

few changes in device_comms

1708b9a

aws_iot dcc and examples

97c774b

added Connect-Disconnect-Timeout mechanism

a57a37b

added and updated docs

7c00eb2

updated Exception handling

4e4b2af

updated example and prop files

7d8a233

removed blank spaces

26d30ab

added newline

4fc61fa

updated comment

99b28ea

updated conf

8179917

bugfix in disconnect timeout

cb6dc3a

removing AWSIoT related.

2aa0639

updated setup.py

b15c18b

made transport configurable

7d6bc0c

mess_attr -> msg_attr

updated

e635067

bug fix

fa1b0c2

KohliDev reviewed Dec 21, 2016

View reviewed changes

Venkat2811 added 2 commits December 21, 2016 23:03

removed sys.exit(), updated comments, added on_unsubscribe()

99ccee9

added paho to requirements

5aa4a13

Merge branch 'master' into mqtt

8b1cede

pmasrani approved these changes Jan 9, 2017

View reviewed changes

KohliDev suggested changes Jan 9, 2017

View reviewed changes

Venkat2811 added 5 commits January 10, 2017 15:21

made changes as per review

b1329c1

Merge branch 'master' into mqtt

eb29aac

updated comment

073774b

updated comment

6d2c35b

renamed DccIdentity > RemoteSystemIdentity

bfed00d

KohliDev reviewed Jan 10, 2017

View reviewed changes

Venkat2811 added 2 commits January 10, 2017 19:55

updated comment

6e44969

updated log

c9b9d9d

KohliDev approved these changes Jan 16, 2017

View reviewed changes

KohliDev merged commit 944eb15 into vmware-archive:master Jan 16, 2017

Venkat2811 mentioned this pull request Jan 17, 2017

Providing MQTT Comms Support #54

Open

Venkat2811 deleted the mqtt branch February 17, 2017 12:20



		except RegistrationFailure:
		print "Registration to IOTCC failed"

		log = logging.getLogger(__name__)


		class MqttDccComms(DCCComms):

MQTT Transport Implementation #48

MQTT Transport Implementation #48

Conversation

Venkat2811 commented Oct 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KohliDev left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Venkat2811 Nov 21, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KohliDev Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Venkat2811 Jan 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KohliDev Jan 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KohliDev left a comment

Choose a reason for hiding this comment

Venkat2811 commented Oct 14, 2016 •

edited

Loading

KohliDev left a comment •

edited

Loading

Venkat2811 Nov 21, 2016 •

edited

Loading

KohliDev Nov 15, 2016 •

edited

Loading

Venkat2811 Jan 9, 2017 •

edited

Loading

KohliDev Jan 9, 2017 •

edited

Loading