Skip to content
This repository has been archived by the owner on Mar 15, 2021. It is now read-only.

AWSIoT DCC, examples, README #71

Merged
merged 35 commits into from
Mar 18, 2017
Merged

Conversation

Venkat2811
Copy link
Contributor

@Venkat2811 Venkat2811 commented Jan 17, 2017

Fix for #63

  • AWSIoT DCC and example with various options available with LIOTA's MQTT
  • Added README.md for MQTT
  • Moved store_edge_system_uuid from iotcc.py > utility.py so that it can be used by Mqtt Dcc also.
  • Moved lib/identity > 'lib/utility'

# Encapsulates Identity Parameters related to Dcc
dcc_identity = RemoteSystemIdentity(root_ca_cert=config['broker_root_ca_cert'], username=None, password=None)
# Encapsulates Identity Parameters related to EdgeSystem
edge_system_identity = EdgeSystemIdentity(edge_system=edge_system, cert_file=config['edge_system_cert_file'],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EdgesystemIdentity can be part of edge_system, let's not create a separate entity. Same rule applies for dcc_identity. And we should make this changes for iotcc.py and its examples as well.

LIOTA's Mqtt client supports both Username-password based authentication & certification based authentication. Users can configure appropriate parameters using these objects
while initializing MqttDeviceComms and MqttDccComms:

* ![EdgeSystemIdentity & RemoteSystemIdentity](/liota/lib/identity/identity.py)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this part related to EdgeSystemIdentity as it will become part of EdgeSystem and make it part of main README.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, this will be part of transports. So, this will not be added to README.md

* Publish retain flag will be **False** and
* Subscribe call_back will be **None**

**(b)** Use custom single publish and subscribe topic for an EdgeSystem, its Devices and Metrics.
Copy link
Contributor

@KohliDev KohliDev Jan 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add that other then Point (a) you have to mention the custom parameters(Topics, QoS Level, flag etc.) in sampleProp.conf file (give the path of sampleProp.conf file for reference)

* ![TLSConf](/liota/lib/identity/tls_conf.py)

## QoS
LIOTA also supports configuration of QoS related parameters like in_flight size, queue_size and retry timeout using **QoSDetails class** in ![mqtt.py](/liota/lib/transports/mqtt.py).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to refer mqtt.py

it follows `Option (a)` as mentioned in the above section.
* **Option (a)** could be achieved by passing **mqtt_msg_attr=None while initializing MqttDccComms**.

Examples for this will be provided once the development is complete.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember to update this line later or remove it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remember.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove it

reg_light_metric = aws.register(light_metric)
aws.create_relationship(reg_light_sensor, reg_light_metric)
# Publish topic for this Metric
reg_light_metric.msg_attr = MqttMessagingAttributes(pub_topic='home/living-room/light')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention a comment here that custom topic can also be referred from sampleProp.conf file and user should take care of referring it correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

"""
def __init__(self, con, enclose_metadata=False):
"""
:param con: AWSMQTTDccComms Object
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generic MQTT Object not AWSMQTTDccComms

def _get_entity_hierarchy(self, reg_entity):
"""
:param reg_entity: RegisteredMetric Object
:return: Entity hierarchy delimited by '->'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don`t use delimiter use an array or map or any other data structure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

@@ -31,6 +31,9 @@ or a traditional socket endpoint.
## DCC (Data Center Component)
The abstract class DCC represents an application in a data-center. It is potentially the most important and complex abstraction of liota. It provides flexibility to developers for choosing the data-center components they need and using API’s provided by liota. With help of this abstraction developers may build custom solutions. The abstract class states basic methods and encapsulates them into unified common API’s required to send data to various DCC’s. Graphite and Project Ice are currently the data-center components supported with AWS, BlueMix and ThingWorx to come soon. New DCC’s can easily be integrated in the abstraction.

## Transports
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention in the README.md that Gateway object should be created before creating DCC_comms object. And also how to set the Edge system identity and DCC identity should be made part of main REDME.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay

"""
_list = []
met_cnt = reg_metric.values.qsize()
if met_cnt > 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

met_cnt is zero then we`ll be sending a blank metric_data payload? Will it get filtered out later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should have been if met_cnt == 0 return.

# Initializing GenericMqtt DCC using MqttDccComms
# AWSIoT broker doesn't support session persistence. So, always use "clean_session=True"
# Publish topic for all Metrics will be 'liota/generated_local_uuid_of_edge_system'
aws = GenericMqtt(MqttDccComms(edge_system=edge_system, url=config['BrokerIP'], port=config['BrokerPort'],
Copy link
Contributor

@KohliDev KohliDev Feb 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge_system to hold name as well as client certificates, don't take them individually in any DCC_comms. Also, Make edge_system to be passed to all the DCC_Comms.

retry = 5
CustomPubTopic = "home/gateway/metrics"
LivingRoomTempTopic = "home/living-room/temperature"
LivingRoomHumTopic = "home/living-room/humidity"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LivingRoomHumidityTopic

LIOTA offers MQTT protocol as transport at both Device & DCC ends via ![MqttDeviceComms](/liota/device_comms/mqtt_device_comms.py) & ![MqttDccComms](/liota/dcc_comms/mqtt_dcc_comms.py).


## QoS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, explain about credentials and TLS_Conf details which can be passed using the Utilities.


class Credentials:
"""
This class encapsulates credentials related to a connection both at Dcc and Device side.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class encapsulates certificates or credentials related to a connection used at both DCC and Device side.

README.md Outdated
@@ -22,7 +24,7 @@ The Metric subclass of Entity is the local object representing a stream of (numb
The abstract class DeviceComms represent mechanisms through which devices send and receive data to/from edge systems. Some examples are CAN bus, Modbus, ProfiNet, Zibgee, GPIO pins, Industrial Serial Protocols as well as sockets, websockets, MQTT, CoAP. The DeviceComms abstract class is a placeholder for these various communication mechanisms.

## DCCComms
The abstract class DCCComms represents communication protocols between edge systems and DCCs. Currently, liota supports WebSocket and plain old BSD sockets. In near future it will support MQTT and CoAP. Both are ‘Session’ or layer-5 protocols. MQTT is a pub-sub system using TCP and CoAP implements reliable UDP datagrams and a data format specification. These protocols are capable of satisfying most of the use cases for transferring data from IoT gateways to data-center components. With the current implementation the gateway acts as either a WebSocket client, establishing a connection with the server using the WebSocket protocol. e.g.
The abstract class DCCComms represents communication protocols between edge systems and DCCs. Currently, liota supports MQTT, WebSocket and plain old BSD sockets. In near future it will support CoAP. Both are ‘Session’ or layer-5 protocols. MQTT is a pub-sub system using TCP and CoAP implements reliable UDP datagrams and a data format specification. These protocols are capable of satisfying most of the use cases for transferring data from IoT gateways to data-center components. With the current implementation the gateway acts as either a WebSocket client, establishing a connection with the server using the WebSocket protocol. e.g.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are ‘Session’ or layer-5 protocols? Which both? Needs rewrite.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I edited In near future it will support MQTT and CoAP line alone. Didn't notice this. Will rewrite.

Copy link
Contributor Author

@Venkat2811 Venkat2811 Feb 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebSocket, MQTT and CoAP are both application layer (layer-7) protocols. https://en.wikipedia.org/wiki/Category:Application_layer_protocols

LIOTA also supports configuration of QoS related parameters like in_flight size, queue_size and retry timeout using **QoSDetails class**.


## Credentials and TLS Configurations

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Credentials class is generic, not specific for MQTT, i think it will be better to document it in common Readme, not in MQTT readme.

It enables the following options for developers in LIOTA.

**(a)** Use single publish and subscribe topic generated by LIOTA for an EdgeSystem, its Devices and its Metrics. By default:
* Publish topic for all Metrics will be **liota/generated_local_uuid_of_edge_system**

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be a good idea to consider liota/request/generated_local_uuid_of_edge_system and liota/response/generated_local_uuid_of_edge_system as topics. This way, if required some client can subscribe to all communication and some to just requests and some to just responses. Also, current naming is not very obvious. It's difficult to figure out from topic names what it stands for. liota-resp also seems to be unnecessarily shortened.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using liota/stats/local_uuid and liota/response/local_uuid instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can refactor it more:
liota/gw-uuid/requests
&
liota/gw-uuid/response

Gives much more subscription possibilities.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea. This will give edge_system level subscription.

Will use stats instead of requests as it'll suit much better.


**(b)** Use custom single publish and subscribe topic for an EdgeSystem, its Devices and Metrics. Along with custom topics, other parameters (QoS, retain_flag ,etc.,) can be referred from ![Property File](/examples/mqtt/aws_iot/awsSampleProp.conf).

**-** In the above two cases, MQTT message's payload MUST be self-descriptive so that subscriber can subscribe process accordingly to a single topic by parsing payload.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please elaborate more, using an example, what is self-descriptive?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea there is an example. Will provide reference to it.

Self-Descriptive means when each metric is published, its edge_system's_name, device's_name will also be appended. IoTCC does it by appending uuids, GenericMqtt Dcc does it with a configurable Boolean parameter enclose_metadata



# Random number generator, simulating living room temperature readings.
def living_room_temperature():

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_living_room_temperature instead of just living_room_temperature?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

# Initializing GenericMqtt DCC using MqttDccComms
# AWSIoT broker doesn't support session persistence. So, always use "clean_session=True"
# Publish topic for all Metrics will be 'liota/generated_local_uuid_of_edge_system'
aws = GenericMqtt(MqttDccComms(edge_system_name=edge_system.name, url=config['BrokerIP'], port=config['BrokerPort'],

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you format this better, one parameter per line?

Copy link
Contributor Author

@Venkat2811 Venkat2811 Feb 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of one parameter per line, grouping related parameters together in one line.



### Configuration for Non-ProjectICE Scenario
![GenericMqtt DCC](/liota/dccs/generic_mqtt.py) is a Data Center Component to connect with MQTT brokers using LIOTA in non-ProjectICE scenarios. **Options (a), (b), (c) and (d)** mentioned in the above section

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe there is a design flaw here. DCCs should not be very specific to protocols. That's the very reason of extracting out DCCComms as separate module. Hence, i see AWSIoT as DCC and we can have AWSIoT over MQTT, HTTP or MQTT + WebSocket (http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html). It seems incorrect to implement AWSIoT as GenericMQTT.

Copy link
Contributor Author

@Venkat2811 Venkat2811 Feb 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conventional MQTT is MQTT over TCP and there is also support for MQTT over WebSocket. Most of the MQTT brokers (AWS, EMQTT, etc.,) support it. There is no difference in configurations except for parameter transport in MQTT configuration.

Yes. That would be a DCC specific to AWSIoT only. Here what we are trying to showcase is that, MQTT support in LIOTA is generic so that it can be used to publish data to any DCC that supports MQTT. As of now, this example showcases publishing data to AWSIoT over MQTT. Examples using Azure is ready https://github.com/Venkat2811/liota/tree/azure-example/examples/mqtt/azure_iot. We're working on IBM Bluemix too. Once we have support for multiple transports like, HTTP-REST, AMQP, we can provide platform specific Dcc. We're planning to raise Azure and IBM examples as separate PR

return value


class Credentials:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason we moved away from identity module and included this in utility? We should try to provide only very basic read/write kind of functionality in utility and not include many disjoint classes/methods here. We also have to rename this file and probably split it into multiple files based on functionalities.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EdgeSystemIdentity and RemoteSystemIdentity didn't suit well. Credentials will be simply a data object (utility) that holds cert paths and credentials used in various classes. Goal is to reduce number of constructor params of DccComms & DeviceComms related to Credentials. So, we thought why to create a separate module for this and kept it in Utility itself.

Configuring MqttDccComms and MessagingAttributes for ProjectICE and non-ProjectICE related use-cases is explained in the following sections.

### Configuration for ProjectICE Scenario
![IoTCC DCC](/liota/dccs/iotcc.py) will support both WebSockets and MQTT as transports. **This is WIP**. Once developed it'll allow plug-and-play approach to easily switch between these two protocols. However, for MQTT

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have to do separate configuration for project ICE?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Project ICE, publish topic would be liota/generated_local_uuid. In non-Project ICE it need not be the case. So, there is separate configuration for non-project ICE. For Project ICE scenarios, it'll remain similar to existing configuration.

broker_root_ca_cert = "/etc/liota/mqtt/conf/ca.crt"
broker_username = None
broker_password = None
edge_system_cert_file = "/etc/liota/mqtt/conf/client.crt"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this location be /etc/liota/conf/mqtt/ ? All configurations should go together, and MQTT configurations be specific, isn't it? Also, will other DCCComms not need client.crt and client.key? Why this has to be specific to mqtt?

Copy link
Contributor Author

@Venkat2811 Venkat2811 Feb 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't decide on location for cert files as they are user defined and path is obtained from sampleProp.conf and not from liota.conf. However, for representation, it can be /etc/liota/conf/mqtt/.

If a platform supports certificate based authentication, DCCComms will need them. If we're publishing to two different DCCs, certificates might (not always) be different.

@Venkat2811 Venkat2811 changed the title GenericMqtt DCC, Example using AWSIoT, README AWSIoT DCC, examples, README Mar 9, 2017
@@ -41,10 +41,8 @@
from liota.entities.devices.simulated_device import SimulatedDevice
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename folder mqtt/iotcc

iot_cc creates confusion

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay. Will change to mqtt/device_comms/iotcc

userdata=None, protocol="MQTTv311", transport="tcp", mqtt_msg_attr=None, keep_alive=60,
enable_authentication=False, conn_disconn_timeout=10):
def __init__(self, edge_system_name, url, port, identity=None, tls_conf=None, qos_details=None,
client_id="", clean_session=False, userdata=None, protocol="MQTTv311", transport="tcp", keep_alive=60,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is client_id changed from None to "" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keeping it consistent with paho library

setup.py Outdated
['examples/mqtt/device_comms/iot_cc/iotcc_simulated_mqtt.py',
'examples/mqtt/device_comms/iot_cc/samplePropMqtt.conf']),
(os.path.abspath(os.sep) + '/../etc/liota/examples/mqtt/dcc_comms/aws_iot',
['examples/mqtt/dcc_comms/aws_iot/sampleProp.conf',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add all of this extra packages to Manifest.in

Copy link
Contributor

@KohliDev KohliDev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raise a small PR next time for faster review.


reg_entity_child.parent = reg_entity_parent

def _get_entity_hierarchy(self, reg_entity):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment this might be taken out in utilities if other DCC requires hierarchy.

**(a)** Use single publish and subscribe topic generated by LIOTA for an EdgeSystem, its Devices and its Metrics. By default:
* Publish topic for all Metrics will be **liota/generated_local_uuid_of_edge_system/request**
* Subscribe topic will be **liota/generated_local_uuid_of_edge_system/response**
* Publish and Subscribe QoS will be **One**

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this default configuration specific to (a) or to all?


# ----------------------------------------------------------------------------------------------------------------
# In this example, we demonstrate how data for a simulated metric generating
# random numbers can be directed to GenericMqtt data center component using Liota.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"GenericMqtt "? Update comment.


# ----------------------------------------------------------------------------------------------------------------
# In this example, we demonstrate how data for a simulated metric generating
# random numbers can be directed to GenericMqtt data center component using Liota.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GenericMqtt?

# Creating EdgeSystem
edge_system = Dell5KEdgeSystem(config['EdgeSystemName'])
# Encapsulates Identity
identity = Identity(config['broker_root_ca_cert'], None, None,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is difficult to understand what None arguments are. Can you name the arguments?

self.password = password
self.cert_file = cert_file
self.key_file = key_file
log.debug("Created Identity.")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For debugging, these kind of logs are not very useful. It is better to provide more information in debug logs like with what username the identity is created etc.

@KohliDev KohliDev merged commit b67d1e7 into vmware-archive:master Mar 18, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants