-
Notifications
You must be signed in to change notification settings - Fork 120
Conversation
# Encapsulates Identity Parameters related to Dcc | ||
dcc_identity = RemoteSystemIdentity(root_ca_cert=config['broker_root_ca_cert'], username=None, password=None) | ||
# Encapsulates Identity Parameters related to EdgeSystem | ||
edge_system_identity = EdgeSystemIdentity(edge_system=edge_system, cert_file=config['edge_system_cert_file'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
EdgesystemIdentity can be part of edge_system, let's not create a separate entity. Same rule applies for dcc_identity. And we should make this changes for iotcc.py and its examples as well.
examples/mqtt/README.md
Outdated
LIOTA's Mqtt client supports both Username-password based authentication & certification based authentication. Users can configure appropriate parameters using these objects | ||
while initializing MqttDeviceComms and MqttDccComms: | ||
|
||
* ![EdgeSystemIdentity & RemoteSystemIdentity](/liota/lib/identity/identity.py) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this part related to EdgeSystemIdentity as it will become part of EdgeSystem and make it part of main README.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed, this will be part of transports. So, this will not be added to README.md
examples/mqtt/README.md
Outdated
* Publish retain flag will be **False** and | ||
* Subscribe call_back will be **None** | ||
|
||
**(b)** Use custom single publish and subscribe topic for an EdgeSystem, its Devices and Metrics. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add that other then Point (a) you have to mention the custom parameters(Topics, QoS Level, flag etc.) in sampleProp.conf file (give the path of sampleProp.conf file for reference)
examples/mqtt/README.md
Outdated
* ![TLSConf](/liota/lib/identity/tls_conf.py) | ||
|
||
## QoS | ||
LIOTA also supports configuration of QoS related parameters like in_flight size, queue_size and retry timeout using **QoSDetails class** in ![mqtt.py](/liota/lib/transports/mqtt.py). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to refer mqtt.py
examples/mqtt/README.md
Outdated
it follows `Option (a)` as mentioned in the above section. | ||
* **Option (a)** could be achieved by passing **mqtt_msg_attr=None while initializing MqttDccComms**. | ||
|
||
Examples for this will be provided once the development is complete. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remember to update this line later or remove it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remember.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remove it
reg_light_metric = aws.register(light_metric) | ||
aws.create_relationship(reg_light_sensor, reg_light_metric) | ||
# Publish topic for this Metric | ||
reg_light_metric.msg_attr = MqttMessagingAttributes(pub_topic='home/living-room/light') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mention a comment here that custom topic can also be referred from sampleProp.conf file and user should take care of referring it correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure
liota/dccs/generic_mqtt.py
Outdated
""" | ||
def __init__(self, con, enclose_metadata=False): | ||
""" | ||
:param con: AWSMQTTDccComms Object |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generic MQTT Object not AWSMQTTDccComms
liota/dccs/generic_mqtt.py
Outdated
def _get_entity_hierarchy(self, reg_entity): | ||
""" | ||
:param reg_entity: RegisteredMetric Object | ||
:return: Entity hierarchy delimited by '->' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don`t use delimiter use an array or map or any other data structure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure
@@ -31,6 +31,9 @@ or a traditional socket endpoint. | |||
## DCC (Data Center Component) | |||
The abstract class DCC represents an application in a data-center. It is potentially the most important and complex abstraction of liota. It provides flexibility to developers for choosing the data-center components they need and using API’s provided by liota. With help of this abstraction developers may build custom solutions. The abstract class states basic methods and encapsulates them into unified common API’s required to send data to various DCC’s. Graphite and Project Ice are currently the data-center components supported with AWS, BlueMix and ThingWorx to come soon. New DCC’s can easily be integrated in the abstraction. | |||
|
|||
## Transports |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mention in the README.md that Gateway object should be created before creating DCC_comms object. And also how to set the Edge system identity and DCC identity should be made part of main REDME.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay
liota/dccs/generic_mqtt.py
Outdated
""" | ||
_list = [] | ||
met_cnt = reg_metric.values.qsize() | ||
if met_cnt > 0: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
met_cnt is zero then we`ll be sending a blank metric_data payload? Will it get filtered out later?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should have been if met_cnt == 0 return
.
# Initializing GenericMqtt DCC using MqttDccComms | ||
# AWSIoT broker doesn't support session persistence. So, always use "clean_session=True" | ||
# Publish topic for all Metrics will be 'liota/generated_local_uuid_of_edge_system' | ||
aws = GenericMqtt(MqttDccComms(edge_system=edge_system, url=config['BrokerIP'], port=config['BrokerPort'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Edge_system to hold name as well as client certificates, don't take them individually in any DCC_comms. Also, Make edge_system to be passed to all the DCC_Comms.
retry = 5 | ||
CustomPubTopic = "home/gateway/metrics" | ||
LivingRoomTempTopic = "home/living-room/temperature" | ||
LivingRoomHumTopic = "home/living-room/humidity" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LivingRoomHumidityTopic
LIOTA offers MQTT protocol as transport at both Device & DCC ends via ![MqttDeviceComms](/liota/device_comms/mqtt_device_comms.py) & ![MqttDccComms](/liota/dcc_comms/mqtt_dcc_comms.py). | ||
|
||
|
||
## QoS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, explain about credentials and TLS_Conf details which can be passed using the Utilities.
liota/lib/utilities/utility.py
Outdated
|
||
class Credentials: | ||
""" | ||
This class encapsulates credentials related to a connection both at Dcc and Device side. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class encapsulates certificates or credentials related to a connection used at both DCC and Device side.
README.md
Outdated
@@ -22,7 +24,7 @@ The Metric subclass of Entity is the local object representing a stream of (numb | |||
The abstract class DeviceComms represent mechanisms through which devices send and receive data to/from edge systems. Some examples are CAN bus, Modbus, ProfiNet, Zibgee, GPIO pins, Industrial Serial Protocols as well as sockets, websockets, MQTT, CoAP. The DeviceComms abstract class is a placeholder for these various communication mechanisms. | |||
|
|||
## DCCComms | |||
The abstract class DCCComms represents communication protocols between edge systems and DCCs. Currently, liota supports WebSocket and plain old BSD sockets. In near future it will support MQTT and CoAP. Both are ‘Session’ or layer-5 protocols. MQTT is a pub-sub system using TCP and CoAP implements reliable UDP datagrams and a data format specification. These protocols are capable of satisfying most of the use cases for transferring data from IoT gateways to data-center components. With the current implementation the gateway acts as either a WebSocket client, establishing a connection with the server using the WebSocket protocol. e.g. | |||
The abstract class DCCComms represents communication protocols between edge systems and DCCs. Currently, liota supports MQTT, WebSocket and plain old BSD sockets. In near future it will support CoAP. Both are ‘Session’ or layer-5 protocols. MQTT is a pub-sub system using TCP and CoAP implements reliable UDP datagrams and a data format specification. These protocols are capable of satisfying most of the use cases for transferring data from IoT gateways to data-center components. With the current implementation the gateway acts as either a WebSocket client, establishing a connection with the server using the WebSocket protocol. e.g. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both are ‘Session’ or layer-5 protocols? Which both? Needs rewrite.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I edited In near future it will support MQTT and CoAP
line alone. Didn't notice this. Will rewrite.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WebSocket, MQTT and CoAP are both application layer (layer-7) protocols. https://en.wikipedia.org/wiki/Category:Application_layer_protocols
examples/mqtt/README.md
Outdated
LIOTA also supports configuration of QoS related parameters like in_flight size, queue_size and retry timeout using **QoSDetails class**. | ||
|
||
|
||
## Credentials and TLS Configurations |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since Credentials class is generic, not specific for MQTT, i think it will be better to document it in common Readme, not in MQTT readme.
examples/mqtt/README.md
Outdated
It enables the following options for developers in LIOTA. | ||
|
||
**(a)** Use single publish and subscribe topic generated by LIOTA for an EdgeSystem, its Devices and its Metrics. By default: | ||
* Publish topic for all Metrics will be **liota/generated_local_uuid_of_edge_system** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be a good idea to consider liota/request/generated_local_uuid_of_edge_system and liota/response/generated_local_uuid_of_edge_system as topics. This way, if required some client can subscribe to all communication and some to just requests and some to just responses. Also, current naming is not very obvious. It's difficult to figure out from topic names what it stands for. liota-resp also seems to be unnecessarily shortened.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using liota/stats/local_uuid and liota/response/local_uuid instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can refactor it more:
liota/gw-uuid/requests
&
liota/gw-uuid/response
Gives much more subscription possibilities.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea. This will give edge_system level subscription.
Will use stats
instead of requests
as it'll suit much better.
examples/mqtt/README.md
Outdated
|
||
**(b)** Use custom single publish and subscribe topic for an EdgeSystem, its Devices and Metrics. Along with custom topics, other parameters (QoS, retain_flag ,etc.,) can be referred from ![Property File](/examples/mqtt/aws_iot/awsSampleProp.conf). | ||
|
||
**-** In the above two cases, MQTT message's payload MUST be self-descriptive so that subscriber can subscribe process accordingly to a single topic by parsing payload. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please elaborate more, using an example, what is self-descriptive?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea there is an example. Will provide reference to it.
Self-Descriptive means when each metric is published, its edge_system's_name, device's_name will also be appended. IoTCC does it by appending uuids, GenericMqtt Dcc does it with a configurable Boolean parameter enclose_metadata
|
||
|
||
# Random number generator, simulating living room temperature readings. | ||
def living_room_temperature(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
get_living_room_temperature instead of just living_room_temperature?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure
# Initializing GenericMqtt DCC using MqttDccComms | ||
# AWSIoT broker doesn't support session persistence. So, always use "clean_session=True" | ||
# Publish topic for all Metrics will be 'liota/generated_local_uuid_of_edge_system' | ||
aws = GenericMqtt(MqttDccComms(edge_system_name=edge_system.name, url=config['BrokerIP'], port=config['BrokerPort'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you format this better, one parameter per line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of one parameter per line, grouping related parameters together in one line.
examples/mqtt/README.md
Outdated
|
||
|
||
### Configuration for Non-ProjectICE Scenario | ||
![GenericMqtt DCC](/liota/dccs/generic_mqtt.py) is a Data Center Component to connect with MQTT brokers using LIOTA in non-ProjectICE scenarios. **Options (a), (b), (c) and (d)** mentioned in the above section |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe there is a design flaw here. DCCs should not be very specific to protocols. That's the very reason of extracting out DCCComms as separate module. Hence, i see AWSIoT as DCC and we can have AWSIoT over MQTT, HTTP or MQTT + WebSocket (http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html). It seems incorrect to implement AWSIoT as GenericMQTT.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Conventional MQTT is MQTT over TCP and there is also support for MQTT over WebSocket. Most of the MQTT brokers (AWS, EMQTT, etc.,) support it. There is no difference in configurations except for parameter transport
in MQTT configuration.
Yes. That would be a DCC specific to AWSIoT only. Here what we are trying to showcase is that, MQTT support in LIOTA is generic so that it can be used to publish data to any DCC that supports MQTT. As of now, this example showcases publishing data to AWSIoT over MQTT. Examples using Azure is ready https://github.com/Venkat2811/liota/tree/azure-example/examples/mqtt/azure_iot. We're working on IBM Bluemix too. Once we have support for multiple transports like, HTTP-REST, AMQP, we can provide platform specific Dcc. We're planning to raise Azure and IBM examples as separate PR
liota/lib/utilities/utility.py
Outdated
return value | ||
|
||
|
||
class Credentials: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any specific reason we moved away from identity module and included this in utility? We should try to provide only very basic read/write kind of functionality in utility and not include many disjoint classes/methods here. We also have to rename this file and probably split it into multiple files based on functionalities.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
EdgeSystemIdentity and RemoteSystemIdentity didn't suit well. Credentials
will be simply a data object (utility) that holds cert paths and credentials used in various classes. Goal is to reduce number of constructor params of DccComms & DeviceComms related to Credentials. So, we thought why to create a separate module for this and kept it in Utility itself.
examples/mqtt/README.md
Outdated
Configuring MqttDccComms and MessagingAttributes for ProjectICE and non-ProjectICE related use-cases is explained in the following sections. | ||
|
||
### Configuration for ProjectICE Scenario | ||
![IoTCC DCC](/liota/dccs/iotcc.py) will support both WebSockets and MQTT as transports. **This is WIP**. Once developed it'll allow plug-and-play approach to easily switch between these two protocols. However, for MQTT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we have to do separate configuration for project ICE?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Project ICE, publish topic would be liota/generated_local_uuid
. In non-Project ICE it need not be the case. So, there is separate configuration for non-project ICE. For Project ICE scenarios, it'll remain similar to existing configuration.
broker_root_ca_cert = "/etc/liota/mqtt/conf/ca.crt" | ||
broker_username = None | ||
broker_password = None | ||
edge_system_cert_file = "/etc/liota/mqtt/conf/client.crt" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this location be /etc/liota/conf/mqtt/ ? All configurations should go together, and MQTT configurations be specific, isn't it? Also, will other DCCComms not need client.crt and client.key? Why this has to be specific to mqtt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We didn't decide on location for cert files as they are user defined and path is obtained from sampleProp.conf
and not from liota.conf
. However, for representation, it can be /etc/liota/conf/mqtt/
.
If a platform supports certificate based authentication, DCCComms will need them. If we're publishing to two different DCCs, certificates might (not always) be different.
@@ -41,10 +41,8 @@ | |||
from liota.entities.devices.simulated_device import SimulatedDevice |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename folder mqtt/iotcc
iot_cc creates confusion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay. Will change to mqtt/device_comms/iotcc
userdata=None, protocol="MQTTv311", transport="tcp", mqtt_msg_attr=None, keep_alive=60, | ||
enable_authentication=False, conn_disconn_timeout=10): | ||
def __init__(self, edge_system_name, url, port, identity=None, tls_conf=None, qos_details=None, | ||
client_id="", clean_session=False, userdata=None, protocol="MQTTv311", transport="tcp", keep_alive=60, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is client_id changed from None to "" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keeping it consistent with paho
library
setup.py
Outdated
['examples/mqtt/device_comms/iot_cc/iotcc_simulated_mqtt.py', | ||
'examples/mqtt/device_comms/iot_cc/samplePropMqtt.conf']), | ||
(os.path.abspath(os.sep) + '/../etc/liota/examples/mqtt/dcc_comms/aws_iot', | ||
['examples/mqtt/dcc_comms/aws_iot/sampleProp.conf', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add all of this extra packages to Manifest.in
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Raise a small PR next time for faster review.
|
||
reg_entity_child.parent = reg_entity_parent | ||
|
||
def _get_entity_hierarchy(self, reg_entity): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a comment this might be taken out in utilities if other DCC requires hierarchy.
examples/mqtt/README.md
Outdated
**(a)** Use single publish and subscribe topic generated by LIOTA for an EdgeSystem, its Devices and its Metrics. By default: | ||
* Publish topic for all Metrics will be **liota/generated_local_uuid_of_edge_system/request** | ||
* Subscribe topic will be **liota/generated_local_uuid_of_edge_system/response** | ||
* Publish and Subscribe QoS will be **One** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this default configuration specific to (a) or to all?
|
||
# ---------------------------------------------------------------------------------------------------------------- | ||
# In this example, we demonstrate how data for a simulated metric generating | ||
# random numbers can be directed to GenericMqtt data center component using Liota. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"GenericMqtt "? Update comment.
|
||
# ---------------------------------------------------------------------------------------------------------------- | ||
# In this example, we demonstrate how data for a simulated metric generating | ||
# random numbers can be directed to GenericMqtt data center component using Liota. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GenericMqtt?
# Creating EdgeSystem | ||
edge_system = Dell5KEdgeSystem(config['EdgeSystemName']) | ||
# Encapsulates Identity | ||
identity = Identity(config['broker_root_ca_cert'], None, None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is difficult to understand what None arguments are. Can you name the arguments?
liota/lib/utilities/identity.py
Outdated
self.password = password | ||
self.cert_file = cert_file | ||
self.key_file = key_file | ||
log.debug("Created Identity.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For debugging, these kind of logs are not very useful. It is better to provide more information in debug logs like with what username the identity is created etc.
Fix for #63
store_edge_system_uuid
fromiotcc.py
>utility.py
so that it can be used by Mqtt Dcc also.lib/identity
> 'lib/utility'