New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic MQTT V5 updates #334

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@icraggs

icraggs commented Sep 26, 2018

These are my initial changes for MQTT V5 support. They allow V5 packets to be sent and received, as shown in the test_mqttv5.py. I'm submitting the changes at this stage for any comments or suggestions before I continue.

Remaining work:

  • V5 functionality: AUTH packets (probably for later), handling incoming DISCONNECT packet, session expiry (clean start vs clean session), server receive maximum (flow control)
  • doc
  • tests
  • examples

@icraggs icraggs added the enhancement label Sep 26, 2018

@icraggs icraggs requested a review from PierreF Sep 26, 2018

@icraggs

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 26, 2018

Oh, yes. It only works in Python 3 at the moment, I just found out. Looking into it.

icraggs commented Sep 26, 2018

Oh, yes. It only works in Python 3 at the moment, I just found out. Looking into it.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

Just for maintainablity:
Wouldnt it be better to define a small/simple (possibly inline) function that is ether called with a string or the list gets iterated and then the function is called with eath list entry string?

AtosNicoS commented on src/paho/mqtt/client.py in d2d62a1 Sep 27, 2018

Just for maintainablity:
Wouldnt it be better to define a small/simple (possibly inline) function that is ether called with a string or the list gets iterated and then the function is called with eath list entry string?

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

I made minimal changes to the underlying code - I that's a question for Pierre.

Owner

icraggs replied Sep 27, 2018

I made minimal changes to the underlying code - I that's a question for Pierre.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

The previous version of the function was way simpler and the command in the return statement different. I do not know any details about how it work. But did you made sure this code still works with mqtt v3? I am sure it is, just questioning it ;-)

AtosNicoS commented on src/paho/mqtt/client.py in d2d62a1 Sep 27, 2018

The previous version of the function was way simpler and the command in the return statement different. I do not know any details about how it work. But did you made sure this code still works with mqtt v3? I am sure it is, just questioning it ;-)

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

MQTT Version 3 of disconnect has no reason code or properties, basically no parameters, that's why it's simple. It should work with version 3 - I ran the tests I could find, but the CI tests should tell.

Owner

icraggs replied Sep 27, 2018

MQTT Version 3 of disconnect has no reason code or properties, basically no parameters, that's why it's simple. It should work with version 3 - I ran the tests I could find, but the CI tests should tell.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

This :2 Addition is new, also for mqtt3. Is it still correct?

AtosNicoS commented on src/paho/mqtt/client.py in d2d62a1 Sep 27, 2018

This :2 Addition is new, also for mqtt3. Is it still correct?

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

Yes!

Owner

icraggs replied Sep 27, 2018

Yes!

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

I already asked about enums here. I was not sure if your old comment intendet to change this code to enums or to keep it as it is. No matter what, this is just a reminder.

AtosNicoS commented on src/paho/mqtt/packettypes.py in d2d62a1 Sep 27, 2018

I already asked about enums here. I was not sure if your old comment intendet to change this code to enums or to keep it as it is. No matter what, this is just a reminder.

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

Enums are a fairly recent addition, apparently. Depends what version(s) of Python we want to support.

Owner

icraggs replied Sep 27, 2018

Enums are a fairly recent addition, apparently. Depends what version(s) of Python we want to support.

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

Enums were added in Python 3.4
Debian Jessie (current is stretch) uses Python 3.4.x and Ubuntu 16.04 (current is 18.04) uses Python 3.5.x.
Python 3.3 is end of life: https://www.python.org/dev/peps/pep-0398/#x-end-of-life
So I do not see any problem is using enums.

AtosNicoS replied Sep 27, 2018

Enums were added in Python 3.4
Debian Jessie (current is stretch) uses Python 3.4.x and Ubuntu 16.04 (current is 18.04) uses Python 3.5.x.
Python 3.3 is end of life: https://www.python.org/dev/peps/pep-0398/#x-end-of-life
So I do not see any problem is using enums.

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

Are they in Python 2.7?

Owner

icraggs replied Sep 27, 2018

Are they in Python 2.7?

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

As said before, I'd rename the whole class rather than making a comment.

AtosNicoS commented on src/paho/mqtt/properties.py in d2d62a1 Sep 27, 2018

As said before, I'd rename the whole class rather than making a comment.

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

I've made that change in my latest update - thought I'd put it into this commit, apparently not.

Owner

icraggs replied Sep 27, 2018

I've made that change in my latest update - thought I'd put it into this commit, apparently not.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

What is the reason for specifying the disconnect type twice or even once? A disconnect call should know it is a disconnect. I might missunderstand, but I dont understand the difference between the reasoncodes and the properties.

AtosNicoS commented on src/test_mqttv5.py in d2d62a1 Sep 27, 2018

What is the reason for specifying the disconnect type twice or even once? A disconnect call should know it is a disconnect. I might missunderstand, but I dont understand the difference between the reasoncodes and the properties.

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

Reasoncodes

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345303

and properties:

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345299

are part of the MQTT 5.0 specification. Different subsets of reason codes and properties are applicable to certain packet types. Specifying the packet type on the constructors allows validation.

Owner

icraggs replied Sep 27, 2018

Reasoncodes

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345303

and properties:

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345299

are part of the MQTT 5.0 specification. Different subsets of reason codes and properties are applicable to certain packet types. Specifying the packet type on the constructors allows validation.

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

I think i got a better understanding now. The aName is possibly also a candicate for enums.

AtosNicoS replied Sep 27, 2018

I think i got a better understanding now. The aName is possibly also a candicate for enums.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

Same question here: Why is specifying the unsubscribe type here required. I assume that this is an internal requirement, but I'd rather design the api to pass a dictionary to the function and the API builts the property object and appends the passed in user properties. From a user perspective this sounds easier for me.

AtosNicoS commented on src/test_mqttv5.py in d2d62a1 Sep 27, 2018

Same question here: Why is specifying the unsubscribe type here required. I assume that this is an internal requirement, but I'd rather design the api to pass a dictionary to the function and the API builts the property object and appends the passed in user properties. From a user perspective this sounds easier for me.

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

Or if it is really required an alternative would be to create a wrapper class named UnsubscribeProperties, which might be clearer. You must know best.

AtosNicoS replied Sep 27, 2018

Or if it is really required an alternative would be to create a wrapper class named UnsubscribeProperties, which might be clearer. You must know best.

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

Knowing the packet type for the properties object allows the properties to be validated. Only a subset of properties are allowed for each packet type.

Owner

icraggs replied Sep 27, 2018

Knowing the packet type for the properties object allows the properties to be validated. Only a subset of properties are allowed for each packet type.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

What does the number 2 stand for?

AtosNicoS commented on src/test_mqttv5.py in d2d62a1 Sep 27, 2018

What does the number 2 stand for?

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Sep 27, 2018

Owner

It's in the MQTT standard for retainHandling. We could have constants/enums for the values.

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345441

0 = Send retained messages at the time of the subscribe
1 = Send retained messages at subscribe only if the subscription does not currently exist
2 = Do not send retained messages at the time of the subscribe

Owner

icraggs replied Sep 27, 2018

It's in the MQTT standard for retainHandling. We could have constants/enums for the values.

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345441

0 = Send retained messages at the time of the subscribe
1 = Send retained messages at subscribe only if the subscription does not currently exist
2 = Do not send retained messages at the time of the subscribe

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

Those 3 number should be put into an enum.

AtosNicoS replied Sep 27, 2018

Those 3 number should be put into an enum.

@AtosNicoS

This comment has been minimized.

Show comment
Hide comment
@AtosNicoS

AtosNicoS Sep 27, 2018

A loop_stop() might be missing here.

AtosNicoS commented on src/test_mqttv5.py in d2d62a1 Sep 27, 2018

A loop_stop() might be missing here.

@icraggs

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Oct 1, 2018

OK, I've made this work in Python2.7 as well as 3. The tests are failing because they need an MQTT v5 broker to work against - this can be achieved by running the Paho test broker as does the C client. Is this a good option, or would you prefer the new tests to be run in the same way as existing?

icraggs commented Oct 1, 2018

OK, I've made this work in Python2.7 as well as 3. The tests are failing because they need an MQTT v5 broker to work against - this can be achieved by running the Paho test broker as does the C client. Is this a good option, or would you prefer the new tests to be run in the same way as existing?

@PierreF

New files you created use 2 spaces for indentation. Could you use 4 spaces instead ?

I hadn't yet time to look on the MQTT v5 protocol itself (i.e. I hadn't run anything with a v5 broker). I will try to made some test in the following week.

Thanks for this contribution.

@@ -1288,21 +1298,36 @@ def subscribe(self, topic, qos=0):
if isinstance(topic, tuple):
topic, qos = topic
if self._protocol == MQTTv5 and not isinstance(qos, SubscribeOptions):

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

I don't see where "qos" variable is used in MQTTv5, so why are we doing this check ?
Maybe we should have options = qos in case of MQTTv5 ?

On a side note, we have 3 way to call subscribe: list 2 arguments, with one tuple argument or with one list. We could probably deprecate the one tuple argument. So to go in this direction, I think it's acceptable that MQTTv5 does not work with one tuple argument and require either 2 arguments or one list.

@PierreF

PierreF Oct 7, 2018

Contributor

I don't see where "qos" variable is used in MQTTv5, so why are we doing this check ?
Maybe we should have options = qos in case of MQTTv5 ?

On a side note, we have 3 way to call subscribe: list 2 arguments, with one tuple argument or with one list. We could probably deprecate the one tuple argument. So to go in this direction, I think it's acceptable that MQTTv5 does not work with one tuple argument and require either 2 arguments or one list.

This comment has been minimized.

@icraggs

icraggs Oct 9, 2018

def subscribe(self, topic, qos=0):
def subscribe(self, topic, qos=0, options=None, properties=None):

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

I'm not convinced by the "qos" argument that is ignored with MQTTv5. User may use "subscribe(topic='test', qos=2)" and ends with a QoS=0 subscription on MQTTv5.

We should probably have:

  • A warning/error if qos is set when using MQTTv5
  • Use one argument for qos/option ("qos_or_option" ?).
  • Use qos argument to set the default qos of options. This one raise the question of what happen when qos and options are set :(

I would personally goes with 3th solution, and resolve conflict but forbidding to have both qos and options set. In MQTTv5 user either:

  • set qos as with old MQTTv3, we create a default SubscribeOptions with this QoS
  • set options, we use this options
  • settings with qos and options result in error
@PierreF

PierreF Oct 7, 2018

Contributor

I'm not convinced by the "qos" argument that is ignored with MQTTv5. User may use "subscribe(topic='test', qos=2)" and ends with a QoS=0 subscription on MQTTv5.

We should probably have:

  • A warning/error if qos is set when using MQTTv5
  • Use one argument for qos/option ("qos_or_option" ?).
  • Use qos argument to set the default qos of options. This one raise the question of what happen when qos and options are set :(

I would personally goes with 3th solution, and resolve conflict but forbidding to have both qos and options set. In MQTTv5 user either:

  • set qos as with old MQTTv3, we create a default SubscribeOptions with this QoS
  • set options, we use this options
  • settings with qos and options result in error

This comment has been minimized.

@icraggs

icraggs Oct 8, 2018

I'm happy with option 3. If both qos and options are set, we could issue an error?

@icraggs

icraggs Oct 8, 2018

I'm happy with option 3. If both qos and options are set, we could issue an error?

packed_connect_properties = b'\x00'
else:
packed_connect_properties = self._connect_properties.pack()
print("packet connect properties", len(packed_connect_properties))

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

This is a debug print statement, isn't ?

@PierreF

PierreF Oct 7, 2018

Contributor

This is a debug print statement, isn't ?

This comment has been minimized.

@icraggs

icraggs Oct 8, 2018

Yes. To be clear, I'm going to continue working on the contribution, but I wanted to get your general comments before going further so I wouldn't be faced with wholesale refactoring.

@icraggs

icraggs Oct 8, 2018

Yes. To be clear, I'm going to continue working on the contribution, but I wanted to get your general comments before going further so I wouldn't be faced with wholesale refactoring.

remaining_length += len(packed_props)
packet += packed_props
self._pack_remaining_length(packet, remaining_length)

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

Did I miss something or this will put the remaining length on byte > 2 when there is a reasoncode or properties (i.e. after them) ?

@PierreF

PierreF Oct 7, 2018

Contributor

Did I miss something or this will put the remaining length on byte > 2 when there is a reasoncode or properties (i.e. after them) ?

This comment has been minimized.

@icraggs

icraggs Oct 9, 2018

It shouldn't. It's just increasing the remaining length value to include that of the properties too. The properties are serialized first to determine their length, then added afterwards.

@icraggs

icraggs Oct 9, 2018

It shouldn't. It's just increasing the remaining length value to include that of the properties too. The properties are serialized first to determine their length, then added afterwards.

buffer += [length // 65536]
length %= 65536
buffer += [length // 256, length % 256]
return bytearray(buffer)

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

Any reason to no use struct.pack('!l') ?

@PierreF

PierreF Oct 7, 2018

Contributor

Any reason to no use struct.pack('!l') ?

This comment has been minimized.

@icraggs

icraggs Oct 8, 2018

No - I just never used the struct package, so my first contribution is just as is, with minimal changes, showing it works before any refactoring. The test program shows it works. We can replace code with struct.pack where appropriate.

@icraggs

icraggs Oct 8, 2018

No - I just never used the struct package, so my first contribution is just as is, with minimal changes, showing it works before any refactoring. The test program shows it works. We can replace code with struct.pack where appropriate.

zz = buf.find("\x00") # look for null in the UTF string
if zz != -1:
raise MalformedPacket("[MQTT-1.5.4-2] Null found in UTF data "+buf)
for c in range (0xD800, 0xDFFF):

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

I fear about the cost of this. We are iterating over a rather large range (2047 item) times the buffer length which could also be rather long.
I would said the if we are iterating over the buffer anyway, it may be cheaper to do:

for c in buf:
  if c >= 0xD800 and c <= 0xDFFF:
    raise MalformedPacket
@PierreF

PierreF Oct 7, 2018

Contributor

I fear about the cost of this. We are iterating over a rather large range (2047 item) times the buffer length which could also be rather long.
I would said the if we are iterating over the buffer anyway, it may be cheaper to do:

for c in buf:
  if c >= 0xD800 and c <= 0xDFFF:
    raise MalformedPacket

This comment has been minimized.

@icraggs

icraggs Oct 8, 2018

We could have a 'strict' option in the library which turns such checks on or off? Not sure what the default would be.

I would have thought the payload would potentially be the biggest piece of data, but yes, the UTF-8 strings could be large too.

@icraggs

icraggs Oct 8, 2018

We could have a 'strict' option in the library which turns such checks on or off? Not sure what the default would be.

I would have thought the payload would potentially be the biggest piece of data, but yes, the UTF-8 strings could be large too.

# check that this attribute applies to the packet type
if self.packetType not in self.properties[self.getIdentFromName(name)][1]:
raise MQTTException("Property %s does not apply to packet type %s"
% (name, Packets.Names[self.packetType]) )

This comment has been minimized.

@PierreF

PierreF Oct 7, 2018

Contributor

"Packets" variable does not exists

@PierreF

PierreF Oct 7, 2018

Contributor

"Packets" variable does not exists

This comment has been minimized.

@icraggs

icraggs Oct 8, 2018

True. I can add the Names array into the PacketTypes class.

@icraggs

icraggs Oct 8, 2018

True. I can add the Names array into the PacketTypes class.

@PierreF

This comment has been minimized.

Show comment
Hide comment
@PierreF

PierreF Oct 7, 2018

Contributor

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

Contributor

PierreF commented Oct 7, 2018

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

@icraggs

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Oct 8, 2018

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

I didn't expect that test to run - that was purely for checking that this code actually works before adding some proper tests!

Up to now on the other clients I've always used the Python test broker to test against, but I didn't know whether you would be happy with that approach. See https://github.com/eclipse/paho.mqtt.c/blob/master/travis-install.sh for an example. I think it provides some more flexibility and complicated tests if they are run against a 'real' broker such as this. What do you think?

icraggs commented Oct 8, 2018

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

I didn't expect that test to run - that was purely for checking that this code actually works before adding some proper tests!

Up to now on the other clients I've always used the Python test broker to test against, but I didn't know whether you would be happy with that approach. See https://github.com/eclipse/paho.mqtt.c/blob/master/travis-install.sh for an example. I think it provides some more flexibility and complicated tests if they are run against a 'real' broker such as this. What do you think?

@icraggs

This comment has been minimized.

Show comment
Hide comment
@icraggs

icraggs Oct 8, 2018

Yes, I'll change to 4 space indentation. I have always used 2 spaces myself to maximise space on the right hand side :-)

icraggs commented Oct 8, 2018

Yes, I'll change to 4 space indentation. I have always used 2 spaces myself to maximise space on the right hand side :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment