MQTT V5 support

[icraggs]

These are my initial changes for MQTT V5 support. They allow V5 packets to be sent and received, as shown in the test_mqttv5.py. I'm submitting the changes at this stage for any comments or suggestions before I continue.

Remaining work:

• V5 functionality: AUTH packets (probably for later), handling incoming DISCONNECT packet, session expiry (clean start vs clean session), server receive maximum (flow control)
• doc
• tests
• examples

[icraggs]

Oh, yes. It only works in Python 3 at the moment, I just found out. Looking into it.

[icraggs]

OK, I've made this work in Python2.7 as well as 3. The tests are failing because they need an MQTT v5 broker to work against - this can be achieved by running the Paho test broker as does the C client. Is this a good option, or would you prefer the new tests to be run in the same way as existing?

[PierreF]

New files you created use 2 spaces for indentation. Could you use 4 spaces instead ?

I hadn't yet time to look on the MQTT v5 protocol itself (i.e. I hadn't run anything with a v5 broker). I will try to made some test in the following week.

Thanks for this contribution.

[PierreF]

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

[icraggs]

About the test, I think it would be better to use same kind of tests as already present.
There is already two kind of test: "test" folder which use old tests and "tests" folder which use new testing infrastructure (py.test) but we hadn't moved all tests yet.

I didn't expect that test to run - that was purely for checking that this code actually works before adding some proper tests!

Up to now on the other clients I've always used the Python test broker to test against, but I didn't know whether you would be happy with that approach. See https://github.com/eclipse/paho.mqtt.c/blob/master/travis-install.sh for an example. I think it provides some more flexibility and complicated tests if they are run against a 'real' broker such as this. What do you think?

[icraggs]

Yes, I'll change to 4 space indentation. I have always used 2 spaces myself to maximise space on the right hand side :-)

[AtosNicoS]

Is there any update on this? :)

[icraggs]

Sorry, I'm working on the Java client release (https://github.com/eclipse/paho.mqtt.java/milestone/2) as my top priority. I'll get back to this as soon as I can. I'm open to any offers of help.

[icraggs]

I've fixed the indentation, added constant definitions for the retain handling options, and removed some debug statements. The simple validation test works for me.

I'd like to know what you would like to see from me for this PR to be accepted. Thanks!

[icraggs]

Oh - obviously I should make the tests work! And there are some other comments I haven't yet addressed.

[icraggs]

I've addressed all the points that Pierre made. I've started on the tests - got about a third of the way through those. The remaining work is finishing off the test suite and the documentation. For anyone who wants to try it out, it's basically working.

[icraggs]

@PierreF Any other tasks/changes (apart from doc completion) which you'd like me to do/make? I did run the tests against Mosquitto and found some issues, some of which are definitely in Mosquitto.

[PierreF]

Apart from running test that may need some improvement (if I run tox on my system, it will fail because the testing/interoperability broker is not running), I don't have more changes to do.

Are you going to make documentation change in this PR ?

About running the test, I'm not sure what is the best solution... I think to some kind of conditional test, like "tox --enable-interoperability-tests" which would be used by travis or someone which run the interop broker. But I don't known what the proper way to do this.
If you had time to look at this, it would be great. If not, I'll fix it later (I think using an environment variable and pytest skipif may do the job).

[icraggs]

@PierreF Thanks. Yes, it is my intention to add the docs in this PR, I just wanted to get your feedback before doing that. Did you get chance to try these changes out with Mosquitto?

I'll look into getting the tox tests running without having to run the interop broker separately - it hadn't occurred to me.

[PierreF]

No, I didn't tested them with Mosquitto. But the test_mqtt5 with interop broker seems to have good coverage of MQTT 5. That still on my todo, but I don't think it's mandatory before merging this PR.

[ralight]

I've run test_mqtt5 against Mosquitto and it all seems fine, apart from a few issues with Mosquitto which have been highlighted and mostly fixed.

Having said that, I'm seeing some failures in test_client_topic_alias and test_session_expiry when I run the whole suite but that run fine on their own.

[icraggs]

@ralight there are likely still some timing foibles in the tests, which I think are down to the way the tests are written rather than the client or server behaviour errors. I found stopping and restarting the background thread loop often quite unwieldy for the tests I wanted to write. Maybe there's a better way.

[icraggs]

And @ralight, while you're here :-) I wonder if you have a view on this:

In MQTT 5.0, the session cleanup is governed by the session expiry property, which defaults to 0. This means that if you want to clean up at the start of the session, but be able to reconnect and continue if there are communication breaks until your work is done, you can do it in one go:

connect clean_start=true, session_expiry=99999 (value in seconds)

disconnect also has a session expiry property defaulting to 0, so when you issue a normal disconnect, then the session is cleaned up immediately.

So I think we could do with some setting which means "use clean start on the first connect, but not on subsequent re-connects". This could be implied from the session expiry value and session present connack flag, although that might be trying to be too clever?

[ralight]

@icraggs I agree, the current reconnection process of using the same flags as set at the initial connection is limiting. I also agree that implying it from the session expiry value would be being too clever. I don't know what the best way of presenting it though, perhaps a further argument to the connect() call?

[icraggs]

It might be that the natural default should be that clean_start=true only applies to the first connect and not to subsequent reconnects, as that is what I think most people would want most of the time. Then the other option could be clean_start_always or similar, defaulting to false, and not allowed together with clean_start. Or clean_start could take the possible values [True, False, FIRST_ONLY] in some way, defaulting to FIRST_ONLY.

I think that these should be on the connect call rather than the constructor, as the connect properties are on the connect also (as I have implemented it). The only downside as far as I see it is that this could be a bit confusing for people migrating from V3.

[ralight]

I'd support a three value option, and think it would be better in the connect() call. clean_session was in the constructor because it applied to the whole session, that's not necessarily true any more.

[icraggs]

I've added the three value clean_start option, and put it on connect rather than the constructor. I've also added documentation for the MQTT v5.0 functionality. So I think that's all done, apart from the question of making the V5 tests run with tox only.

[icraggs]

@PierreF I've moved the test broker startup to tox.ini, so the tests work when you use tox outside of Travis too. The only situation that won't work is if one of the ports the test broker (like 1883) is already in use, probably by another MQTT broker like Mosquitto.

So that's all done from my point of view.

[fpagliughi]

Would it be possible to maybe accept this PR into a new 'mqttv5' branch (or something like that)?

Doing so might make it easier for other people to find, test, comment upon, or possibly fix and enhance.

[PierreF]

@icraggs as discussed by email, the last point I wanted to improve was to make easier to have callbacks that works for both MQTTv3 / MQTTv5.

The changes were rather minor, so I pushed commit on your branch. I hope you don't mind.

The changes are indented to make on_connect/on_subscribe easier to works with both MQTTv3 and MQTTv5:

• rc (and interger) / reasonCode (a object) could be compared to interger. So rc == 0 and reasonCode == 0 is valid
• Suggested to add properties=None as last parameter. So the callback could be called without properties (MQTTv3) or with properties (MQTTv5)
• Change to always provide a list of reasonCodes in on_subscribe. Since reasonCode are comparable to integer, it will behave like granted_qos (reasonCodes == [0] will works)

If my changes looks good to you, I'm going to merge it into master.

[PierreF]

Added "six" in test dependency... we use it since addition of pytest IIRC. I think some 3th party dependency (I think to pylama) required six but now only require it with Python below 3.5.

[icraggs]

@PierreF the changes look good to me. Thanks!

[icraggs]

Thanks @PierreF !