A new multi command protocol

[WoodsterDK]

The project as it is already supports the existing protocols, and therefor also the different software that uses them, but what about making a high performance WIFI command protocol, that supports multiple actions in one frame.
This will remove some WIFI overhead.

This can be done using different methods:

• use the same http interface, "daisy chaining" the commands
• adding mqtt as a transport layer and then "daisy chaining" the commands into a JSON.

[sidoh]

Obviously there's nothing hyper optimized, but you can:

(1) Chain multiple settings or commands in a single HTTP request. I tried to make the API truly RESTful. So you can do, for example:

$ curl -vvv -X PUT --data-binary '{"brightness":100,"saturation":20,"hue":200,"status":"on"}' esp/gateways/0x118D/rgb_cct/1
* Hostname was NOT found in DNS cache
*   Trying 10.133.8.158...
* Connected to 10.133.8.158 (10.133.8.158) port 80 (#0)
> PUT /gateways/0x118D/rgb_cct/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 10.133.8.158
> Accept: */*
> Content-Length: 58
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 58 out of 58 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 4
< Connection: close
< Access-Control-Allow-Origin: *
<
* Closing connection 0
true%

(2) There's also an undocumented feature of the v6 UDP protocol that @Lstt2005 found/requested. Basically you can set multiple things in the same packet:

[MiLightUdpServer port 5987] - Handling packet: 80 00 00 00 11 00 23 6C C2 00 3F 00 00 08 00 46 BB 64 00 01 00 AD
Packet size: 22
session: 0023, sequence: 194, group: 1, checksum: 173
Command cmdType: 3F, cmdHeader: 00000800, cmdArg: 46BB6400

This is basically capable of setting any valid combination of settings in a single UDP packet. I implemented it in the udp_presets branch. Hoping for @Lstt2005 to test it out, but I'll merge that and the LT8900 stuff into a version branch soon.

Both of these options are a little more bloated than something that was hyper-optimized for latency and performance, but they seem to work pretty well.

[sidoh]

Do you feel like either of these options accomplish what you want?

[WoodsterDK]

1. Yes, I saw that, and it gives great flexibility with one group/bulbtype.
2. Haven't see this before - any link to documentation.

Still the optimization should be sending commands for multiple groups/bulbtype in one frame.

[WoodsterDK]

Staying within the same bulbtype different deviceIDs can be turned on off in almost complete sync. and if
sticking to the same command type, e.g. on/off, color, white etc. they look to be connected to the same group.

I'm aware of that you can combine up to deviceIDs in each bulb, that could be a solution also, but then these must be linked and handled in the right order by the upper software layer... in my case a C# service.

[sidoh]

Ah, yeah. I can imagine it being important to support multiple groups or bulb types in some circumstances. You can't do that with the REST API (although you could use 0 for :group_id and it would control all bulbs in a group).

lol, no. I don't think documentation for this protocol exists. I was just looking at the raw packets coming in. I think this probably has the same limitation, though. Only supports a single group (or all groups) for a single bulb type.

I think if we wanted to do this, I'd prefer to adapt the REST API. While I love the idea of a protocol that's more performant, I think I'd prefer to not support a fourth protocol (HTTP, v5 UDP, v6 UDP being the other three). Do you feel like that'd work?

Could maybe do something like

$ curl -vvv -X PUT --data-binary '{"brightness":100,"saturation":20,"hue":200,"status":"on"}' esp/gateways/0x118D/rgb_cct/1,2

This seems pretty sane to me, actually. Could potentially even do the same thing for all of the other URL parameters, but that starts to feel a little dicey.

[sidoh]

I should say that nominal documentation for the UDP protocols can be found here, but it's really sparse and even erroneous in some places.

[WoodsterDK]

The rest way seems ok, but if it could be extended to support multiple deviceIDs, together with different groups, then we are getting closer.

[sidoh]

Yeah there's nothing preventing the same pattern from getting used for the other URL tokens. For example:

esp/gateways/0x0000,0x0001/rgb_cct,rgb/0,1

It does start to feel a little janky at this point, but I'm not sure other protocols would feel better.

You're right, though, it would feel weird to support multiple groups without supporting multiple device IDs.

[WoodsterDK]

esp/gateways/0x0000,0x0001/rgb_cct,rgb/0,1
would work.... : )

[sidoh]

Cool.

This is a good idea. Thanks! :)

[WoodsterDK]

This project/tool/gateway just gets better and better...... excellent

[itProfi]

Can any chance make support (maiby in library?) for support miltiple gateways in MiLIght app V3 (in tabs of gateways) works with one ESP - maybe on other UPD ports? Very cool and comfortable application for bulbs and LED strips..

[sidoh]

@WoodsterDK -- woohoo, that's the right direction! :D

@Lstt2005 -- this probably warrants a separate ticket. :) I did make an attempt at figuring out how to do that, but I don't think it's possible.

[itProfi]

this probably warrants a separate ticket. :)

:)

I did make an attempt at figuring out how to do that, but I don't think it's possible.

:((((

[sidoh]

Took a stab at this in the multi_protocol branch. This format should work:

esp/gateways/0x0000,0x0001/rgb_cct,rgb/0,1

It's not as performant as I'd hoped, but I think that's just because the NRF is slow. Maybe you'll have more luck with the LT8900. :)

[WoodsterDK]

Interesting, will have a chance to test it tomorrow.

[WoodsterDK]

It does not seem to work... tried:
http://10.0.0.115/gateways/0x1234,0xE761/rgb_cct,cct/1,4
not luck, changed to:
http://10.0.0.115/gateways/0xE761/cct/4
then it works....

[WoodsterDK]

Regarding performance..... is it the http part, or the decoding of the parameters that is slow do you think?
I noticed MQTT was mentioned, if the patternhandler is flexible, then it can also decode the frame when the same info is received as a MQTT message, then more or less is only one protocol, just two "medias" http or MQTT.

[WoodsterDK]

Ahh. In doubt if I chose the right branch.
Im busy with work the next week or so. Will get back then.

[sidoh]

Yeah, it's still living in the multi_protocol feature branch. Wouldn't work outside of it :)

The HTTP part is actually quite performant considering it involves opening a TCP connection and all that jazz. The part that feels sluggish, I think, is sending the RF packets to a bunch of different devices.

re: MQTT, I'm not opposed if it provides value, but I've had a hard time understanding why MQTT (async by design) is a good fit for something that controls your lights (which feels like it should be synchronous).

[WoodsterDK]

Perhaps skipping the nRF will boost performance feel.

[sidoh]

Skipping the nRF meaning using an LT8900 instead? I hope so, but I still don't have one to test out. :)

[WoodsterDK]

Still cannot get it to work - tried with latest v1.2.0.
http://10.0.0.115/gateways/0x1234,0xE761/rgb_cct,cct/1,4
with body:
{ "status":"off"}

[sidoh]

Do you see errors? Could you maybe compile with DEBUG_PRINTF and verify that it's trying to send multiple packets?

I've tested this pretty extensively with multiple bulb types, groups, IDs, etc., and have had no issues.

[WoodsterDK]

Got it working... the hub got a different IP for the first time in ages.... so right IP and it works... doh.
See what you mean regarding responsivenes.... (if that's a word).
My example is targeting two different bulb types, and that will of course make some extra register settings.
Have you tried feeding the "command" either by serial or MQTT to see if it makes any speed difference?

The solution as is works, and it is only a look wise problem.

[sidoh]

I'm quite sure most of the time spent in each request is spent sending the RF packets and/or switching RF configs. I added some timing printfs to verify this.

Testing multiple commands out of band from the HTTP parsing stuff is a good way to verify this, though. I'll try that within the next few days.