Improving the custom pairer to meross_lan

[agrabren]

Having jumped through multiple hoops and struggling to get everything to work correctly the first time, and dreading the fact that I need to custom pair about 60 switches, I've decided I'm going to make some improvements to a fork of the custom pairing app (and obviously work to push them back to albertogeniola.

I'd like to get a better understanding of two aspects of meross_lan as a way to improve the flow and help people get where they need to faster.

1. The meross_lan add-on does a great job of handling the required MQTT messages. How can this be verified? Given just the pairing app and a connection to the MQTT server, how can I test if meross_lan is properly configured? If this isn't possible (due to no devices being configured yet in HA), do you have any suggestions on how this could be done? I could connect up to HA itself and check for certain add-ons and/or configurations, or maybe you have a different suggestion?

2. Key recommendations: When I first started working on this, I didn't realize what the fields were in the custom pairer app. So now my key is blank, which works but is this a bad idea? Should a wizard that helps walk someone through custom pairing their Meross devices to a private MQTT (or eventually to your built-in one) strongly suggest a key? What about a user?

3. Repeat setup: Do you have any thoughts on ways we the custom pairer could improve the HA onboarding flow? Right now, you do the custom pairer, and the device gets discovered. What would be even better is if the app could collect the "extra details" like a good name for the device before pairing it. It's possible to query the room list and set that, but even just having a human understandable name would be a huge win, as once you start pairing devices, they all blur together.

I know this sounds like a lot of beginner questions, and it is. I'm a newbie on this front. But Android Apps, I'm not a newbie. I've got 10+ years of developing both Android apps and the OS itself. I have even written a commercially used custom pairing app for certain controllers and remotes, including firmware upgrade support (this was at my professional gig, so I had all the documentation)

[krahabb]

Hello @agrabren !
This is a very nice idea and I'm happy to support you so, beside this 'kickstart' feel free to ask or share anything and I'll be happy to help you.
About your points:

• You can check meross_lan is correctly 'MQTT configured' by queueing any formally valid meross protocol message on the standard meross topics (this is the wildcard topic I used: "/appliance/+/publish"). When it detects a message on that topic family, it extracts the device id (second token of the topic path) and starts an identification procedure on 2 steps: This procedure consists in sending an 'Appliance.System.All' request to the device, waiting for a correct response from the device, sending a request for 'Appliance.System.Ability', waiting for the response and then submit the discovered device to the HA configuration flow.
  So, if you inject a 'formally valid' meross message into that topic, meross_lan would publish an 'Appliance.System.All' message on the '/appliance/XXXXXXXXX/subscribe' topic where the XXXs are the same as extracted from the incoming message. This is the proof meross_lan is in place and able to detect devices. At this stage meross_lan keeps an internal state machine in order to complete the identification process. If this process aborts for any reason meross_lan will discard and reset this identification process after 5 minutes of no messages on the same topic/id so, you can 'abuse' it with no reasonable consenquences. Keep in mind meross_lan should be pushing discovery related requests until the process ends (for 5 mins then) so this could 'pollute' the mqtt queue a bit. I would consider adding a specific message identification for the pairer app to better manage this process. We could agree on this later on.
• As for the key, I too started with an empty key field and it works ok: an empty key is like any other key: the only difference is that in 'our community' the empty key might be quite common so easily guessed by an hacker who could use it to manage your devices once it gets access to your LAN. I personally don't trust that much the added level of security of the key mechanism so I'm not worried at all by the fact that my key is easily guessable and I use it since it is easy to remember (and type) when I configure my devices. Still, a 'non guessable' key might provide a slightly better security for the 'consciuos/paranoid'. I personally would leave the choice to the user providing a TextField (initially set to 'blank') and also adding a button to automatically fill in a random key string for the field. I would also save this value in the app storage so to always present the user with the last key used to configure a device
• I see the point and providing a friendly name while pairing would greatly enhance the user experience but.. there's no way we can embed this parameter in the device itself so, the next step would be to propagate this setting to HA/meross_lan by some sort of message exchange and things could start to get messy. We can think about it a bit. In this regard I'm confessing you I've considered (in the past) the idea to directly embed the pairing functionality into meross_lan: it's just a simple message exchange with the device when set in pairing mode and so it is esily achiavable without the need for separate software tools like command line scripts or mobile apps. I think this could be the easiest and friendliest option for any (HA) user and I might reconsider implementing it in the future...right now it's out of the radar though..

[timnolte]

@agrabren any luck with your work on the Custom Pairer app. I've been struggling myself with this whole setup and have a very mixed result set. I've made some notes on an issue over there regarding a lot of the problems I've had: albertogeniola/Custom-Meross-Pairer#10

Most notably, when using the Custom Pairer App, I have ended up with some of my devices somehow connected/paired to Meross LAN via IP directly and some of them paired using MQTT. I'm a little baffled how I managed to pair them with HA/Meross LAN over IP alone but they are working, with some limited functionality(notably the LED indicator control is missing), so I can still at least manage those devices, however they are not using MQTT. I'm not sure if this is a firmware issue/difference with the MSS110 smart plugins & MSS550 smart switches, which are the 2 device models that will only pair via IP. The MSS550 switches return missing data in the Custom Pairer app which leads me to believe that, like the Node Meross Utility, there is just some incomplete/incompatible functionality with some of these devices.

I had sort of started down the road of sticking with Meross because they seem solid enough devices however, as I've also now been in pursuit of moving everything local I'm almost considering changing course and going with Zigbee since there are such a wider range of devices that have more solid support with Zigbee2MQTT. Anyways, if you need someone to test or assist in any way let me know.

[krahabb]

Hello @timnolte ,
meross_lan detect devices both via MQTT (once they're paired) and via DHCP. The DHCP identification is likely 'winning' over the MQTT most of the times since HA should likely receive the DHCP discover/assign frames before anything else but this is not deterministic anyway.
At any rate, once configured, they should really behave the same in terms of functionality.
The differences are (when protocol set to 'auto' or empty in meross_lan configuration):

• meross_lan 'prefers' MQTT when talking to devices discovered via MQTT. Prefers means that if MQTT fails (after few seconds timeout) meross_lan switches to HTTP communication (the ip address of the device is extracted from one of the MQTT payloads carrying general device setup). Once MQTT is active (i.e. meross_lan receives any MQTT message from the device) MQTT is the preferred protocol again.
• meross_lan prefers HTTP for devices configured via an ip address (those discovered via DHCP or manually configured). In this case meross_lan prefers to use HTTP but will switch to MQTT in case HTTP fails. This failover switch will only occour if any message has been received from the device over MQTT (this means the device is correctly mqtt-paired and so able to talk over the protocol)

Keep in mind that even if meross_lan is 'preferring' HTTP it still receives and handles MQTT messages so that status updates are reported instantly when they happen (so you shouldnt experience any delay involved by polling the device).

All of this failover switching and simultaneous MQTT 'sensing' (when preferring HTTP) are valid only for AUTO protocol. If the protocol configuration field is set, it is set!

In some way, having the device configured via DHCP/ip address is slightly better in terms of performance since HTTP transactions are faster than MQTT. If the device is paired to the MQTT broker you too have synchronous updates via MQTT so you have the best of 2 worlds.
The only drawbacks for this setup are:
-should the device IP change you loose the HTTP capability (but meross_lan should failover to MQTT so no disruption)
-HTTP communication is less reliable (in my experience) with some disconnections or rejects from time to time (my guess is the demon inside the device have some trouble sometime..but I can't really tell). This is managed by meross_lan by retrying an HTTP attempt 3 times and then switching to MQTT. This could sometime delay your commands (think when you switch on/off from the HA UI..you could really see the delay sometime..or the command completely rejected). This kind of 'failed commands' never happend over MQTT in my experience.

As for the LED indicator, as I said, there's no reason it's missed when you configure them over IP. The DND light (if this is what you're referring to) should be accessible at least from the device page under 'configuration' entities.

[timnolte]

@krahabb so here is my observation at this point.

After multiple tries I was able to finally get one of the MSS110 smart plugs to pair via MQTT versus IP. What you explained about the DHCP discovery being faster and "winning" over the MQTT makes sense in this case. However, what I am seeing in the case of the MSS550 smart switches is that it seems as though the Custom Pairer app isn't properly configuring the MQTT on the device, but is configuring the WiFi so the device is getting itself connected to the network and then just being discovered via DHCP. If I attempt to force the device to MQTT after it is registered with Meross LAN it just shows up disabled, because like I said I'm pretty sure the device isn't properly paired to MQTT. I'm not quite sure what it is about the MSS550 switches but the Node-based Meross Setup Utility can't even properly communicate with them, or the MSS510 for that matter. The Custom Pairer app does seem to work with the MSS510 however.

The other aspect in all of this is that last point about the LED(DND mode) in that when the devices are paired via MQTT to Meross LAN I do get the DND-mode/LED entity control available, however, the 2 MSS550 smart switches that paired via IP only are missing the DND-mode/LED entity control. I'm about 99.9% sure that the switches properly had the control entity when they were originally paired via the Meross Cloud, I know for a fact it was an option in the Meross on those switches. I'm going to mess around with one of the MSS550 switches and see if I can get manage to get it paired via MQTT. I wish there was some way to prevent the DHCP Discovery from happening so that I could force it to MQTT as the primary.

[krahabb]

Well, nothing comes to mind as for why the DND light is not available...
The only option is that the device is not correctly identified from meross_lan, so it lacks some (or most) of its functionality.

You could try download a diagnostic (from the device page) and upload it here so I could check what's going on.
Anyway, do the other part of the MSS550 work ? i.e. are the switches available to use in meross_lan?

[timnolte]

@krahabb so I'm not sure why but any time I attempt to download the diagnostics from any of the Meross LAN integration/devices I'm faced with an error of Failed - Unknown server error. Please try again, or contact the server administrator. and then a "Resume" button which, if clicked, results in an error 401: Unauthorized. I don't have any problem downloading diagnostic data from other integrations/devices.

[timnolte]

@krahabb also concerning the DNDMode entity, I can confirm that this lack of entities is a result of the direct local IP pairing as the same devices when using the Meross Cloud over IP have the DNDMode(LED) available. It's also possible that this could be due to an issue with the Custom Pairer app and how it's attempting to pair the devices with HA/Meross LAN. In general I have problems with both the Custom Pairer app and Node Meross Setup Utility with the MSS550 smart switches. The Node Meross Setup utility can't connect/pair either the MSS510 or the MSS550. The Custom Pairer app is at least able to pair the MSS510 switches via MQTT but the MSS550 switches end up only being configured via IP/HTTP to Meross LAN.

[krahabb]

@krahabb so I'm not sure why but any time I attempt to download the diagnostics from any of the Meross LAN integration/devices I'm faced with an error of Failed - Unknown server error. Please try again, or contact the server administrator. and then a "Resume" button which, if clicked, results in an error 401: Unauthorized. I don't have any problem downloading diagnostic data from other integrations/devices.

This is happening to me too when I access the HA UI through the nabucasa reverse proxy. The reason is unclear but, since the meross_lan diagnostics are collected over a time frame (i.e. they're not available yet as soon as you click the download button) it looks like the HA frontend UI is not able to manage this when routed through nabucasa. Instead it works ok if I connect to HA UI locally: when I click it, the download will not start immediately but (depending on the diagnostic length configured in meross_lan) no error will be reported and the download will be carried on later when the tracing is finished.
Is this also your case or you're experiencing the issue also when accessing it locally?

[timnolte]

Yes, this is happening locally for me. However, I do have the Nginx SSL proxy running, and access my HA instance local using that. I will try to connect direct without the proxy and see if it works that way.

[timnolte]

@krahabb OK, I've confirmed that this does appear to be an issue with attempting to download the diagnostics through the Nginx SSL Proxy. When I access through HTTP on port 8123 the diagnostics download successfully, they take forever to generate though. I'm feeling like I should be moving this all to a new issue as it seems to be hijacking the original Custom Pairer topic a bit.

[krahabb]

Yep, that's because the logs contain a tracing feature of all of the protocol commands supported and this is started when you ask for diagnostics.
This is lasting 10 minutes by default but you could reduce the time in meross_lan integration configuration. Actually, the real time it takes to query all of the commands is usually far less than 10 minutes but the default duration was originally set in order to allow the user to do some basic interactions with the device through HA or the app and allow the collection of all of the messages exchanged (here a deeper explanation about the overall diagnostic features available in meross_lan)

[timnolte]

@agrabren in an attempt to bring this back around to your topic. I'm wondering if you've also found cases where your devices are being paired with Home Assistant sometimes via IP/HTTP and sometimes via MQTT?

Would you be able to see in the Custom Pairer app how it is going about configuring the devices as my experience seems to indicate that some of my devices are getting their WiFi settings configured but the MQTT connection is not being updated/established and so basically the device is really only in a local polling only mode with Home Assistant.

Another bit of information that is missing in the Custom Pairer app is the device username & password, like is listed with the Node Setup Utility. This would be key information to have so that these credentials could be configured in the MQTT/Home Assistant properly. I'm thinking that with some of the newer HomeKit enabled firmware on these devices that something has changed an some of the information is not available when querying the device like it has been on older devices.

[timnolte]

@agrabren just noting that if you are making improvements to the Custom Pairer app that there are some apparent issues the newer HomeKit versions of the Meross devices. I found a relevant thread where other people have seen this problem. @krahabb so this seems to be the root of the problems I'm having.

[agrabren]

Sorry, got pretty busy lately and I'm still learning the details myself. My house is a mix of MSS510s and MSS550s. I believe that I have successfully paired most of them with MQTT, but now I'm questioning it so I'll need to dig into more to identify exactly how my different devices are connected. At present, I have 47 of them, but two of them are "broken" in one way or another that I haven't had the chance to really dig into. One thing I've noticed is that my MSS510s all pair as outlets, which is mildly frustrating. But that's definitely not as important to me as getting everything working. Either fortunately or unfortunately for me or others, I intentionally did not buy the homekit versions (they would have increased the cost and homekit annoys me when I use an iOS device for pairing). So all of my devices are the "old" style.
My hopes on improving the pairing app are mostly around the gathering and then programming of the data to each switch. I can't really picture that anyone deploying a bunch of Meross switches is actually pairing them to different wifi networks or MQTT servers. And it really could use a "wizard" to walk people through. Figuring out what the different fields meant and where to plug them in was also less intuitive than it could be. That's why I'm hoping to to make it a little "smarter" with verifying the MQTT server is functional and that meross_lan is responding. I would have saved myself days of struggle with a simple walk-through of the process.

[timnolte]

@agrabren yeah, I was fortunate that I got a combo pack of mss510 switches and mss110 plugs that are not HomeKit versions and are working as expected (I too see the mss510's as "outlet" entities). If I could find and replace my msg100 garage door opener and the mss550's with non-HomeKit versions I would but they seem to be unavailable now. I've been tempted to just find ZigBee replacements at this point, though it would be great if the pairing apps could work with these newer devices.

[agrabren]

@timnolte It sounds like there has been a protocol change for the new versions, which isn't surprising with the new features required for HomeKit. When I get a chance, I'll probably order a plug (as I don't want to do a bench setup to work on a wall switch at my desk) to test with and possibly integrate any fixes (or push fixes) to the lower-level communication protocol code.

[timnolte]

@agrabren yeah, I was thinking the same that I might pickup a HomeKit compatible outdoor plug, or maybe a desk lamp, but something that I can easily get power to and test without messing with some setup with and actual wires light switch. I'll share anything I find in the event that it helps.

[timnolte]

I wanted to follow up with something that is interesting. I decided to take one of my HomeKit compatible switches and remove it from Meross LAN and set it up via the built-in HA HomeKit Controller integration. It was easy enough to add, without the need of an Apple device or the cloud. However, I was met with mostly the same lack of the DND/LED control functionality. This sort of confirms for me what I've heard about HomeKit devices is that you may not get the full functionality from HomeKit that you might with a more "native" integration. I did get an Identify entity that will let me cause the LED on the switch to flash, so that is extra. I did lose the signal strength entity that I had with the Meross LAN IP integration. Toggling the phycial switch and watching Home Assistant it seems like the switch is talking directly to Home Assistant and it very responsive. I've been struggling with the garage door opener being a bit lagging with updates in Home Assistant because it's not able to connect via MQTT, so I'm tempted to remove it and reconnect via HomeKit, though it still feels a bit gross to be using HomeKit to be honest. I don't hate Apple I'd just rather not get imprisoned by their tech and pay their crazy prices for things.