tuya-mqtt publishes states only when app is opened.

[jellekpsn]

Describe the bug
First off, I have a feeling this is due to my device and not tuya-mqtt but I thought you might know what's going on.
I'm loving tuya-mqtt and it's working well, so thank you for this great piece of software.

I'm using a power monitoring socket (Elivco 16A). When I'm in the SmartLife app, all the data (e.g. power usage DPS 19) is being pushed nicely to my mqtt broker (refresh rate approx. 5 sec). However, when I close the app, or even look at another device in the app, the power usage is no longer updated. Even when pushing 'get-states' every 5 seconds, I'm not getting back updated results. Only when I open the device in the SmartLife app it updated with current values. Any idea / workarounds how to get real time data?

Expected behavior
Power usage changes to bu pushed to the mqtt broker in real time, even when app is not in use.

Desktop (please complete the following information):\

• OS: [Raspberry OS]
• Version: master branch

[tsightler]

Can you clarify the behavior when you push "get-states"? Are you getting results but they are just the stale/previous values, or are you getting no values at all when you send the "get-states" command? This command forces a re-query of each DPS value for the device, but only if you've created a device template. Can you publish the device template you are using?

[jellekpsn]

I'm getting the stale values. I'm also getting the stale values when I use the tuya-cli:
tuya-cli get --id bfe70c5ef52b769xxxxxxx --key 5e546376xxxxxxx --all true
This is my devices.conf:

[ { name: 'Plug1', id: 'bf00e45f3007xxxxxx', key: 'd4f7b83d65bxxxxx', type: 'SimpleSwitch' }, { name: 'Plug2', id: 'bfe70c5ef52b76xxxxx', key: '5e546376c6xxxxx', type: 'SimpleSwitch' }, { name: 'Plug3', id: 'bf2a17e8f775e3fxxxxxxx', key: 'd6d87f76f1dxxxxxxx', type: 'SimpleSwitch' }, { name: 'Plug4', id: 'bf9c6a3c1500fxxxxxxx', key: 'd0e700f54bxxxxxxx', type: 'SimpleSwitch' }, { name: 'Plug5', id: 'bf2fed75ae3927acxxxxxxx', key: '6b165a0728xxxxxxx', type: 'SimpleSwitch' }, { name: 'Plug6', id: 'bf9ca13cff9f17exxxxxxx', key: '27eaabd6e5dxxxxxxx', type: 'SimpleSwitch' } ]

I appreciate the help!

[tsightler]

OK, so if tuya-cli also returns stale values I'm not really sure what can be done as it request the current DPS value from the device each time it is run, so this implies that the device doesn't even update internally. Somehow the app must be sending commands that trigger it to update.

What would be interesting is if you could firewall the device running the Tuya app from the internet so that the app is forced to control the device locally just as tuya-mqtt does and see if it works correctly. If so, you can capture the traffic between the devices and decode it with tuyadump and it might give us insight into what we'd need to do to make this work.

I'm also curious if it NEVER updates, or if it updates after say 5-10 minutes.

[tsightler]

Also, could you provide the output of one of these devices from tuya-cli?

[jellekpsn]

Seems like it updates every now and then. In this case 20 minutes or so. I Haven't touched the app since 18.30.

Here's the tuya-cli output

{ dps: { '1': true, '9': 0, '17': 16, '18': 237, '19': 344, '20': 2325, '21': 1, '22': 668, '23': 30000, '24': 17426, '25': 2390, '26': 0, '38': 'on', '41': '', '42': '', '46': true } }

I will have a look if I can get my phone (android 10) to run the app while firewalled from the internet, and look if I can get tuyadump to run!

[tsightler]

The app should work when firewalled although it may complain a good bit during first startup, but it does eventually work for me (make sure to stop tuya-mqtt first though). Basically, the app normally communicates to Tuya cloud, but if it can't reach tuya cloud, it will look for the devices locally on the same network and try to communicate with them directly. It's this direct path that TuyAPI/tuya-cli/tuya-mqtt is taking advantage of so if we can manage to get a packet capture of that local traffic, we can decode it and, maybe, figure out what it's doing. My guess is it's sending some command to some DPS key, but I have no idea which one.

[tsightler]

I did a little bit more digging on this, looked at some other Tuya projects that work with power monitoring switches, and they pretty much all have at least one issue reported like this where the power only updates rarely unless the app is opened, so I'm pretty sure there's nothing I can do here.

However, I did find at least one issue where it appeared they were making progress by sending a Type 18 command to the device which appears to get it to update. Unfortunately that means this command type would need to be implemented in TuyAPI before I could do anything with it and it's not really clear to me how that would even work since TuyAPI pretty much only offers set/get functions and doesn't provide access to low level commands. I could probably supplement TuyAPI with my own code for this, but, unfortunately, I don't think I'll have time to attempt that anytime soon.

A packet capture of the app talking to the device locally would still be really useful, if you can get it.

[jellekpsn]

Thanks for the update and going the extra mile. In the grand scheme of things it's a minor issue.
I'm struggling to get tuya dump to run. I'm not really familiar with go.. But will let you know if I succeed

[beatmag]

I've been thinking of updating this project rather than localtuya.

I've got some prototypes hacking command 18 into tuyapi already.

Let me see if I can finish localtuya. It seems to be more work than I had imagined.

[tsightler]

Thanks for the update and going the extra mile. In the grand scheme of things it's a minor issue.
I'm struggling to get tuya dump to run. I'm not really familiar with go.. But will let you know if I succeed

If you can get the trace, and are willing to send it to me and share you device ID and key, I'd be willing to do the tuyadump part to decode it.

[jellekpsn]

Sure, happy to do that. What exactly would you need from me besides the Device ID and key? And how shall I send it to you?
Which trace do you mean exactly? Sorry only recently purchased my first tuya devices so it's kind of new to me :)

[tsightler]

I would need to packet trace file itself, as well is the device ID and key, maybe the IP address of the device. You can send the file either via email (my email address is the same as my Github username, at gmail) or, if it's too large, send me a link to download it.

I'm not saying that I'd have a solution right away though, based on my current work schedule it would probably be at least a few months before I have time to actually try to code anything for this and if @beatmag already has some prototype and is willing to share the work on Github, that might be a quicker path.

[jellekpsn]

I'm new to capturing packet trace files, so not quite sure if i'm getting what you need. I'm capturing packets using wireshark on a windows PC. I see several packets from the plug's IP-address. Are those the ones you want? Or do you want the entire capture over a period of time. The capture was made while I firewalled my phone from internet access and tuya-mqtt was stopped. I could control my device normally trough the app and power consumption was refreshed every ~5 sec, although several 'not connected' error messages appeared.

[tsightler]

Yeah, so that's one of the challenges. Based on your network topology, your Windows system may not be able to see the traffic between the phone and the device. In general I would need to see all traffic specifically between your phone and the device. It should be mostly on ports like 6667/6668. You can filter out all traffic except between those two IP addresses and that would probably be enough.

[jellekpsn]

Done, sent you an email. Hopefully the traffic is in there somewhere.

[stevoh6]

I can confirm, same issue here, power stats are updated only if mobile app is open. is there any option how can I help?

[tsightler]

As this is not a bug in tuya-mqtt, and I don't see any way to address it here, my lean is to simply close the issue. If TuyAPI implements command 18 and some way to trigger it, I'd reconsider this, but I definitely don't have the time to spend reverse engineering the closed Tuya protocol while Tuya Inc. continues to put more efforts into blocking any such attempts at local control.

As far as I can tell most devices do update at least every 5-10 minutes, even when the app isn't open. If you need more granular updates than this then I don't think tuya-mqtt is going to be able to provide it in the near term, and perhaps never.

[jellekpsn]

Sounds fair enough!

[tsightler]

Closing this. Will re-open if TuyAPI eventually supports a method for polling such data more frequently.

[stevoh6]

HI @tsightler thanks for support, but can I ask about some more information about "command 18"? I see many posts about this mystery, but no specific way how to use it, make at least some temporary hack or contribute to move it forward. I struggling to find the starting point, where to began.
@jellekpsn any ideas?

[tsightler]

@stevoh6 There's no "hack" until someone develops code that can send Tuya type 18 commands in TuyAPI (or somewhere). If you look at the TuyAPI message parser near the top it defines a host of Tuya Command Types. CONTROL: 7, STATUS: 8, HEART_BEAT: 9, DP_QUERY: 10 and I whole host of other commands. These are all commands of the Tuya protocol which have been reversed engineered from capturing and decoding packets, they even have different headers and other requirements between 3.1 and 3.3 protocols which have to be dealt with in the TuyAPI message parser.

When you look at that list you see that there's no Command 18 at all, which means that, until recently, we weren't even aware of it. However, based on packet captures, it's clear that it's the command that is telling the device to update it's internal values. Whenever the device is under local control you can see a packet with the message type going out and coming back between the app and the device, but of course it is encrypted. I was able to decode it partially, but tuyadump doesn't seem to understand it, and it appears to use an old protocol 3.1 header instead of the newer header of the 3.3 protocol, even though the device I have a capture from is a 3.3 device.

TuyAPI itself is also mostly in maintanance as far as I can tell. I've contributed some code to fix or at least workaround a few issues, but, for example, it also doesnt's support command 13, which is used by some modern Tuya devices.

By far the person that seems to have done the most with this "command 18" is beatmag over in the localtuya project. It sounds like he had gone further in decoding the packet and was at least attempting to implement it. I'd personally start there: rospogrigio/localtuya#194

However, tuya-mqtt doesn't send any commands directly to Tuya devices, rather it uses TuyAPI which abstracts all the details of talking to Tuya devices and the protocol version, etc with a simple API that manages the connection and allows control with simple set()/get() commands.

The summary version of what needs to happen for this to work in tuya-mqtt is this:

1. Someone needs to fully decode the packets and understand the packet structure of Tuya command 18. It seems like beatmag has done this but I've not seen him share the information publicly It doesn't sound difficult overall.
2. Once the packet structure is understood, TuyAPI message parser should be updated to be able to send/receive these message types
3. TuyAPI will also need to be enhanced to allow it to use these commands since they are not required for the vast majority of devices. I see two ways this could be addressed:

• Implement a configuration option for the device that enables these messages to be sent at a configured interval (it already sends HEARTBEAT messages every 10 seconds so a duplicate of that function would be a pretty easy path, just with an option to enable and set the interval). This would be mostly transparent to tools like tuya-mqtt.
• Alternatively, TuyAPI could implement command 18 and provide an "update()" command and leave the interval configuration up to the consuming application (in the case of this project, tuya-mqtt)

4. Once TuyAPI has any support for any of this then tuya-mqtt would potentially need to be modified to support whatever capability TuyAPI implements.

There's probably some stuff I'm missing, but suffice it to say, while all of the above can be accomplished, it's a good bit of work, and is especially difficult for me since I have no devices that even need this and no time even if I did have a device and, since almost none of this is directly related to this project itself. If someone is passionate about it and has the skills to perform all of those tasks, they can do it, submit the PRs to TuyAPI and any required for tuya-mqtt. If it ever gets to the point where TuyAPI supports it, it would likely be pretty trivial to add to tuya-mqtt, but as I'm stepping away from this project, I can't promise anything even there.

[stevoh6]

@tsightler Thank you much for lot of helpful points :-) They really helped.

Also I found really good project and possible solution for others. jasonacox/tinytuya#8 (comment)