No datapoints attributes when the plug is disconnected from Internet

[D33M]

Maybe I do not understand it correctly...

I have Google Nest Wifi, in which I can enable/disable Internet access for devices. Since I have successfully managed to move to the localtuya, I believed that the communication is now done locally, without the need to access the Tuya API cloud. This is why I disabled the Internet for the Tuya smart plugs.

After I've done this, I can still power on/off the device, but no extra attributes are displayed in the Home Assistant - only the last value is stored (no updates). If I enable Internet access again, localtuya correctly updates attributes from data points.

Are my assumptions correct, or the values are being downloaded from the Tuya cloud API...? Maybe this is some weird bug or Google Nest WiFi router blocked some ports for the device...?

Looking into the debug logs (device 753802832462ab39a83c), only dps "1": true i passed, if there is no Internet connection:

Internet disabled

2020-11-28 00:10:38 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Send payload: b'{"devId":"753802832462ab39a83c","uid":"753802832462ab39a83c","t":"1606518638","dps":{"1":true}}'
2020-11-28 00:10:38 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Waiting for sequence number 44
2020-11-28 00:10:38 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Dispatching message TuyaMessage(seqno=44, cmd=7, retcode=0, payload=b'', crc=1364689567)
2020-11-28 00:10:38 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Dispatching sequence number 44
2020-11-28 00:10:38 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Decrypted payload: {}
2020-11-28 00:10:39 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Dispatching message TuyaMessage(seqno=0, cmd=8, retcode=0, payload=b"3.3\x00\x00\x00\x00\x00\x03\x8b\x99\x00\x00\x00\x01\xe9\x1che/\x8f\xef\x16h\xf9\xb6\xe6E\x1b\r\xc2#|W\xa5W\xb0O\x87d\x80\xc8lE\xc45\xbd\x05`X~\xdf\xae\x81\xb6_\xca~\x97\xb5\xbf\x00\xfb\xba2:c\xf5\xac\x98%)-\xa1\x18\x83|\x19:\x1eDyD\xa2)\x990\xfd*A'R\xa7\xd7\x9c", crc=1123560208)
2020-11-28 00:10:39 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Got status update
2020-11-28 00:10:39 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Decrypted payload: {"devId":"753802832462ab39a83c","dps":{"1":true},"t":1606518638}

Internet enabled:

2020-11-28 00:14:35 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Got status update
2020-11-28 00:14:35 DEBUG (MainThread) [custom_components.localtuya.pytuya] [753...83c] Decrypted payload: {"devId":"753802832462ab39a83c","dps":{"4":85,"5":192,"6":2258},"t":1606518875}
2020-11-28 00:14:36 DEBUG (MainThread) [custom_components.localtuya.pytuya] [604...ff5] Dispatching message TuyaMessage(seqno=0, cmd=8, retcode=0, payload=b'3.3\x00\x00\x00\x00\x00\x04\xe8\xa0\x00\x00\x00\x01V:\xa8\xb2^h\x12\xe0o\x87f\x93Z\x1b\xed\x0f]\xd4od\x925\x8b\xf4\xac$e\xef\xc3\xbd\x08\x8c\x00\xf5+\xe7I\xa4!\xdf}|\x84\xd89\xd1\xd4\x1d\x0c\xd6\x15v36\x90\\\xd3\x16X\xc3]\x87\x8e\x88c\x8a\xa2\xf3\xe0Z$H\x06R\x17#\x02\xf0O\xa2', crc=414067488)

[KTibow]

#195

If you plan on integrating these devices on a network that has internet and blocking their internet access, you must block DNS requests too (to the local DNS server eg 192.168.1.1). If you only block outbound internet then the device will sit in zombie state, it will refuse / not respond to any connections with the localkey. Connect the devices first with an active internet connection, grab each device localkey and then implement the block.

[D33M]

@KTibow why does this work like that, if I can ask? Does the device ping the DNS server, which then correctly returns the IP of the cloud server - and waits infinitely since it has no connection to it?

I wonder how to set up my router to allow the local connection, but block the DNS, probably I can't... :(

[KTibow]

Don't know. Just quoting someone else.

[llluis]

Did anyone succeed on using it without internet access?

[D33M]

I guess the best idea is to set up a Pi Hole, then set it up as the main DNS server for the router and block the plugs in the iptables...?

This way plugs will get the connection refused error when contacting the DNS server.

[llluis]

My DNS is already blocked. Even if I just unplug the router from the modem, it's still the same behaviour.
It simply doesn't work without internet.

[cjj25]

I'm the author of the original comment on the README. I experienced the same problems you are facing now with my tuya plugs on the latest fw. I spent far too long trying to get these to work locally while blocking outbound access and found the only solution was to block (silently) the DNS too.

I use PfSense as my router / network firewall and this offers far greater control over the devices. It's important to note the difference between "block" and "reject" here, a block will drop the packets silently and simulate a full offline experience as if your internet was down. Whereas a "reject" will return a packet back to the tuya device. The only way for my devices to work locally was "block", then after around 3-5 minutes of booting I could control them via the local network.

Using third-party DNS servers such as pihole and adguard may not have the desired outcome as they typically remap the DNS entries to a local 127.0.0.1 (and return a response). This isn't the offline behaviour you're wanting to prevent this zombie state. The device needs to think it's completely offline for it to work as desired (in my experience).

However, in some instances I did need to allow them access to the internet and then initiate the block. Very frustrating but it does work.

In the end I cracked my plugs open using some silicone wrap and a bench vice to avoid damage and flashed Tasmota as I wanted more control over the devices.

[vinhdizzo]

@cjj25 Hi, I am following up on our discussion since you mentioned that you made a more detailed explanation here.

I was able to get Adguard Home up and running and assigned it as my DNS server on my router. I did set up a filter to "block" all DNS requests from my tuya devices. Block is the term used by AdGuard; I'm not sure if it is the same "block" as your description. After less than an hour, I see many (1,000) DNS requests from the Tuya devices, and AdGuard shows they are all blocked.

I have a question. You use "zombie" state to describe the Tuya devices when DNS is not blocked. Is this the same as what is described here (tuya devices showing up as "unavailable" in HA)? If so, then perhaps DNS blocking is not necessary when that latest fix is rolled out? Please let me know if I am confusing two separate issues as one.

Thanks.

[cjj25]

@vinhdizzo I'm not entirely sure and unfortunately I have no way of testing this now as my devices are now flashed with Tasmota. The zombie state is aimed more towards the device going unresponsive and/or no longer providing data points (energy consumption etc)

[beatmag]

i've been experimenting with tuya smart sockets. kogan ones, they are likely sp22 ones.
So I have blocked the internet off, blocked the dns port off. full drop packets (ie ignore traffic).
I am seeing that the plugs still attempt Dns , and still doing multicast on 6667.

The power switches dont work, until i use the tuya smart app to try them. the surprising thing is that the app can some how make them work. This is on a wifi with no internet, dns blocked as well. 4G turned off. hence the app has no internet access either. Is the app doing something different to what we are doing? For some reason the app is able to some how enable them, and then they start to work in localtuya.

Internet Requests Dropped, DNS requests Dropped (observations)

1. Switch returns no power data at all. (local tuya then exceptions, if add switch only with no sensors it worked for a bit)
2. Switch does not discover correctly
3. Switch timesouts when local tuya attempts to access it.

I tried downgrading back to local tuya 3.1.0 as there was some reports that 3.2.0 caused timeouts.
Nothing is working. The only thing to work is to get the app to some how activate the switches!!!
This is whilst its in the zombie like state where it doesn't respond at all.

Switch always auto discovers though, so the broadcasting from the device is still happening.

Are there any local tuya implementations known to work without internet? as we can investigate what is different?
From what I can tell, the App is doing something...........to the switches to activate them locally.

[andrus2049]

The power switches dont work, until i use the tuya smart app to try them. the surprising thing is that the app can some how make them work. This is on a wifi with no internet, dns blocked as well. 4G turned off. hence the app has no internet access either.

AFAIK the Tuya app requires you to log-in for using it, so some Intenet access has to be available if you are using the app!

Anyway, the Smart Life app is better than Tuya.

[vinhdizzo]

Reporting my findings here. I ended up upgrading local tuya to 3.2.0, removed all tuya devices in HA, removing all tuya devices in the Tuya App, and started from scratch.

I can confirm that once I block DNS via AdGuard Home to my tuya devices, the devices eventually become "Unavailable" in Home Assistant. I believe this also happens if I block internet from my router via Parental Controls.

After I removed the DNS filter and restarted AdGuard, the devices are now online in Home Assistant.

My conclusion is yes, the tuya devices somehow need access to the internet to work with HA via local tuya 3.2.0.

[andrus2049]

I had to install the TuyaLocal integration v 3.2.0 a few days ago because in my area there was a wide and prolonged Internet connection outage, and for about 3 hours my tuya devices worked correctly w/o Internet access (I have only switches).

Many (if not all) IoT devices actually need access to a NTP server, I see this in my router logs. I have a local NTP server (based on GPS) so maybe their real need for Internet is to get a valid date/time value... and this might explain why my devices worked w/o Internet.

[beatmag]

The Tuya App is able to make the Switch work again, with Energy Datapoints appear again.
It was able to do this whilst having no internet at all.
Yes, first time adding switch to the Tuya App, you need internet.

No restarts were required. The Tuya app is doing something , to make the switches respond again..
Is anyone able to capture those messages??

[andrus2049]

It was able to do this whilst having no internet at all.

Are you able to start and use the Tuya app w/o Internet and after removing the running app from the device memory (or after restarting the smartphone) and also after having cleared its data/cache ?

[beatmag]

So I've got some captures of the messages from the Tuya App to the Kogan Energy Monitor App.
This is protocol v3.3. So it looks encrypted.

Does anyone know where I can get some information on how to decipher the messages?
The app appears to have made 2 message, and got 2 replies from the switch.
I'm pretty sure the app, is sending something to the switch to kind of request for Energy Data.
Without this request the switch only replies DPS 1 which is switch on/off.

If we can decipher the messages, we can make tuya local perform the same messages and obtain the energy stats.

[KTibow]

@beatmag this might help you: https://github.com/rospogrigio/localtuya/blob/master/custom_components/localtuya/pytuya/__init__.py
You could try also making a new issue.

[beatmag]

Good news everyone. I got the messages deciphered!

Here is the summary:

App -> Switch
commandByte: 10 DP_QUERY
{ devId: '8xxxxxxxxxxxx', gwId: '8xxxxxxxxxx' }

Switch -> App
commandByte: 10 DP_QUERY
raw data:json obj data unvalid

App -> Switch
commandByte: 18 (WHAT IS 18??????)
{ dpId: [ 9 ] }

App -> Switch
commandByte: 18 (WHAT IS 18??????)
{ dpId: [ 18, 19, 20 ] }

Switch -> App
commandByte: 8
{
devId: '8xxxxxxxxxxxxxx',
dps: { '18': 0, '19': 0, '20': 2400 },
t: 809
}

So a few issues:

1. Command Byte/Type 18. No idea what 18 is.
2. dpId ?? Is this the same as DPS? Its requesting for 9 and then energy monitor 18,19,20, in 2 seperate calls.
3. Response comes back as commandByte 8 (STATUS).
4. DP_QUERY is not recognised and rejected. (This is related to the other invalid JSON error, its device specific I guess.)

This is the Kogan https://www.kogan.com/au/buy/kogan-smarterhome-smart-plug-energy-meter-5v-24a-usb-ports-4-pack/
These have been updated so you cannot use Tasmota Convert.

I will try to use some low level api lib to try and reproduce those calls soon.
I've been trying on tuya-cli. But it seems its expecting commandbyte 18 in the response.

I have confidence we can fix this issue now.
By the way, internet is fully blocked on this WIFI I'm using for these test and packet capturing.
So the problem is not Blocking the Internet, the problem is the device behaves differently to expected from the code. And when the internet is on, it behaved in the expected way normally I guess.

[postlund]

@beatmag That's great work! DP 18 seems undocumented, at least in tuyapi (see here), so I guess we'll have to make up a new for now.

So to answer your questions:

1. As said above, it seems to be unknown so we'll make something up.
2. dpId is a list of datapoint ids, most commands accept that.
3. This is normally how "push updates". So the command likely just requests an update to certain datapoints and the values are then pushed back.
4. Different devices (firmwares) support different things, that's why we have multiple methods to query datapoints for instance.

One follow-up question I have though is, how often is DP 18 sent? Is it only done once per connection or periodically? Maybe it's used to enable certain datapoints, or at least push updates for them?

[beatmag]

I'm still looking into it. For some reason I have yet to get it working, I'm building a tuyapi node test console app.
I have yet managed to get a CommandByte 8 return with the energy value.
I've tried sending the Command 18 across just like the packet capture.
I need more time, to get some more captures and try reproduce in sample app first.

It definitly looks like the device does'nt accept QUERRY_DP sometimes, but sometimes i've seen it reply correctly to QUERY_DP.
Its abit strange.

[beatmag]

hi all,

I think I have figured it out. The Command 18 message request cant use the extended 3.3 header. It needs to user the old 3.1 no header.
After making that change. It started to work.

From my testing Command 18, appears to be request the Kogan Switch to update its DPS 18, 19, and 20. After this update. The command 10 DP Query is able to return all 3 values, and no longer show invalid JSON Object. So it appears that the values 18, 19, and 20 were most likely null according to the device so when it attempted to form the DP_QUERY Response it failed, and responded with invalid JSON.

Command 18, appears to provide a COMMAND 8 Status Response, but only updated or changed values appear in this response. Command 18 needs to be continously invoked for Energy values to be updated.

My sample app is performing COMMAND 18 (18,19,20) then COMMAND 10 to obtain all 3 values.
COMMAND 18 only provides values to DPS that were updates. Hence not all the time, the 18,19, or 20, DPS were all found within the Command 8 Response. So it seems better to request COMMAND 18 and ignore the COMMAND 8 response, and then Request Command 10, and use the Command 10 to get the set of 18, 19, 20 DPS values.

Alot of people commented that this Kogan switch is not able to report Power below 5w. I think this is correct. at 2w it showed as 0. However at 8 watts it was able to show.

Now that we understand how to refresh energy monitoring DPS for this particular Kogan Switch, I will investigate whether I will integrate these findings into localtuya, or a change needs to be made in tuyapi.

My ultimate goal is to have the switches working Locally without internet on Home Assistant.
I will need to investigate how localtuya attempts to get energy dps values, and see how I can inject the COMMAND 18 in.

I did not look at how often the app requested COMMAND 18, but the app was updating every few seconds so my assumption is COMMAND 18 was being called every 2-5 seconds.

Under my testing, 5+ seconds intervals of calling command 18 and command 10 seemed ok for this device.

[beatmag]

I’ve been looking at the repo and trying to work out how to implement the changes.

I want to be able to set an option to invoke command 18 periodically. Probably in a similar loop to heart beat every 10-30 seconds.

Difference during the discovery phase this fall needs to be invoked too.

I just need to invoke this call first prior to attempting to resolve any dps values.

[beatmag]

I made some code changes to invoke command 18 for dps 18,19,20 before attempting to get status.

This works and energy values update correct and a DP_Query works without invalid json afterwards.

But I am facing another problem. Dps 1 ie whether the switch is on or off. Is not resolved after power on switch with no internet.

I've gone back to do my packet captures and smart tuya app also doesn't know the status of the switch. Tuya app assumes switch is on and is unable to obtain dps1 until either a manual button press or a remote turn on or turn off.

I've tried to use command 18 to query dps 1. But it does nothing.

So with internet off the switch is unable to report on or off until the switch gets flipped manually or via command.

I'm wondering whether eventually the switch is able to report its status periodically if it has no internet.

Edit: does anyone know how to fix dps 1 not returning? Do we just ignore status and assume off and wait for user to manual press button or turn on remotely?

[beatmag]

I read up on all the unvalid json errors across multiple projects. I'm going to follow the setting of dps 1 to false as a default if not dps 1 is returned.

If this prototype works I will try to put this into this project without affecting other device types. Possibly just checking if dps 1 exists if not return a false for dps 1. This would allow the code to continue and pretend that the dps 1 is off. Ie from a switch perspective the switch is off.

[idanre1]

May I ask which dns queries shall I block?
What to configure in the router?

[mafrosis]

I'm still working through exactly what I think is the best solution, but what is apparently working for me with two Tuya fans:

1. Blocked all internet access via firewall
2. Allowed DNS requests via firewall
3. Forward all port 53 traffic to a pihole
4. Subsequently blacklisted m2.tuyaeu.com in pihole (NOTE: I'm on the EU central datacenter on iot.tuya.com)

The fourth step is not actually required from my testing, but is a reasonable thing to do if the Tuya devices have no internet access anyway 🤷

Most importantly, I had previously blocked DNS entirely in a firewall rule, and that made the Tuya devices inoperable. My findings suggest the message in the README is misleading at best

[sibowler]

@mafrosis - I'm also having issues with my Tuya fans, so interested to find your experience.

Just wondering what address you sinkhole m2.tuyaeu.com to? 127.0.0.1?

How long after a power restart does it take for the devices to be available locally? Do you need to send any additional commands to get them to wake up, or they start reporting status as normal?

[jasonacox]

@beatmag COMMAND 18 seems to also used to cause Tuya devices (mostly energy monitoring ones) to refresh and report specified DPS. Here is a command list: https://github.com/jasonacox/tinytuya/blob/master/tinytuya/core.py#L113

UPDATEDPS       = 0x12 # 18 # FRM_LAN_QUERY_DP 

[Ro4cHii]

Do you have an idea what I am doing wrong?

I did the following:

• Block all ports in my Fritzbox except 53
• Forward all port 53 traffic to Adguard Home
• Add a3.tuyaeu.com and m2.tuyaeu.com to the "forbidden domains" list (requests are silently discarded)

Yet, all DPs are frozen.

[KTibow]

did you try blocking dns?

[Ro4cHii]

Yes, I also tried blocking access completely (every port) and port 53 only (hard and soft block).

[maxime1992]

Hey guys, looking in the thread above I couldn't find any answer to what I'm facing.

I setup localtuya a few days ago. Everything seemed to be working ok even though I blocked all the tuya subdomain from pihole.

I think I restarted my server or the integration and I noticed that it didn't start properly. I tried to manually restart the integration and same error.

I went to pihole, disabled temporarily the DNS block for a few minutes, re-run the restart of the integration and this time it worked.

I think the localtuya integration shouldn't fail if it can't connect to the internet and use the data it had from the previous time no? Otherwise how a DNS block would ever work? I may be missing something here

[andrus2049]

For true local use of Tuya devices with the LocalTuya integration you need to grab by yourself the device's Keys and IDs.
IMHO If you conversely use your cloud API account for the integration you aren't completely local, as at startup HA needs to get these data from the Tuya cloud. If you block Internet access, all your devices aren't discovered by the integration.

I use a version 3 of LocalTuya and have found by myself the keys and IDs data and entered them in the integration config, so no Internet access is ever required.

NOTE: You must have your Tuya device's Key and ID in order to use LocalTuya. The easiest way is to configure the Cloud API account in the integration. If you choose not to do it, there are several ways to obtain the local_keys depending on your environment and the devices you own. A good place to start getting info is https://github.com/codetheweb/tuyapi/blob/master/docs/SETUP.md or https://pypi.org/project/tinytuya/.

NOTE 2: If you plan to integrate these devices on a network that has internet and blocking their internet access, you must also block DNS requests (to the local DNS server, e.g. 192.168.1.1). If you only block outbound internet, then the device will sit in a zombie state; it will refuse / not respond to any connections with the localkey. Therefore, you must first connect the devices with an active internet connection, grab each device localkey, and implement the block.

https://github.com/rospogrigio/localtuya

[maxime1992]

IMHO If you conversely use your cloud API account for the integration you aren't completely local, as at startup HA needs to get these data from the Tuya cloud.

100% agree on the first part. I didn't read carefully enough and assume it was to retrieve the IDs only once and then it wouldn't use the cloud API again. Just saw this as a convenient feature to have the IDs retrieved the first time but I was wrong.

I'll give that a try thanks.

EDIT: It worked perfectly and I'm able to reload the integration with the DNS block in pihole, no worries anymore. I'll see if it keeps working nicely in the next few days 🤞. Thanks @andrus2049 !

[lennvilardi]

How did you managed to make it worked ?
If I block completely internet on the tuya device (on the UDM Pro) and block dns request in pihole the device is not updating anymore. If I only block DNS the device still connect to the cloud :-(