Slow response or hang editing devices with Decoder references.

[berkinet]

I discovered the cause of the slow response/hang problem when editing Shim devices, using the Decoder function. The device tag <states_list type="vector"> is continually extended to contain the Decoded values from the MQTT message. For example:
<states_list type="vector">
data_format
humidity
temperature
pressure
acceleration
acceleration_x
acceleration_y
acceleration_z
tx_power
battery
movement_counter
measurement_sequence_number
mac
data_format
humidity
temperature
pressure
acceleration
acceleration_x
acceleration_y
acceleration_z
tx_power
battery
movement_counter
measurement_sequence_number
mac
...
...
</states_list>

[FlyingDiver]

Where is <states_list type="vector"> coming from? The string "vector" is not in the plugin code.

[berkinet]

Here is the device from the Indigi database.
saved-shim-device.txt

[FlyingDiver]

Oh, that's how the Indigo database saves a list. I need to see your Decoder code. I expect you're doing something recursive that's expanding the state list.

[berkinet]

from ruuvi_decoders import Df5Decoder

class Ruuvi(object):
    def __init__(self, name):
        self.name = name
        self.decoder = Df5Decoder()

    def decode(self, payload):
    
        new_states = {}
        decoder = Df5Decoder()
        ruuvi_data = payload['data']
        ruuvi_data = ruuvi_data.split("FF9904")[1]
        new_states = decoder.decode_data(ruuvi_data)
        if len(new_states):
            return new_states
        else:
            return None

[berkinet]

BTW, the list grows coincident with each processed message.

[FlyingDiver]

You're overloading the "decoder" name. And creating a new Df5Decoder every time.

from ruuvi_decoders import Df5Decoder

class Ruuvi(object):
    def __init__(self, name):
        self.name = name
        self.df5decoder = Df5Decoder()

    def decode(self, payload):
    
        new_states = {}
        ruuvi_data = payload['data'].split("FF9904")[1]
        new_states = self.df5decoder.decode_data(ruuvi_data)
        if len(new_states):
            return new_states
        else:
            return None

[berkinet]

Ok. I see, I did not change the decoder reference to self.df5decoder...
Thanks.

[FlyingDiver]

I renamed it.

[berkinet]

Ah, ok. I missed that as the name was also used earlier in another context. Tested and it works. <states_list> remains empty.

Thanks again.

[berkinet]

Unfortunately, the problem is back. I had waited several minutes after replacing the Decoder code and did not see the problem. However, while editing a control page just now I once again noticed a very sluggish response when opening the shim device to select a display state. On checking the IndigoDB I see the same issue I reported earlier. For reference, here is the Decoder code:

from ruuvi_decoders import Df5Decoder.

class Ruuvi(object):
    def __init__(self, name):
        self.name = name
        self.df5decoder = Df5Decoder()

    def decode(self, payload):
    
        new_states = {}
        ruuvi_data = payload['data'].split("FF9904")[1]
        new_states = self.df5decoder.decode_data(ruuvi_data)
        if len(new_states):
            return new_states
        else:
            return None

[FlyingDiver]

Turn on debug logging and post the log section from when the issue happens.

[berkinet]

One possibility... When I loaded the newest plugin (from the plugin store) I did not at first realize you had moved the Decoders directory to the Python3-includes directory. So, I tried adding the decoder code to the Decoder directory embedded in the plugin. That was the code with the overloaded reference. I have now removed that file. However, is it possible the plugin for some reason, at some point, read that file instead?

[berkinet]

I have cleaned up the Device descriptions and restarted the plugin in debug mode.

[FlyingDiver]

The plugin no longer reads decoders from the embedded folder. Only from the Python3-includes folder. The embedded folder is only used as a source to populate the external folder with sample decoders.

[FlyingDiver]

But it's possible that since the old path was already saved to the device, it might have used it. You can't select that location anymore.

[berkinet]

That is what I figured. OTOH, I wanted to give you all the information I had, just in case it meant something to you. In the meantime, I am monitoring the IndigoDB to see if/when the problem occurs.

[berkinet]

Here is the debug file from around when the plugin added new entries to the states_list. What is weird is this does not happen on every message processed. One other possible factor. I have two separate shim devices for the ruuvi tags. They differ on the second part of the topic (the mac address of the tag). However, both devices use the same code.
shim_debug.txt

[FlyingDiver]

I don't see anything unusual in that log. I don't see any indication that the plugin is doing anything weird.

If the only problem you're seeing is the sluggish behavior when editing control pages, I'm inclined to attribute it to Indigo client issues with a device that's updating multiple times per second. If you disable the trigger when doing your edits (so no updates are going to the Shims), does it behave?

[berkinet]

I am pretty sure the sluggish behavior is due to the very large size of the Device XML. If I hand edit the DB (with the server not running) then it is snappy. But, over time, as the device size grows, it becomes increasingly sluggish. Also, I have 10 other shim devices and the sluggish behavior is only happening with the two shim devices that use the Decode option.

[FlyingDiver]

Why are you even looking at the XML, much less editing it?

[FlyingDiver]

Hmm. Do a right click on the shim device in the Indigo UI and print the device details to the log. Then post that to this issue.

[berkinet]

I was trying to find the source of the problem, and did. Of course, I saved copies of the DB. Attached is the device printout from the log.
saved_device.txt

[FlyingDiver]

I think you need to delete those two devices and create them again from scratch. I think they got messed up when you had the recursion going in the decoder.

[berkinet]

One other thing, the error occurs at around 15 minute intervals. I will delete and recreate the devices.

[FlyingDiver]

What do you mean? You get an error message? What happens every 15 minutes?

[berkinet]

I am monitoring the IndigoDB (grepping for a string that occurs in the additional states) once a minute. When I see the could increase I know the error occurred

[berkinet]

I have deleted and replaced the two devices. BTW, just in case this might have an impact, to limit the message volume, I am only enabling the MQTT connector device for 4 seconds every minute. I have it at 1 minute intervals now for testing. Also, Ruuvi will be adding the ability to limit the message frequency so I won't have to do anything in Indigo.

[berkinet]

New device, same problem...
saved_device.txt

[FlyingDiver]

Try https://github.com/FlyingDiver/Indigo-Shims/releases/tag/2023.0.4

You'll need to re-create the devices again.

[berkinet]

I've installed the new version, so far so good. Out of curiosity, what was the problem?

[FlyingDiver]

There's a delay between when I update the state list on the server and when it'll return the new values to me. The plugin host process caches the device def for a while. So the plugin didn't see the updated states right away so it wasn't accounting for them properly. I now do the merging of the existing states and the new states from the decoder differently.

[berkinet]

Thanks for the explanation, and thanks for the fix.  It's been an hour since I updated the plugin, and no further problems. I'll call this fixed.