tdm crashes on MQTT connect since I have added a pattern

[Waldmensch1]

Hi,

System: Win64, Python 3.7

I was able to run tdm and connect to my Broker. One device were visible. I thought this is because I have to configure a pattern. My toppics are all in the same Style stadtweg/og/light1. So I have entered a pattern %prefix%/stadtweg/ Because I don't know what prefix should be used. maybe tele ? Since I did that TDM crashes after Start while connecting Broker. Window just closes, no Errormessage. Where can I find the Settings to make the pattern undone? What would be the right pattern in my case?

[jziolkowski]

The prefix, as it's written in the dialog, should be %prefix%

The TDM config is in \TDM\tdm.cfg, patterns are there.

[Waldmensch1]

If I start from console (did it over link before) I get a speaking error:

C:\Users\tommy\Downloads\tdm-master-1\tdm-master>C:\Users\tommy\AppData\Local\Programs\Python\Python37\python.exe tdm.py
qt.qpa.fonts: Unable to open default EUDC font: "C:\\Windows\\FONTS\\EUDC.TTE"
Traceback (most recent call last):
  File "tdm.py", line 341, in mqtt_message
    possible_topic_cmnd = p.replace("%prefix%", "cmnd").replace("%topic%", possible_topic) + "FullTopic"
TypeError: replace() argument 2 must be str, not None

C:\Users\tommy\Downloads\tdm-master-1\tdm-master>

[Waldmensch1]

There is no tdm.cfg

C:\Users\tommy\Downloads\tdm-master-1>tree /f
Auflistung der Ordnerpfade für Volume OS
Volumeseriennummer : BC91-7E12
C:.
└───tdm-master
    │   .gitignore
    │   CHANGELOG.md
    │   LICENSE
    │   README.md
    │   tdm.py
    │
    ├───GUI
    │   │   Broker.py
    │   │   BSSID.py
    │   │   Buttons.py
    │   │   ClearLWT.py
    │   │   Console.py
    │   │   Devices.py
    │   │   GPIO.py
    │   │   Modules.py
    │   │   Patterns.py
    │   │   Power.py
    │   │   Prefs.py
    │   │   Rules.py
    │   │   SetOptions.py
    │   │   Switches.py
    │   │   Telemetry.py
    │   │   Templates.py
    │   │   Timers.py
    │   │   __init__.py
    │   │
    │   ├───icons
    │   │       auto_telemetry.png
    │   │       buttons.png
    │   │       clear.png
    │   │       color.png
    │   │       connect.png
    │   │       console.png
    │   │       copy.png
    │   │       disconnect.png
    │   │       download.png
    │   │       logo.png
    │   │       once.png
    │   │       open.png
    │   │       P1_OFF.png
    │   │       P1_ON.png
    │   │       P2_OFF.png
    │   │       P2_ON.png
    │   │       P3_OFF.png
    │   │       P3_ON.png
    │   │       P4_OFF.png
    │   │       P4_ON.png
    │   │       P5_OFF.png
    │   │       P5_ON.png
    │   │       P6_OFF.png
    │   │       P6_ON.png
    │   │       P7_OFF.png
    │   │       P7_ON.png
    │   │       P8_OFF.png
    │   │       P8_ON.png
    │   │       power.png
    │   │       P_OFF.png
    │   │       P_ON.png
    │   │       refresh.png
    │   │       restart.png
    │   │       rules.png
    │   │       save.png
    │   │       setoptions.png
    │   │       settings.png
    │   │       sliders.png
    │   │       status_high.png
    │   │       status_low.png
    │   │       status_medium.png
    │   │       status_offline.png
    │   │       stop.png
    │   │       switches.png
    │   │       telemetry.png
    │   │       time.png
    │   │       timers.png
    │   │       upload.png
    │   │       web.png
    │   │
    │   └───__pycache__
    │           Broker.cpython-37.pyc
    │           BSSID.cpython-37.pyc
    │           Buttons.cpython-37.pyc
    │           ClearLWT.cpython-37.pyc
    │           Console.cpython-37.pyc
    │           Devices.cpython-37.pyc
    │           GPIO.cpython-37.pyc
    │           Modules.cpython-37.pyc
    │           Patterns.cpython-37.pyc
    │           Power.cpython-37.pyc
    │           Prefs.cpython-37.pyc
    │           Rules.cpython-37.pyc
    │           SetOptions.cpython-37.pyc
    │           Switches.cpython-37.pyc
    │           Telemetry.cpython-37.pyc
    │           Templates.cpython-37.pyc
    │           Timers.cpython-37.pyc
    │           __init__.cpython-37.pyc
    │
    └───Util
        │   commands.json
        │   models.py
        │   mqtt.py
        │   setoptions.json
        │   __init__.py
        │
        └───__pycache__
                models.cpython-37.pyc
                mqtt.cpython-37.pyc
                __init__.cpython-37.pyc


C:\Users\tommy\Downloads\tdm-master-1>

[jziolkowski]

I told you home directory, not downloads directory.

See c:\users\tommy\TDM

[Waldmensch1]

Well, found in the meantime and got it running again by deleting the pattern. But now I get only one device which not follows my normal topic convention. Only found is topic avatar01 all my 40 devices which have topic in form stadtweg/og/light1 or stadtweg/pir/hh/pir1 are not found. What I'm doing wrong?

[Waldmensch1]

This is how it looks like in WebGUI of devices:

[jziolkowski]

Please read the Autodiscovery page in this wiki project. You need to move some parts from topic to fulltopic to make it work

[Waldmensch1]

Okay, that's sad, this way it is not usable for me. I would have to change my whole homeserver setup and all devices, including their direct dependencies, to get it running.

Edit: Switched to DEBUG log and I don't understand why there is no solution. All devices are known to TDM

...
2019-12-04 07:08:33 [DEBUG] DISCOVERY: LWT from an unknown device tele/stadtweg/og/shutter/wz1/LWT
2019-12-04 07:08:33 [DEBUG] DISCOVERY: Asking an unknown device for FullTopic at cmnd/stadtweg/FullTopic
2019-12-04 07:08:33 [DEBUG] DISCOVERY: LWT from an unknown device tele/stadtweg/og/shutter/wz2/LWT
2019-12-04 07:08:33 [DEBUG] DISCOVERY: Asking an unknown device for FullTopic at cmnd/stadtweg/FullTopic
2019-12-04 07:08:33 [DEBUG] DISCOVERY: LWT from an unknown device tele/stadtweg/og/shutter/sz/LWT
2019-12-04 07:08:33 [DEBUG] DISCOVERY: Asking an unknown device for FullTopic at cmnd/stadtweg/FullTopic
2019-12-04 07:08:33 [DEBUG] DISCOVERY: LWT from an unknown device tele/stadtweg/og/shutter/ku/LWT
2019-12-04 07:08:33 [DEBUG] DISCOVERY: Asking an unknown device for FullTopic at cmnd/stadtweg/FullTopic
...

[jziolkowski]

I'm open for suggestions how to parse such compound topic without knowing before-hand which part is the prefix to send commands to :)

[jziolkowski]

Hm, actually I might have an idea. Indeed there's something that helps solve an issue when trying to describe a problem to someone else. Stay tuned.

[jziolkowski]

On a side note, moving that path parts to fulltopic doesn't impact how your setup or automation work, because in the end, the whole device MQTT path looks exactly the same. It's just a change in tasmota side.

After changing your topic to wz2 and your fulltopic to %prefix%/stadtweg/og/shutter/%topic%, it will look exactly as you have now.

Then you just add %prefix%/stadtweg/og/shutter/%topic% to TDM and it will work.

[jziolkowski]

Okay, I did some tests and TDM discovery won't work with granular topics.
Like I mentioned, moving the path parts from topic to fulltopic doesn't change the final path, ergo all your home setup configs will work as they did before. You only need to modify your Tasmota MQTT configs.

[Waldmensch1]

The full topic on LWT is IMHO always tele/x/y/z/LWT. So split by "/"

I'm not so confident with python but something like that should give you prefix and topic. Maybe there is also a kind of regex to do that.

pseudo-code

arr = fulltopic.split("/");
prefix = arr[0];
LWT = arr[length(arr)-1];

for(i=1;i<length(arr)-1;i++)
{
   topic += arr[i] + "/";
}

javascript

function getPrefixTopic(fulltopic){
    var arr = fulltopic.split("/");
    var prefix = arr.shift();
    var LWT = arr.pop();
    var topic;
    while( (i = arr.shift()) !== undefined ) {
        topic += i + "/";
    }
    return {prefix, topic};
}

[jziolkowski]

No, LWT follows the FullTopic. If you changed FullTopic to some/test/%topic%/hahah/cow/%prefix%/ with topic chicken then LWT will arrive from some/test/chicken/hahah/cow/tele/LWT.

Trust me, I spent a LOT of time fiddling with topics and fulltopics to make autodiscovery work.

[jziolkowski]

I actually changed my test device to this example, and upon connecting to "#" topic, I get this:

[jziolkowski]

[Waldmensch1]

okay, so we have two cases

1. tele/x/y/z/LWT (default)
2. x/y/z/tele/LWT (SetOption19)

I have no clue whether someone uses the Fulltopic manipulation as you describe and for what reason.

I'm at home in JS so I'm not familar with possibilities in python. I would do it this way in JS: https://jsfiddle.net/o14b60my/1/

If someone defines an own pattern to cover his special fulltopic construction you could fork into your actual code with regex.

function getPrefixTopic(fulltopic, option19){
    var arr = fulltopic.split("/");
    var LWT = arr.pop();
    var prefix = (option19) ? arr.pop() : arr.shift();
    var topic = "";
    while( (i = arr.shift()) !== undefined ) {
        topic += i + "/";
    }
    return {prefix, topic};
}

[jziolkowski]

You're trying to convince me to a solution that covers your way of thinking, but mine covers all possible FullTopic patterns, with a simple inclusion of custom pattern in TDM config. And now you insist that custom patterns should be forked.

And TBH I don't spend time thinking why people use such complex patterns. I just give them an option to have it work almost out of the box, as long as they use some consistency.

All the code snippets are already covered in TDM defaults. And I know how to split a string by "/" to extract first, element, middle and the last part.

Like it or not, but Tasmota recommends using a single-word topic, and additional path elements in fulltopic. And I repeat again and again: from your home automation side, it doesn't matter if the path is in topic or in fulltopic, because the full path will look exactly the same.

But if you like to drive a screw using a butter knife because you think that's better: I'm not going to stop you/

[jziolkowski]

Oh, another example: I've seen many users prepend all their Tasmota FullTopics with tasmota/%prefix%/%topic% because they want Tasmota messages under a single root topic. Now write me a snippet of code which - without knowing that before hand - parses the topic using a simple split.

BTW: do you know that some users change their prefixes also? So then you must know not only the fulltopic beforehand, but also possible prefixes.

more real examples:

home/basement/%prefix%/%topic%
garage/gate/%prefix%/%topic%
garage/lights/%prefix%/%topic%

etc etc ad infinitum.

[Waldmensch1]

then stay with the way it is. I already closed the ticket a while ago. Means I'm done with. I've some other things on todo right now, maybe I make it running for me later. Shouldn't be that complicated. Faster than endless discussions anyway.