RFC: MQTT re-design

[thess]

Requesting some feedback on what is going to be included in v2.0. The major changes are mostly internal and include using JSON formatted messages for the indicators and the inclusion of CallerID data. Attached are some implementation notes.

Callattendant MQTT client

Base-topic default is 'callattendant'. Indicator names are one of:
RING, Approved, Blocked, Message, MessageCount

Examples:

callattendant/Approved {"TimeStamp": 1714510621, "State": "BLINK", "Period": 700, "Count": 2}
callattendant/CallerID {"TimeStamp": 1714512345, "Number": 6175551234, "Name": Unknown, "Action": "Screened", "Reason": "")

JSON formatted messages:

• TimeStamp is either Unix epoch (seconds) or localtime as text.
• CallerID Number can be either direct from provider or formatted for display.

Config file additions:

#MQTT_TIME_FORMAT: The format of the timestamp in the MQTT message. Valid values are: ISO, UNIX
# ISO = "YYYY-MM-DD HH:MM:SS"
# UNIX = Seconds since epoch (Usually: 1970-01-01 00:00:00)
MQTT_TIME_FORMAT = "UNIX"
#MQTT_CALLERID_FORMAT: The format of the callerid in the MQTT message. Valid values are: RAW, DISPLAY
# DISPLAY uses the PHONE_DISPLAY_FORMAT to format the number
# RAW uses the number from the provider
MQTT_CALLERID_FORMAT = "RAW"

# MQTT_NOTIFICATION_TYPE: The type of notification to send. Valid values are: EVENT, STATE
# This setting controls the RETAIN attribute of MQTT messages.
#   EVENT: Indicator topics are not retained
#   STATE: Indicator topics are retained
MQTT_INDICATOR_TYPE = "STATE"

Callattendant MQTT message templates:

IndicatorName: {"TimeStamp": 0, "State": "", "Period": 0, "Count": 0}

• Period := time period in ms
• Count := Number of cycles (Period * Count)
• State := ON, OFF, CLOSED, BLINK, PULSE or count of messages

CallerID: {"TimeStamp": 0, "Number": "", "Name": "", "Action": "", "Reason": ""}

• Number may be raw/internal or formatted text.

[thess]

If anyone wants to try out the new/revised MQTT integration, you can install a test directly from the mqtt-dev branch. You can run it from a separate python venv or as an update to your existing installation (if brave enough). Additional configuration options are supported as described in MQTT-Client writeup above.

$ source venv/bin/activate
$ pip3 install "callattendant@git+https://github.com/thess/callattendant@mqtt-dev"

[whitedavidp]

I am so new to MQTT, I don't think I can be of much help (at least not yet). But the ability to get caller id info via MQTT is exactly what I hope for. Thanks

[whitedavidp]

Just letting you know that I am now able to get and display LastCallerID info via MQTT on both my Windows desktops and my Android devices! On Windows I am using EventGhost and this MQTT Plugin with this simple script:

<?xml version="1.0" encoding="UTF-8" ?>
<EventGhost Version="1722" Guid="{635FF80E-832E-496E-BCAA-B84EB064D29F}" Time="1715036300.75">
    <Autostart Name="Autostart" Expanded="True">
        <Plugin Identifier="MQTTthreads" Guid="{D5CCABA6-8E20-4B59-A7A7-9C197F91037F}" File="MQTT Client">
            gAIpLg==
        </Plugin>
        <Action>
            EventGhost.NewJumpIf(XmlIdLink(7), 0, False)
        </Action>
    </Autostart>
    <Macro Name="Show Message" id="5" Expanded="True">
        <Action>
            EventGhost.ShowOSD(u'Call from {eg.event.payload}', u'0;-24;0;0;0;700;0;0;0;1;0;0;2;32;Arial', (255, 255, 255), (0, 0, 0), 3, (0, 100), 0, 5.0, True)
        </Action>
    </Macro>
    <Macro Name="MQTT Client: Start a new MQTT subscription " id="7" Expanded="True">
        <Action>
            MQTTthreads.MQTTclient(u'DesktopPc', u'192.168.1.59', 1883, u'callattendant/LastCallerID', False, '967095524795', False, False, u'username', u'password', False, u'', u'', u'', u'TLSv1.2')
        </Action>
    </Macro>
    <Macro Name="Receive MQTT From callattendant" Expanded="True">
        <Event Name="MQTT.callattendant/LastCallerID" />
        <Action Enabled="False">
            EventGhost.ShowMessageBox(u'Call Attendant', u'Incoming call from {eg.event.payload}', u'', u'', False, 64, 0, 0)
        </Action>
        <Action>
            EventGhost.PythonScript(u'import eg\nimport json\n\njson_string = json.loads(eg.event.payload)\n\nif json_string["Action"] == \'Permitted\':\n    eg.event.payload = \'Call from \' + json_string["Name"] + \' on \' + json_string["Number"]\n    eg.result = True\nelse:\n    eg.result = False')
        </Action>
        <Action>
            EventGhost.NewJumpIf(XmlIdLink(5), 0, False)
        </Action>
    </Macro>
</EventGhost>

And on Android I am using a combination of Tasker and this MQTT Client app that has hooks for use with Tasker.

Really great and fast enough for my needs. Note: this is confined to my LAN only. And I am filtering out all but Permitted calls via these mechanisms. With CA in place, I am very happy indeed to ignore everything else. Thanks!

[comet424]

since i been running Callattendant Orginal with MQTT for 2 years now that i modded myself but i shelved things as i wanted to expand Formatting...so i shelved it till i found this new fork and MQTT of this new programmer and his interest in the MQTT too as i do Current Call coming in to Displays for my Mother. Through Home Assistant

but id like to see

Name:   Read from Direct Line  or from Database  But Be the Current Call  not Last Call
Number: Formatted so its not like 12351231231
Time:  Formatted Time    (If your not using like Home Assistant Formatted Time)
Status:  Approved/Blocked/Screened:     This way you can do in a display BLOCKED with different color or anything

these be some ideas i also worked on webpage tab for for the Callattendant would view better on a cell phone as it didnt view right but i also shelved it along time ago when it wasnt working right.. and ment to work on it again

i have 6 of these displays through out my moms house... and 3D printed the cases and made the display using Nexition Screen ESP32 using ESphome and Home Assistant

[whitedavidp]

In my experience with version 2.x so far, I get the LastCallerID MQTT message while the phone is ringing. So in a way, this may be a semantic issue only. Perhaps might be better to call the topic CurrentCallerID (although then I would have to change all my code). But it would be more clear what this is, I think.

So if you want to display the info from the currently incoming/ringing call, that is what you will get if you subscribe to LastCallerID.

[whitedavidp]

With regards to the format of the message, I think that the current, JSON format is good and generic so that all sorts of message receivers can process the data and put it into whatever format they might wish - prior to displaying the data. I know nothing of H/A personally, but in general, it seems to permit users to perform very complex work. This might including filtering based on Permitted,Screened,Blocked or re-formatting dates/times, basically anything you might want.

[comet424]

if you wanna do Last call and a current caller then send 2 MQTTS

Current_Caller_Name:
Current_Caller_Number
Current_Caller_Time:
Current_Caller_Status:

Last_Caller_Name:
Last_Caller_Number
Last_Caller_Time:
Last_Caller_Status

this way with a Home Automation you can keep a log or Display of the Current Call coming in and the Last Caller

so be like

Current_Caller_Name:   John Smith
Current_Caller_Number 1-234-334-5555
Current_Caller_Time: 08:22pm
Current_Caller_Status: Approved

Last_Caller_Name: Joe Blue
Last_Caller_Number 344-555-5555
Last_Caller_Time: 11:22am
Last_Caller_Status  Blocked

[whitedavidp]

So far, I am very happy with what has been offered so far. Thanks!

One thing I do note with regards to Permitted calls: even though I have a number + name on my permitted caller list, what comes through in the MQTT is the name provided by the modem/telecom, not the name assigned to the number in the caller list. So for example, my telecom sees my cell phone as "Wireless Caller" (not sure why but I have no way of changing this) while I have CA registering the number with my proper name. In the CA dashboard, my name is shown but in the MQTT, "Wireless Caller" is what comes through. Clearly, using the CA information is more helpful/informative. But I would not wish to see this add much overhead and reduce the timeliness of the MQTT being published.

Thanks!

[comet424]

ya your wanting to send the Database name... ya i tried working on that long time myself but since i not a good coder at python i just did what i did... but i did mention it to the programmer... so hopefully that will be part of a release... when i worked on it i kept crashing the software lol... been running the CA 4 years now been helpful for my mom and 2 years now running mqtt to displays as she cant read the phones as much.. next improvement i probably do is Speak the Callers Name from the displays or a bluetooth speaker

[whitedavidp]

I will also say that I personally change the line:
client.publish(self.topic_prefix + topic, message, retain=True)
to
client.publish(self.topic_prefix + topic, message, retain=False)
in the file hardware/mqttindicators.py. I really want the LastCallerID info to reflect only what is currently happening in real time. I cannot say about other use-cases people may have for this, but I certainly do NOT want a flood of potentially old and out of date MQTT coming in. This seems to avoid that.

[thess]

Well, there is more than one use case supported here albeit with some client intelligence. I believe it's up to clients to filter and/or remember last items seen from the source (persistence). Also note there is a timestamp in the message for a reason. Anyway, isn't FOSS wonderful. Hmmm... maybe another option?

[whitedavidp]

Well I certainly do not claim expertise. I would simply suggest a config option to control this. No need to put the owness of the client beyond that. But again, changing the single line of code is no big deal (at least for me - an ancient software engineer)... Thanks

[thess]

Added the option -- revised RFC and doc

[whitedavidp]

Cool! Please let me know when this round of updating is complete so I can download and test it out. Cheers

[whitedavidp]

Ok, I am running 2.0.4 now. Changed the MQTT topic on my clients to CallerID from LastCallerID. Added the new config item and set to EVENT. Was already using ISO and DISPLAY formats for MQTT. Seems to be running fine.

When diffing against what I had for 2.0.2:

1. I notice changes to the service.example that appear to be pretty pi-oriented and seemingly were not there/needed before...
2. Somewhere I noticed what seemed to be an inconsistency in user-id/password between "" and None. It says the default is None. But somewhere I saw "" being used. Possibly not important but seems it should be consistent.

Thanks for all of these improvements. They make CA even better than before. Much more useful. Have you a PayPal I can donate to?

[thess]

As for the mqtt user/psw either empty or None will do - interface is not picky. Was just doing some cleanup for consistency -- and, maybe a bit more. No need for you to change anything here.

The service example was expanded a bit to improve logging and work better in a Python Virtual Env.. Replace the user-name and path names with whatever you use. I prefer a direct launch in a service script rather than relying on environmental defaults.

As far as donations go, I prefer none. Less obligations that way. Give some money to a food-bank somewhere instead.

[comet424]

@thess hi ted been a while since checked in or worked on the callattendant.. if there is an update on your mqtt version how do i update the mqtt.. or i also re installed raspbian a new micro sd card to do a clean install of callattendant if thats how to do the updated mqtt version

i also was working on callattendant Poller as the odd time.. Callattendant will crash.. but i cant get it work from how i read up on it... maybe you know what you need to do... so what i doing is say every 10 min its MQTTs a Yes to say Callattendant is still up and running so so the home automation will know if its up or crashed... as Callattendant will crash the odd time...

so i used the import sched but from what i followed isnt working this is what i did
maybe you can add it to your code since your more fluent and see where i went wrong and that can be added to future Callattendent update

class CallAttendant(object):
    """The CallAttendant provides call logging and call screening services."""

    """ Pole Every 10 Min That Callattendant Hasnt Crashed.. MQTT Topic "ALIVE" Message "Yes"    """
    def update_alive():
        os.system('mosquitto_pub -t "callerid/alive" -m ' + "Yes" + ' -h 192.168.0.12 -d --will-qos 1 -u "mqtt-user" -P "*******"')        
        scheduler = sched.scheduler(time.time, time.sleep)
    def repeat_task():
        scheduler.enter(600, 1, update_alive, ())
        scheduler.enter(600, 1, repeat_task, ())

        repeat_task()
        scheduler.run()

    def __init__(self, config):

i used this web link for trying to learn

https://pravash-techie.medium.com/python-sched-for-automating-tasks-in-python-396618864658#:~:text=A%20scheduler%20is%20a%20tool,use%20interface%20for%20scheduling%20tasks.

so what id maybe like is in the Config file
Keep_Alive = 10min
and then it would MQTT a Yes every 10 min say least thats what i working on to incorporate into Home Assistant

[thess]

I am locking this thread because the topic is closed, implemented and released. Start a new topic if you want to discuss further features and problems.

@comet424 - If you need help with custom code, this is not the topic to post it under. In general, this repository is also not the place to ask for general programming help. That is best handled in other forums where someone else may be able to guide you.