New Release: 0.7.5-beta

[freakent]

Dear All,

I have made a new release of the dbus-mqtt-driver that completely removes the YAML dependency from this project. This should address the problems that have plagued this project for months caused by the need to install the PyYaml library on Venus. It should also pave the way to make the driver installable by setuphelper in a subsequent release.

I have switched to using ini files which are natively supported by Python. This feels like quite a substantial change to the internals of the driver, so I am releasing this first version without YAML as a BETA. I have performed a lot of testing myself but I would like to collect feedback from users before making a final release.

To upgrade just follow the install instructions in the v0.7.5-beta readme as before.
https://github.com/freakent/dbus-mqtt-devices/tree/v0.7.5-beta

If you try this release, please comment below whether it works successfully for you or not. All feedback is appreciated.

-- updated release number to v0.7.5-beta

[freakent]

This release addresses issues #55, #56 and #57.

FYI @stundenblume, @Hofyyy, @GallusMax, @pos-ei-don, @stromibaer

[SoulRaven]

Version: 0.7.4-beta

something is wrong i guess, i have this payload on "device/serverRoom/Status ":

{
"clientId": "serverRoom",
"connected": 1,
"version": "v0.7.4-Beta",
"services": {
"t1": "temperature"
}
}

but i get:

@4000000064eb42d006c17054 INFO:device_manager:Received device status message {'clientId': 'serverRoom', 'connected': 1, 'version': 'v0.7.4-Beta', 'services': {'t1': 'temperature'}}
@4000000064eb42d006f8ef5c WARNING:device_manager:status.service type temperature is not supported, please check services.yml

[freakent]

@SoulRaven Thanks for reporting that. New beta version 0.7.5-beta available.

[SoulRaven]

thx, now is working, but some things are not setup,
version, and device name using instance ID.

also, is it possible to setup the "Name" directly from device topic?

L.E.

@4000000064eb73613055a264 INFO:device_service:Registering service temperature for client ha_sonoff_th at path com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room
@4000000064eb736131562274 this is what we found <configparser.ConfigParser object at 0xb5f0a220>
@4000000064eb73613156459c INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted
@4000000064eb736136a3ccf4 INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/TemperatureType does not exist yet or must be adjusted
@4000000064eb73620056ebf4 INFO:settingsdevice:Setting /Settings/Devices/mqtt_ha_sonoff_th_server_room/ClassAndVrmInstance does not exist yet or must be adjusted
@4000000064eb7362063ede8c INFO:vedbus:registered ourselves on D-Bus as com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room
@4000000064eb736207892554 INFO:device_service:Registered Service com.victronenergy.temperature.mqtt_ha_sonoff_th_server_room under DeviceInstance 3
@4000000064eb736207ca0e84 INFO:device_manager:publish {'portalId': 'c0619ab2f11e', 'deviceInstance': {'server_room': '3'}} to device/ha_sonoff_th/DBus, status is 0
@4000000064eb7362082430bc INFO:device_service:setting changed, setting: CustomName, old: My Temperature Sensor, new: My Temperature Sensor
@4000000064eb7362086eb6dc INFO:device_service:setting changed, setting: TemperatureType, old: 2, new: 2
@4000000064eb737823e10d0c INFO:device_manager:Received device status message {'clientId': 'ha_sonoff_th', 'connected': 0}
@4000000064eb737828f6e67c INFO:device_service:Unregistered mqtt_ha_sonoff_th_server_room from dbus
@4000000064eb73782903b3d4 INFO:device:Removed Service server_room from client ha_sonoff_th
@4000000064eb737829102754 INFO:device_manager:**** Unregistering device ha_sonoff_th ****

[freakent]

Sorry, can you clarify

thx, now is working, but some things are not setup,version, and device name using instance ID.

[SoulRaven]

Sorry, can you clarify

thx, now is working, but some things are not setup, version, and device name using instance ID.

the Device name is not set also the Firmware version. What i have see in code Firmware Versions contains the value from payload "version" and the device name is the "client id"
In log seams that

INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted
somehow is not setup the costumName

Also will be great to have the possibility to setup directly from topic the "Name" of the sensor

[freakent]

This has me puzzled, it's not an area in the code that was changed. I checked in MQTT explorer what is being returned by Venus and it does include these values:

although they do not show in the UI

I am testing on pre-release Venus 3.1-32, so could be a venus bug. Which version of Venus OS are you testing on?

[freakent]

INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ha_sonoff_th_server_room/CustomName does not exist yet or must be adjusted somehow is not setup the costumName

This is just a debug message output by the Venus dbus library, it is always output when initialised.

Also will be great to have the possibility to setup directly from topic the "Name" of the sensor

I don't want to complicate the registration process. Once registered and you have a deviceid you can send a message to W/<portal id>/temperatire/<deviceid>/CustomName to change the name programmatically. You can do this for any of the parameters you wish to set not just CustomName.

[SoulRaven]

i have attached the printscreen with VenusOS version

[Becker884]

Hi, i tried today with 3.01 on RPi2, setup takes only few seconds, my Gridmeter & PV runs:

only the " " weren't there before

do you have that too?

[freakent]

@Becker884 Thanks for trying it out and spotting this error. I'll update the grid.ini config file to remove the quotes in the next release.

# Total bought energy over all phases
description: "kWh"
format: "{} kWh"

[Ac/Energy/Reverse]
# Total sold energy over all phases
description: "kWh"
format: "{} kWh"
  
[Ac/Power]
# Total power of all phases, preferably real power
description: "W"
format: "{} W"

[Ac/Current]
description: "A AC"
format: "{} A"

[Becker884]

Can I correct this on my own?
if yes -> where to find the grid.ini?

[freakent]

Yes, you can. It's services/grid.ini

See https://github.com/freakent/dbus-mqtt-devices/blob/Config-Files/services/grid.ini

[Becker884]

ok, but where can I find this file on my RPi2 ?

[freakent]

/data/drivers/dbus-mqtt-devices-0.7.5-beta/services/grid.ini

[mdkeil]

I used the beta with corrected grid.ini @venusos 3.10 ~32 without any problems for the moment.

[mdkeil]

although they do not show in the UI

same to me with an grid sensor. no firmware, Connection.. doesnt matter for me.

[freakent]

My concern is that it might affect how these devices are shown in VRM. I need to investigate this issue further.

[gharshaw]

@freakent
I installed on 3.10~32 CCGX today. I'm seeing some anomalies between the old MQTT and the new Flash-MQ. Victron has left old MQTT on port 1883 and the new FlashMQ on port 1884.
Tanks and Temp sensors show up on old MQTT correctly with all the pertinent topics/ values using MQTT Explorer

but do not show up on FlashMQ port 1884 using MQTT Explorer.

Also looking at dbus-spy the topic/values are missing in all temp and tank sensors

Tanks and sensors are missing from the HTML app and new gui-v2 app as well. Most likely due to the missing data in Flash-MQ

Tanks and Sensors are appearing on the console

however device data is missing

Are you seeing the same issues?
I can help with logs and any quick code changes you might want to test

Best,
Gene

[freakent]

@gharshaw, thanks for your investigations, that's all very interesting. I think I need to figure out if this issue is introduced by my code changes or the new FlashMQ in Venus OS. I think I might have to downgrade my test Raspberry PI and run tests there to be sure.

[gharshaw]

@freakent
Further investigations. I was monitoring the Flash-MQ log on device registration and see that the devices appeared and then immediately disappeared in Flash-MQ.

@4000000064ed181437b9fbcc [2023-08-28 21:56:26] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.temperature.mqtt_nr001_t1' with owner ':1.605'
@4000000064ed18152bdb8bf4 [2023-08-28 21:56:27] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.temperature.mqtt_nr001_t2' with owner ':1.606'
@4000000064ed1816129d8e6c [2023-08-28 21:56:28] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.temperature.mqtt_nr001_t3' with owner ':1.607'
@4000000064ed18163470fed4 [2023-08-28 21:56:28] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.temperature.mqtt_nr001_t4' with owner ':1.608'
@4000000064ed18171a49b6a4 [2023-08-28 21:56:29] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.temperature.mqtt_nr001_t5' with owner ':1.609'
@4000000064ed18182328bd74 [2023-08-28 21:56:30] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.tank.mqtt_nr001_tk1' with owner ':1.610'
@4000000064ed181913e22044 [2023-08-28 21:56:31] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.tank.mqtt_nr001_tk2' with owner ':1.611'
@4000000064ed181a0ae4d374 [2023-08-28 21:56:32] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.tank.mqtt_nr001_tk3' with owner ':1.612'
@4000000064ed181a35af0cb4 [2023-08-28 21:56:32] [INFO] [OwnerChange] Service appeared: 'com.victronenergy.tank.mqtt_nr001_tk4' with owner ':1.613'
@4000000064ed181b008527e4 [2023-08-28 21:56:32] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.temperature.mqtt_nr001_t1' with owner ':1.605'
@4000000064ed181b04ca110c [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.temperature.mqtt_nr001_t2' with owner ':1.606'
@4000000064ed181b07eeb93c [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.temperature.mqtt_nr001_t3' with owner ':1.607'
@4000000064ed181b0b717374 [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.temperature.mqtt_nr001_t4' with owner ':1.608'
@4000000064ed181b0f2678ac [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.temperature.mqtt_nr001_t5' with owner ':1.609'
@4000000064ed181b1512a934 [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.tank.mqtt_nr001_tk1' with owner ':1.610'
@4000000064ed181b189f2b54 [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.tank.mqtt_nr001_tk2' with owner ':1.611'
@4000000064ed181b1c3e4efc [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.tank.mqtt_nr001_tk3' with owner ':1.612'
@4000000064ed181b201032d4 [2023-08-28 21:56:33] [INFO] [OwnerChange] Service disappeared: 'com.victronenergy.tank.mqtt_nr001_tk4' with owner ':1.613'
@4000000064ed181b2a3ef6dc [2023-08-28 21:56:33] [ERROR] Error on 'GetValue' from com.victronenergy.temperature.mqtt_nr001_t5: org.freedesktop.DBus.Error.Se

[Becker884]

Do both brokers run parallel ?
Isn't it enough just to use Mosquitto?

[mdkeil]

Yes, but in future versions (VenusOS 3.20+) mosquitto will completely replaced by flashmq.

[Thxeverybody]

Great driver, helps me a lot. Upgraded from 0.6.2 to 0.7.5 beta without any problems. Use it to emulate a pv inverter behind a shelly and a grid meter with Hichi/Tasmota.

[freakent]

@Thxeverybody Thanks for the feedback, I'm glad to hear it is working for you. I don't hunk there will be another Beta release after I have done more testing.

[gharshaw]

@freakent I just updated my CCGX to Venus OS v3.11 and dbus-mqtt-driver 0.7.5-beta without any problems. The setup on the CCGX went smoothly and everything appears to be working correctly. Next hurdle, I imagine, will be with dbus-flashmq and the removal of dbus_mqtt.
Best Regards

[freakent]

If you are using the driver in a production system I highly recommend you use the latest release not the Beta.

I don't anticipate the move to FlashMQ is not such an issue. It will require a slight change to the installer that's all. This driver interacts with the dbus directly, not via dbus-Mqtt.

[gharshaw]

@freakent Does the latest release remove the YAML dependency? I thought the 0.7.x versions is where you are removing that dependency

[freakent]

That's correct, 0.6.x uses pyyaml.

[DJZorro]

Thank you for your driver,
With this I was able write to the dbus.
I picked the weatherinfo from Openweather.API.
Imported it in OpenRed.
Pulled out the information I wanted to display
Modified some date, Wind direction, wind speed, Sunrise and Sunset.
Pubished those data with MQTT and your driver to dbus.
Picked them up from dbus by with modify gps.ini
Pubished them on the screen with modifying OverviewHubEnhanced.qml

Long way but succesfull in the end:

[gurkc006]

Hi!
Just a comment from me: I used 0.7.5-beta for the last weeks without any problems on VenusOS 3.20_10. I use it to emulate one 3-phase PV Inverter from an 3-phase-AC-meter using a ESP32 and also for two temperature sensors. Worked very stable the last weeks.
Today I tried upgrading to VenusOS 3.20_33. After rerunning the install.sh script everything seems to run again. BUT: the despite the fact, that the different devices seem to be created correctly, there is no data available. this might be due to the fact that 3.20_33 switched now completley to FlashMQ. So the basic device creation seems to work, but there is no data available. I checked also with dbus-spy: only the name available, all other values are not visible.


Here are the topics for device creation in MQTT Explorer:

Here is a screenshot of the remote console:

Devices are visiblen, but empty...
So maybe I was a little bit naive thinking it might acutally work out of the box?!
Can I provide further information? Or is there a plan to update the driver because the FlashMQ-driver will be the future for VenusOS and it would be great, if the dbus-mqtt-devices would evolve to a dbus-flashMQ-devices :-)
Thanks a lot for the great work! I really appreciate it!
Christian

[freakent]

Thanks for providing the results of your research. Have you tried running the current release version of this driver (not the beta) on VenusOS 3.20_33?

Whilst VenusOS is being switched from Mosquitto to FlashMQ (both are MQTT brokers), I would avoid testing the Beta driver in a beta version of VenusOS, there are too many moving parts. The beta driver is very experimental at this time and not ready for production use.

[gurkc006]

Hi freakent,
yes I know, installing the latest beta of driver and OS was maybe a little be too much :-)
I tried installing 0.6.3 release, but had no luck with installing the YAML-dependency. I followed the steps from the installation instructions (ModuleNotFoundError: No module named 'dataclasses' -> opkg install python3-modules) but this doesn't seem to help. Always getting this dataclass error. Here you can find the console output AFTER installing the python3-modules and rerunning setup.sh: console.txt
So with this verion I don't get the driver to run properly. I had this behaviour earlier also with older VenusOS versions (that's the reason I switched to 0.7.x at all).
So for the moment I suppose switching back to VenusOS 3.20_10 and dbus-mqtt-devices 0.7.5_beta (that was working properly) and hoping, that the error will be fixed soon.
Thanks

[gurkc006]

Just for debugging purpose: VenusOS 3.20_33, dbus-mqtt-devices config-files branch head. Just restarted Venus and waited for the dbus-mqtt-devices logfiles (/var/log/dbus-mqtt-devices/current) to populate: creation.txt
For me, it looks not so bad for the first request (meter1). But no reaction on the second request (ds3g).
I tried to find some other entries in different logfiles, but nothing interesting...
Maybe that helps finding the problem? Is there anything I can test? I'm a little bit careful "playing" around with creating new devices using mqtt, as I still have some leftovers from such tests from a year ago, which are still visible in my VRM portal sidebar... But if necessary I would test of course. Thanks

[freakent]

I would prefer to figure out why python3-modules is not installing for you. What is the output from opkg install python3-modules? What does the output from opkg list-installed tell you?

[gurkc006]

Sorry for the delayed answer. Had some test this morning but had to get my system running again, as it's kind of productive... Here are some files from this evening (no solar at the moment, therefore I can test a little...)
I switched to VenusOS 3.20_33 and try using dbus-mqtt-devices 0.6.3 main branch from github. Symptoms are, that the driver seems to crash around every minute, probably when it receives new "device creation messages" via mqtt.
The logfile shows:
vOS_3.20.33_dbus-mqtt-devices0.6.3main.txt
Here are the outputs from list-installed:
vOS_3.20.33_list-installed.txt
and python -m pip list:
vOS_3.20.33_pip-list.txt
Don't know, if that helpes? There seem to be a YAML parse error somewhere.
I will switch back now to VenusOS 3.20_10 with running dbus-mqtt-devices 0.7.5_beta as this is running stable for me at the moment.
Good night