D-Bus ItemsChanged: reduce CPU load caused by D-Bus traffic

[mpvader]

Update 2021-12-15: after some back and forth, this was finally implemented as "ItemsChanged". Its now been done in all important services.

Reducing D-Bus traffic - increasing scalability

For example a solar charger, sends multiple PropertiesChanged signals every second, one for each D-Bus path. Also cgwacs and dbus-fronius do: 9 signals per second per PV inverter. And for reference, on a CCGX, the max signals per second is approx 30.

The idea is to send only one every (for example) second, from root, with a Dict with all paths that have an updated value.

The plan:

1. 1. Proof of concept and prepare the used Python, C as well as Cpp libs for both receiving and sending dict-updates. Mostly done, except for transmitting from Cpp-based services.
2. 2. Update all consumers
3. 3. Update the heavy users: vedirect-interface, vecan-dbus, dbus_fronius and cgwacs - and maybe mk2-dbus if not too difficult?
4. 4. (ibu) Report again on the difference in CPU load; and discuss its consequences:
   • can the plan to rewrite dbus-mqtt to something other than Python be dropped?
   • increase specified max number of VE.Direct devices?
   • are there more consequences?
5. 5. Update the rest of the services, will probably happen only some day
6. 6. Once most are over, we might want to move the rest over as well, including localsettings, so we stick to one method, one implementation. To be discussed/decided then.

Services that consume (and thus need updating first)

• dbus-generator-starter - tracks values from systemcalc, should be an easy swap
• dbus-pump - Tracks values from systemcalc, should be an easy swap
• dbus-systemcalc-py - Tracks values from lots of services, easy swap
• dbus-mqtt - Uses its own PropertiesChanged handler, not velib_python, but shouldn't be difficult
• vrmlogger - Tracks values from lots of services, easy swap
• gui - tracks lots of services - easy and already done
• hub4control - tracks systemcalc, mk2-dbus, and dbus-fronius - easy and already done
• dbus-modbustcp - tracks lots of services - should be easy; not done yet.
• dbus-recorder - essential for recording, not essential for demo mode; has its own code to record, not as simple as swapping velib_python but needs doing!
• dbus-spy
• dbus-vecan
• mk2vsc
• dbus-vebus-to-pvinverter
• mqtt-rpc

Consumers that are not immediately required:

• vesmart-server - Tracks values from localsettings only, so not essential, but making it compatible with dict updates is easy since using velib_python for dbus receiving.

Producers of highest prio:

• vecan-dbus
• vedirect-interface
• dbus-cgwacs
• dbus-fronius
• dbus-systemcalc-py (systemcalc also produces)

Producers that can be done some day:

Python:

• dbus-bornay-windplus
• dbus-digitalinputs
• dbus-fzsonick-48tl
• dbus-modbus-client - becoming more relevant due to it being used for AC load monitoring
• dbus-imt-si-rs485tc
• dbus-modem - only tracks localsettings which does not send dict updates, no need to produce dict updates
• dbus-vebus-to-pvinverter
• localsettings - doesn't use velib_python, doesn't subscribe to PropertiesChanged, doesn't have to send dict updates
• mqtt-rpc - does not monitor dbus, does its own GetValue/SetValue when required
• venus-button-handler - only tracks settings from localsettings, which does not send dict updates

C:

• can-bus-bms
• dbus-adc
• gps-dbus
• dbus-ble-networking (this is not in use!)
• dbus-valence
• mk2-dbus
• mqtt-n2k - bridges can-bus and mqtt, not a dbus player.
• dbus-motordrive

Cpp:

• dbus-qwacs - nobody knows what this is using; perhaps this is the moment we drop support for those meters and remove dbus-qwacs entirely.

Notes per language:

Python:

In the majority of cases all that is required is updating velib_python after merging dict_updates. In some cases, a project might need a small change, ie. using the with statement notation, to send updates in a batch.

[mpvader]

vedirect-interface branch is here:
https://github.com/victronenergy/vedirect_interface/commits/thiemo/WIP_dict_updates

velib branches are here:
https://github.com/victronenergy/velib/tree/thiemo/WIP_Item_Dict_Leafs
https://github.com/victronenergy/velib/tree/thiemo/WIP_Item_Dict_Producer (added)

velib_python is here:
https://github.com/victronenergy/velib_python/tree/dict_updates

[izak]

Some results from testing.

Test environment

1. Venus 2.65 base system
2. vedirect-interface replaced with a binary that sends dict updates on root
3. systemcalc and vrmlogger patched to use velib_python that understands dict updates
4. gui recompiled with new velib so it understands dict updates
5. Rest of the system left as is (should have minimal effect on CPU usage).

This was tested vs an unmodified setup.

First test: Test effect on CPU use on a working system

System has:

1. CCGX
2. 2 x VE.Direct solar chargers
3. 1 x Multiplus
4. 1 x BMV 700
5. System was set up to feed solar power into the grid, so that there would be constant power changes to communicate.

Unoptimised system

CPU use:

• systemcalc: +-10%
• vrmlogger: +-10%
• dbus-daemon: +-6%
• vedirect-dbus: +-2%

CPU idle: +-30%

Optimised system

CPU use:

• systemcalc: +-8%-9%
• vrmlogger: +-8%-9%
• dbus-daemon: +-5%
• vedirect-dbus: 3%-5% (slight increase)

CPU idle: +-40% (handsome improvement)

Second test: DBus throughput on a heavily loaded system

System has:

1. CCGX
2. 2 x VE.Direct solar chargers
3. 1 x Multiplus
4. 1 x BMV 700
5. System was set up to feed solar power into the grid, so that there would be constant power changes to communicate.
6. An additional 5 simulated solar chargers that change voltage/power/current values once a second.
7. dbus-statistics look specifically at PropertiesChanged signals and ignores SetValue/GetValue calls.
8. Statistics were assessed using wireshark and dbus-monitor's --pcap option.

Unoptimised system

• Dbus RTT: > 5000ms (round trip time was over 5 seconds)
• Dbus traffic: 7400 bytes/sec
• Messages per second: 37
• Average packet size: 197 bytes
• dbus-daemon CPU load: 10%-15%
• Watchdog rebooted the system as load average exceeded 10.

Optimised system

• Dbus RTT: > 100ms-200ms
• Dbus traffic: 5900 bytes/sec
• Messages per second: 15
• Average packet size: 388 bytes
• dbus-daemon CPU load: 5%
• Load average was still very high (due to simulated solarcharger processes).

Very heavily loaded optimised system

Another two simulated solarchargers were added, for a total of 7 simulated and 2 real solarchargers (plus a BMV).

• Dbus RTT: 2000ms
• Dbus traffic: 6400bytes/sec
• Messages per second: 17
• Average packet size: 376 bytes
• dbus-daemon CPU load: 5%
• Watchdog rebooted due to very high load, even though the dbus-daemon remained fairly responsive

Conclusion

• Overall CPU idle time was increased (good)
• dbus is no longer the bottle-neck
• Messages/second reduced to a little less than half (good)
• Message size roughly double what it used to be
• dbus-daemon CPU use remains low (good)

Other observations

• vrmlogger's rtt implementation is skewed under heavy load, because the load retarts the event loop and increases the measurement.
• There are still a lot of improvement to be had by also implementing this in dbus-cgwacs and dbus-fronius, especially in a three-phase system where these two create heavy traffic, and especially where multiple PV-inverters are present. This was not yet done as it requires work on velib to also send dict updates.