howto add a driver to Venus

Intro

Every once and a while I get a question similar to 'Hi, I have the idea to read data from sensors such and such (for example tank senders) and I want to show that information on a CCGX, how can I do that?', or 'Hi, how can I make the CCGX read data from my ModbusTCP enabled generator?'.

On this page I'll give points on where to start with that.

Please feel free to improve this text to make it easier to understand for a newcomer! Writing these types of explanations while head deep inside is not so easy.

For questions, use the mailing list.

Note that this page refers to Venus devices in General, not only the CCGX or Venus GX.

• 1. What's involved
• 2. Developing a driver
• 3. Installing a driver
• 4. Stability - exceptions handling

1. What's involved

To show the data on the GUI, it first needs to be made available on the D-Bus. D-Bus is the internal databus within Venus. The drivers publish their information on it, and the GUI, and also the ModbusTCP bridge for example, take the data from it. Messages can be sent back on it when the user changes a setting in the device. A schematic overview is given here, the specs of how we use D-Bus are here, and a list of parameters on D-Bus is here: dbus.

So, with that in mind, adding a new sensor (for example a tank sender) or reading a new device (for example a generator) involves the following two steps:

1. Choose a protocol: uart (the VE.Direct ports), tcp/ip, canbus, bluetooth, rs485 (requires a usb-rs485 converter) or any other use of usb.
2. Implement a process that interfaces between the comm port and publishes the data on the D-Bus: the driver

Above is all that is needed for data that is "more of the same", ie. types of sensors and/or devices that are not new to Venus OS. For other data, there is more code in Venus OS that needs modifying:

3. Modify the GUI, adding pages for the new data
4. Modify vrmlogger & the vrmportal, to make the new data available on the VRM Portal - this can only done by Victron.
5. Modify dbus-modbustcp, to make the new data available on ModbusTCP. See github.com/victronenergy/dbus_modbustcp project, especially attributes.csv.

2. Developing a driver

tbd ..

• different languages can be used (ie c, cpp, python), and we have D-Bus drivers for them available.
• A trick to do this with a minimal software effort is to use the dbus-dummy service Python script: make a dbus-dummyservice containing the dbus service name as well as paths you need, make all paths writable from the outside. Then from your external device, which could be a PLC in this case, write to the Venus device with ModbusTCP. For existing data sets, no changes to the modbus mapping list in Venus would be necessary.
• when your product supports ModbusRTU or ModbusTCP, take a look at our modbus client code in python, by installing the latest release candidate a. That code is currently not available on github; hence the need to install. Its in /opt/dbus-modbus-client/.
• when developing in C, Cpp or other language that needs compiling & linking, see our SDK documentation for gcc and more.
• look at existing bridges. For existing data (ie tanks, solar chargers, generators, etc), make sure that the paths exported to D-Bus by your bridge exactly replicate the existing ones. To go back to the tank sensor example, see the readme of dbus-adc for how that should be.
• for something new, stick to the Venus D-Bus API definition, and also discuss the names of D-Bus paths etcetera with us.
• in case there are settings that cannot be stored on the sensor itself, see localsettings for storing them in Venus.
• for automatically running your new software at startup & not losing it after an update of Venus, see here and here.

3. Installing a driver

There are two related ways to "install" a driver. Most drivers are tied to a serial port and thus can be set up to be invoked by serial-starter. For other drivers, see the second technique below.

serial-starter: installing a plug-and-play driver that is tied to a serial port

The service that's messing up your port is the so-called serial-starter. To see serial starter in action, run ps while grepping for your port. You might want to run it a few times.

root@raspberrypi2:~# ps | grep ttyUSB0
 1300 root      1580 S    supervise vedirect-interface.ttyUSB0
 1308 root      1596 S    multilog t s99999 n8 /var/log/vedirect.ttyUSB0
 1390 root      1580 S    supervise gps-dbus.ttyUSB0
 1402 root      1596 S    multilog t s99999 n8 /var/log/gps-dbus.ttyUSB0
 5271 root      3048 S    {start-gps.sh} /bin/bash /opt/victronenergy/gps-dbus/start-gps.sh ttyUSB0
 5284 root      3144 S    /opt/victronenergy/gps-dbus/gps_dbus -v --banner --dbus system --timeout 2 -s /dev/ttyUSB0 -b 38400

In above example you see the gps daemon being started against ttyUSB0, at 38400 bps. All lines showing supervise and multilog can be ignored.

Howto stop serial-starter on a tty port

While developing a driver, just tell serial-starter to skip that port:

/opt/victronenergy/serial-starter/stop-tty.sh ttyUSB0

Howto make serial-starter ignore certain USB types

Alternative to stopping it manually, add a line in /etc/udev/rules.d/serial-starter.rules to make serial-starter ignore the type every time its plugged in. Since the root filesystem is mounted read-only, you will need to remount it to read-write before you can edit the file. There are three ways to change that:

• temporally: issue the command mount -o remount,rw / (which holds until the next reboot or issueing mount -o remount,ro /)
• semi-permanent: issue the command /opt/victronenergy/swupdate-scripts/remount-rw.sh (which holds until the next firmware update)
• permanent: adding one of the above commands to /data/rc.local (which holds permanently)

Then run this code to see the ID for your device:
udevadm info --query=property --name=/dev/ttyUSB0 | sed -n s/^ID_MODEL=//p

Ftdi serial converters, and other brands as well I suppose, can be programmed with a unique ID_MODEL. See ft_prog for ftdi.

If it is unique and you want serial-starter to ignore it, add the following line to serial-starter.rules

ACTION=="add", ENV{ID_BUS}=="usb", ENV{ID_MODEL}=="MY_UNIQUE_ID_MODEL", ENV{VE_SERVICE}="ignore"

(replace MY_UNIQUE_ID_MODLE)

Then restarted the Venus device, or just unplug and replug the USB cable, and then run the ps | grep command a few times again to make sure the serial port is left alone. And/or check the serial-starter log files.

Howto add a driver to serial-starter

Once your driver is completed, and you want plug-and-play to work; you'll need to add your device & driver to the serial-starter.

In brief, the udev & serial-starter do the following for each serial-device detected on the USB:

1. read its ID_MODEL
2. look up the device class in /etc/udev/rules.d/serial-starter.rules
3. look up the available drivers in /etc/venus/serial-starter.conf
4. run all available drivers one by one.

The drivers are implemented such that if they can't detect a device at the other end of the line that they support, they exit again. Serial-starter will then try the next one, and so forth. Restarting from scratch and forever continuing when none of the drivers sticks.

Incomplete pointers explaining how to add your own driver:

Prerequisites:

1. install (or at least symlink in case you have your code on the data partition) your software in /opt/victronenergy/ like all the drivers and also other executables are. To make you code survive a firmware update install your file in a folder under /data/ and set up a symlink e.g. /data/etc/your-program with a symlink like ln -s /data/etc/your-program /opt/victronenergy/service-templates/your-program. Add/Create a file called /data/rc.local that restore this symlink if it does not exist (like after a firmware update). /data/rc.local is run at boot by VenusOS.
2. make sure there is a service directory for your program in /opt/victronenergy/service-templates/your-program. Look at another driver, for example vedirect-interface/service. Note the TTY placeholder in the run file and /log/run file. TTY and PRODUCT are replaced with values by serial-starter, so for example use TTY to pass the device port to your driver. Also check its start.sh.
3. since USB devices trigger the serial-starter via udev, simply restarting the serial-starter is not enough. So, reboot the device, or replug your USB device to make the magic work. Alternatively you can do echo 0 >/sys/bus/usb/devices/1-1.2/authorized; echo 1 >/sys/bus/usb/devices/1-1.2/authorized where 1-1.2 is the ID for the cable (eg search dmesg for ttyUSB0)
4. and check logs: tail -F /data/log/serial-starter/current | tai64nlocal and tail -F /data/log/your-program+port/current

Then:

1. determine the device Id of your serial cable: udevadm info --query=property --name="/dev/ttyUSB0" | sed -n "s/^ID_MODEL=//p"
2. add/modify a line in /etc/udev/rules.d/serial-starter.rules that maps that ID_MODEL to a device class.
3. add/modify a line in /etc/venus/serial-starter.conf that maps your device class to your driver (name of driver directory in /service). You can also add this to the file /data/conf/serial-starter.d which will be included from /etc/venus/serial-starter.conf and survive a firmware update.

Installing a driver that doesn't depend on a serial port

You may wish to add a driver that doesn't connect to a serial port and, thus, is not something that would be invoked by serial-starter as described above. For example, like the dummy dbus service example, you may wish to publish a set of objects to the D-Bus that are writeable by an external service, to surface data onto Venus from somewhere else. Another example might be a driver that filters or re-processes some of the existing sensor data on the D-Bus into a new service, e.g., smoothing the data coming from a tank level sender.

For these situations, follow these steps:

1. Set up your new driver under the /data disk so that it will survive software updates.
2. Add a run script in your service's directory that daemontools can use to invoke your service.
3. Optionally, add a log/run script that daemontools can use to turn on log management for your service. See the daemontools FAQ or another Venus service for an example of how to do this.
4. Symlink your service directory to /service so that it will be invoked automatically at startup.
5. Optionally, add some shell script logic to /data/rc.local to re-symlink your service if the symlink goes missing. rc.local is invoked at system startup and survives new Venus updates, whereas your symlink will not survive. By adding a check in rc.local you can ensure your service will be scheduled to start again after upgrade.

How to make changes that don't get lost on a firmware update

See the root access document.

4. Stability - exceptions handling

Once officially implemented and added to Venus, the process will be managed by daemontools. And in some cases (when it involves a tty) also under the serial-starter.

Which means: don't code too defensively. Don't try to recover from an unrecoverable error situation. Instead, keep the code clean, make sure it exits and let the baby-sitters (daemontools & serial-starter) take care of you.

Do make sure the code does not hang in a weird state.

In more detail:

• when the tty disappears: just exit. serial-starter will restart the driver in case the tty comes back.
• when whatever strange thing happens: just exit. daemontools will fix that, and provide for a clean start.
• when the dbus disappears: doesn't matter what will happen: Venus will reboot anyway in such case. No need to add code to properly exit in such case; and most probably your code will crash anyway. Disappearing dbus-es is not an use case.

Notes for Python

• without proper precautions, a thread can crash without taking down the full process with it: not good, a complete crash is better. See daemon=True, for Python.
• when you use gobject.timeout_add, then the callback you specify must return True if it wants to be rescheduled. But if there is an exception in that callback that is not handled, the callback is not rescheduled. The main thread continues running though; leaving the program hanging. We have exit_on_error() for that.