New driver interface for tomato-1.0

[PeterKraus]

Motivation

In the tomato-0.2 branch, the driver and job interfaces were basically merged into one piece of code: each job talked to each physical device separately, which was causing some race conditions (#28). With only one supported type of device, i.e. the biologic driver, this kind-of made sense at the time.

In tomato-1.0, we want to pave the road for a "device dashboard", meaning the device statuses have to be accessible from outside of the jobs. As a reminder, the relationship between jobs and devices in tomato-1.0 is shown below:

Basically, each device (a digital twin of a physical device) is managed by a driver, and there is only one driver process running managing all devices of that type. All communication with the physical device is therefore handled by the driver, the individual physical devices (and channels within them) can be addressed when one knows the device address and channel.

Requirements

1. We would like the new driver interface to handle a wide range of physical devices, e.g.:

• thermocouple readers, e.g. PicoLog, which are almost completely read-only
• volumetric flow meters, e.g. Mesa DryCal, which have a few adjustable parameters, have a start/stop function
• mass flow controllers, e.g. Bronkhorst, which have setpoints
• temperature controllers, e.g. Jumo, which might have ramps
• gas chromatographs, e.g. Fusion, where a measurement might take ~ 5 minutes and might have to be scheduled
• potentiostats, e.g. Biologic, where a single set of instructions can contain a whole cycling protocol

2. The rest of tomato should be completely driver-agnostic, i.e. everything relevant for the measurement comes from the driver (available parameters, units, adjustable limits, etc.). This means the list of techniques and parameters, i.e. the driver specific language (DSL), has to be defined and documented in the driver docs.

3. Some functionality, e.g. task scheduling or conditional interruption, should be probably implemented just once and made available to every driver via specific keywords. Currently I can think of the following examples:

   • if I want to measure temperature for 10 minutes every 15 seconds, but only poll for new data every 60 seconds, I need to be able to tell the driver that it's supposed to communiate with the device with a 15 second resolution, caching the data. Then, tomato calls the driver's get_data() every 60 seconds, and sends a stop signal after 10 minutes.

Implementation

• Each driver can be a separate python package, e.g. tomato-biologic or tomato-bronkhorst, for easier maintenance.
• Tomato provides a central abstract Driver class, which is inherited from and exposed by in these packages. The current model for the class looks like this:
  driver_proto_v01.txt

Design questions

• what can be abstracted safely and be handled tomato-side?
• are we missing any key functionality?
• how to indicate Driver features (e.g. long acquisition time in GC requiring scheduling, or batching of requests for multi-channel devices)?

[PeterKraus]

Notes from our meeting on the 22th of May:

• reading, concatenating and dumping device output into pickles after every task_data() call will be costly when files grow large; this idea might need further thought.
• need to design a Driver-agnostic vocabulary for things like:
  • maximum task runtime which limits a task run length
  • task samplerate which is the measurement rate for more basic Drivers, ie how often the Driver calls dev_get_data()
  • task pollrate which is the device polling rate for all Drivers, i.e. how often the job calls task_data()
• the Driver should be able to signal to the job once a task is finished
  • perhaps by passing a flag via zmq
• the Biologic driver allows querying of multiple channels:
  • the channel arg of the Driver interface can be optional, in which case all channels sharing an address get polled.
• the Biologic driver allows the submission of multiple tasks:
  • perhaps via a third status flag, such as can_queue

[PeterKraus]

ToDo for tomato-1.0:

• implement Driver (abstract) class and re-implement the example driver using it: by mid June
• implement first basic version of tomato-bronkhorst by the end of June together with @AlexN7-Shepard
• implement support for tomato-drycal, perhaps @NukP is interested in contributing?
• reimplement support for Biologic devices in tomato-biologic once Driver class is available, with @g-kimbell
• update Aurora-related functionality with @edan-bainglass
• aim for tomato-1.0 release by end of Oct 2024

[NukP]

@PeterKraus Can you elaborated on 'implement support for tomato-drycal' more? Do you mean writing a driver that handle the communication (write and read data) between tomato and the DryCal flow meter?

[PeterKraus]

Yup, basically incorporating your work on the DryCal into tomato. We also have one in Berlin, so testing would be fairly easy.

[NukP]

@PeterKraus OK, I should be able to allocate time for this (hopefully it doesn't take long). I have a look at the driver folder in the Master branch and from what I talked to Graham, do I understand it correctly that you haven't finished designing the dummy driver? If so, I would wait until this is the case.