Parameter declaration in controller to avoid boilerplate coding in plugins

[bernhardlang]

With @seb5g I recently discussed a possible new way to declare an instrument plugin which could avoid quite some boiler plate coding when values and parameters are to be handled in a common, generic fashion.

The idea: a class variable of type dict in the controller declares the facilities of the controlled physical device (actuators, detectors, global parameters, per channel (actuator or detector) parameters and, if necessary, the corresponding getter and setter methods. The corresponding plugins are then created on the fly from this dictionary. On the other hand, if any specialized operations are required which cannot be performed inside the controller by the setter/getter methods, the existing way of declaring a plugin can be taken.

advantages:

• double coding in controller and plugin as well as boilerplate code in the plugin are avoided in generic situations
• 100% backward compatibility because the feature is added on top of the existing infrastructure
• possible inspiration from pymeasure's way of declaring drivers

The declaration could be done in the following spirit (example of an Avantes line scan spectrograph with 10 digital outputs, two analog inputs and two analog outputs)

devices = [
  {'name': 'spectrometer', 'device-type': 'detector 1D', 'data-type': 'uint16', 
   'channels': 1, 'axis': True,
   'parameters': [
       {'name': 'integration_time', 'title': 'Integration time [ms]',
        'type': 'float', 'min': 0.025, 'max': 60000, 'storage': 'persistent' },
       ],
   },
  {'name': 'analog-in', 'device-type': 'detector 0D', 'data-type': 'uint16', 'channels': 2 },
  {'name': 'analog-out', 'device-type': 'actuator', 'data-type': 'float',
   'channels': 2, 'storage': 'volatile', 'min': 0, 'max': 10, 'default': 0 },
  {'name': 'digital-out', 'device-type': 'actuator', 'data-type': 'bool',
   'channels': 10, 'storage': 'volatile', 'default': False },
]

parameters = [
    {'name': 'serial_no', 'title': 'Serial number', 'storage': 'persistent', 'readonly': True},
]

The controller would have to provide the methods

def open_communication(self):
    ...

def set_param(self, value, device_name, param_name, channel=None):
    ...

def get_param(self, device_name, param_name, channel=None):
    ...

def set_actuator(self, value, actuator_name, axis_name=None):
    ...

def get_actuator(self, actuator_name, axis_name=None):
    ...

def get_axis(self, detector_name, channel=0):
    """Sends a numpy array with wavelength values"""
    ...

def grab_data(self, detector_name, channel=0):
    """Sends a numpy array with pixel values"""
    ...

In case that for whatever reason other names have to be used, a corresponding entry named 'getter' / 'setter' / 'grab' has to be provided.

A different approach for the controller could be that it exposes an object per detector and per actuator axis on which the getter and setter methods are implemented as python properties.

Please comment ...

[Ashwolaa]

So the approach I was actually following with the "Capabilities" and Observables/Variables share some features with it.
The idea was to have an entry similar to the oen you give in devices of this sort:

from pymodaq.control_modules.capabilities import (
      Capabilities, Observable,
      ContinuousVariable, DiscreteVariable,
  )

  # ── 1. Simple actuator (single axis motor) ───────────────────────────────────
  class MyMotorPlugin(DAQ_Move_base):
      _declared_capabilities = Capabilities(
          variables=[
              ContinuousVariable(
                  name='position',
                  label='Stage position',
                  units='mm',
                  lo=-50.0,
                  hi=50.0,
                  epsilon=0.001,   # move-done tolerance
              )
          ]
      )

  # ── 2. Multi-axis actuator ───────────────────────────────────────────────────
  class MyXYStagePlugin(DAQ_Move_base):
      _declared_capabilities = Capabilities(
          variables=[
              ContinuousVariable(name='x', label='X axis', units='mm', lo=-100, hi=100),
              ContinuousVariable(name='y', label='Y axis', units='mm', lo=-100, hi=100),
          ]
      )

  # ── 3. Discrete actuator (filter wheel with named positions) ─────────────────
  class MyFilterWheelPlugin(DAQ_Move_base):
      _declared_capabilities = Capabilities(
          variables=[
              DiscreteVariable(
                  name='filter',
                  label='Filter wheel',
                  units='',
                  shape=(1,),          # one value at a time
                  choices=['open', 'ND1', 'ND2', 'ND3', 'blocked'],
              )
          ]
      )

  # ── 4. Mixed detector: spectrum + scalar temperature sensor ─────────────────
  class MySpectrometerPlugin(DAQ_Viewer_base):
      _declared_capabilities = Capabilities(
          observables=[
              Observable(
                  name='spectrum',
                  label='Intensity spectrum',
                  units='counts',
                  dtype='uint16',
                  shape=(2048,),       # 1-D array of 2048 pixels
              ),
              Observable(
                  name='temperature',
                  label='CCD temperature',
                  units='degC',
                  dtype='float64',
                  shape=(1,),          # scalar
              ),
          ],
          variables=[
              ContinuousVariable(
                  name='exposure',
                  label='Exposure time',
                  units='ms',
                  lo=1.0,
                  hi=60000.0,
              ),
              DiscreteVariable(
                  name='gain',
                  label='Amplifier gain',
                  units='',
                  shape=(1,),
                  choices=[1, 2, 4, 8],  # integers are fine too
              ),
          ]
      )

When loading the plugin, it would go through a method called: infer_capabilities which would make the associated DAQ_move / DAQ_viewer.

Then one has to give specific methods that match the name of the variables and the observables.

In practice, I was thinking as having two methods (name is not important):

def  query_data(self, name):
     if name is "X":
           return self.controller.method_to_get_X()

def change_to(self, name, value):
   if name is "X":
        self.controller.method_to_change_X(value)

• When loading the plugin, it would go through a method called: infer_capabilities which would make the associated DAQ_move / DAQ_viewer according the listed_capabilities.
• There would be an extra step to update and add capabilities that can only be obtained when the instrument is running (for example getting the correct shape of the data).
• Lastly, the idea was to add an extra keyword/context menu entry to the param_settings to propose to convert it to a DAQ_move / DAQ_viewer.

What I see though with your example is that the concept of channel/axis can be pretty relevant. In theory, no need to repeat X times an entry if they are just different by an index. This can still be fairly well handled with a list comprehension without too much boiler plate though but I am thinking that DAQ_move could maybe handle multiaxes (with similar settings) more directly.

[seb5g]

I see various things we should clarify.

1. how to extend the capabilities of existing plugins
2. how to better code future plugins (or rewrite existing) to simplify development while inherently use any of the instrument setting as a "Capability" either read/write

Details:

1. how to extend the capabilities of existing plugins:

   • principle:
     The goal of this is to be able to dynamically create/add a DAQ_Move/DAQ_Viewer from any writable/readable settings in any of the "usual" instrument plugins. This would allow somehow to quickly use for instance, a exposure_time setting as a DAQ_Move in the Dashboard hence do a scan with it as variable parameter. For this one has to create a special DAQ_Move_FromSetting object that would set/read settings from an associated "real" DAQ_Move/DAQ_Viewer. I think this approach is 1) easy, 2) dealt with on the DashBoard level (so no need to write new code in an instrument plugin) and 3) will provide all flexibility people are looking sometimes and that could so far be done only by reimplementing an instrument plugin using the same controller object (master/slave) stuff.
   • implementation:
     1. Either: The DAQ_Move_FromSetting object would receive a "controller" being an object able to send signal to the real associated plugin and listen to signal from the real associated plugin. This would be very light and quite efficient using signal/slots from Qt
     2. Or: As this implementation would not "send" the DAq_Hardware to another thread we can pass to it the instance of the DAQ_Move or DAQ_Viewer or simply their settings object to be write/read (simpler than 1)

2. how to better code future plugins
   As you both mentioned the list of Capabilities would be coded within the call preamble of the instrument plugin. However one should NOT require the "controller" object to be changed or updated because most of the time we are using controller we are not controlling ;-) (I mean we are not the author and it would be complex to do/wait for a PR to be accepted before one can use the updated controller). This is why the Capability object should have a setter and/or getter defined from a method or property of the controller object. The question of still using axes should be thought of. Maybe one could create a AxisCapabaility or a ChannelCapability adding it its creator a ref to the rigth channel (just like in pymeasure)

[bernhardlang]

Turning any settings parameter into something which can be controlled by a move is a very good idea. We talked about this during and after the visio on #954. But I forgot about that aspect, brain's getting old ...

However, I had something somewhat different in mind, namely in the situation where it is me who writes the controller and has therefore full control over its code (which is the case for all the instrument plugins I've written so far). Since the controller has to be aware of the capabilities of the device, why shifting their declaration to the plugin? Simply because that's the way it is done until now and we don't want to break that.

My idea goes along the following lines: during search for declared plugins, scan also the hardware folder for a controller class inherited from a base class / identifiable by naming convention or whatever, which declares the capabilities of the controlled device in a dict which is compatible to the Parameter class. A factory for DAQ_Move_FromSettings and DAQ_View_FromController would extract the necessary dict items from the declaration in that controller and inject them into the plugins to be created. The mechanism for doing so can be the same as what you propose for a plugin from settings. If there's no such controller or declaring dict, no action is taken, assuming that the "classical road" is taken.

In such scenario, the only thing the coder has to do, once being able to talk to the device using python, is to declare the capabilities in a controller class and write some getter/setter methods with appropriate calling convention where needed. The plugin creation is then done by PyMoDAQ's initialization machinery.

[Ashwolaa]

I see a lot of overlabp between your view and what was suggested in #664 by @TheFermiSea

[seb5g]

It seems to me a bit impractical for the reason that controller are not often produced by the plugin author (maybe in your case it's True Bernhard but that's not the most common case). Morevover I would advice to add the controller/driver to other packages dedicated to this (pymeasure, pylalblib). Like this other people would benefit from your work at the cost of following what's recommended in those packages. Then we would have full customization from the plugin we would have to work on anyhow (even if only the list of capabilities). And because we would have to deal with many types of controllers/drivers we would need anyhow some tweaking within the plugins... So overall I would prefer we go along the lines of what @Ashwolaa wrote above I find already quite advanced

[bernhardlang]

Morevover I would advice to add the controller/driver to other packages dedicated to this (pymeasure, pylalblib). Like this other people would benefit from your work at the cost of following what's recommended in those packages.

Very important point indeed!

I was actually more thinking about a mixin approach which could also make already existing controllers (in the wide sense) "fit for PyMoDAQ". I'll probably give it a shot when I'm fully in the plugin/controller business for my transient absorption experiment again.