Additional repository for DLL/COM devices?

[BenediktBurger]

Introduction

Currently, pymeasure is geared towards devices, which exchange serial messages (mostly, but not only, ASCII) without the need for further libraries.

There are, however, devices, which require special manufacturer's libraries like DLLs or via COMs (component object model).

Obstacle

I do not see, how they could profit from the pymeasure instrument core, nor how we could test or maintain them reliably.
Therefore, I do not like the idea to include them in pymeasure itself.

History

• One issue, which asked how to implement them, was The best way to implement DLL/COM devices #246 .
• A more generic question in the same direction is Question regarding prefered implementation of non-SCPI devices #296 .
• Thorlabs DLL instruments: Generic Powermeter and Chrolis LED source #417 was closed.
• DAQmx is not an Instrument #662 discusses the place of the current NI instrument (which uses such a dll) in pymeasure
• Dev/thorlabs kpz101 #1057 wants to merge such a device.
• I have some in the backhand, but did not consider them for pymeasure for above mentioned reasons.

Possible solution

One possibility could be opening another repository, which collects such instrument drivers.
These drivers should be in the style of pymeasure (properties) with according doc strings (more we cannot enforce, due to the libraries).

Pros

• These instrument drivers could be shared among users

Cons

• Maintenance is quite difficult, as you cannot inspect the driver itself (I experienced it with a picoscope driver: you have to look at the code and guess, whether it works or not)
• Dependencies will be different for each instrument.
• Each driver will require knowledge about that instrument to maintain it.

Question

What do you think @bilderbuchi , @CasperSchippers , @msmttchr ?

[bilderbuchi]

I do not see, how they could profit from the pymeasure instrument core, nor how we could test or maintain them reliably.
Therefore, I do not like the idea to include them in pymeasure itself.

I agree that they are not a good fit, as they won't need most of the data/command string wrangling features that we provide. However, common abstractions like #416 could still be useful for them.

One approach could be to hide all the dll/COM stuff behind a custom adapter, define the properties with the usual command strings, and have the adapter's read and write trigger calls of the appropriate dll functions (e.g. with pattern matching from py310) on the command string.
Then the fit to pymeasure would be more natural, and the dll/COM nature is contained in the adapter. Of course, implementation effort is higher as one would need to implement both the properties and the associated clause in the adapter.
Maybe there is a better way?

I agree that we should not pull these driver binaries (and any associated dependencies) into pymeasure itself, this is a scope limitation for pymeasure that I fully agree with.

However, what we might be able to do is to define and document a "plugin" format, where we could allow others to implement and maintain such drivers in a separate package, but allow them to add into e.g. a pymeasure.instruments.external namespace, e.g. using the "namespace package" mechanism. See https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/
I have not prototyped any of this, though.
We have to think though what kind of stability expectations we can maintain (so that those authors are not surprised by pymeasure changes). Another thing is what constraints such a plugin mechanism puts on us.
Maybe this is more reasonable to do once we hit 1.0?

Maybe one way to prototype/try this is to pull the NI things we talked about removing (can't find the issue right now) into such a plugin, and see how that works out? Not sure if we then want to maintain it, though. Considering our workload, I would say we should not spend too much effort on these out of scope instrument types.

[BenediktBurger]

Thanks for your ideas

Maybe one way to prototype/try this is to pull the NI things we talked about removing (can't find the issue right now)

It's #662, I added it to the history section in the initial post.

Not sure if we then want to maintain it, though. Considering our workload, I would say we should not spend too much effort on these out of scope instrument types.

I agree.

[msmttchr]

@BenediktBurger, I think I understand your question, but I am afraid I have no proposal except, perhaps, to avoid supporting such kind of instruments.
Relying on external binary code will make pymeasure:

• difficult to maintain
• difficult to test
• non portable across OS
• not clean wrt licensing issues

I think we all agree on these points and sadly I have no proposal how to handle the proposal.
I appreciate the proposals made, but still I think it's better to do nothing.

[BenediktBurger]

The problem is, that for some instruments, you have to use external binary code and you have to write your python code around that.
Obviously, every user could write their own code, but I prefer to share that code for others to use.
In some cases, there are repositories for devices by one manufacturer or a large collection (pylablib), for others there are not.

It's sad, that it is so difficult to share that work.

[seb5g]

I'm interested in that discussion also because for pymodaq development of some plugins we had to develop our own python driver on top of many DLLs/COM/.Net libraries. Ah the moment we keep those into the corresponding manufacturer plugin together with the actual pymodaq plugin using this driver. But it's sad as not all people can easily benefit from that work as there is no clear visibility of such driver.

On the other hand, more and more. Manufacturer are offering python driver of their own dll, making the need of such a work a bit less important...

[BenediktBurger]

What about a single Repository per manufacturer (if there is no python repo yet), maybe under the pymeasure organization.

Whenever someone wants to maintain one Repository, he may do so.

The quality of these repositories will vary, but it will be available for people to use.

As they are part of the pymeasure organization (or some other), the pymeasure organization may appoint new maintainers, if the previous one does not respond anymore.

[seb5g]

I'm ok with this as pymeasure already has good visibility for its drivers. Is there some kind of base class/interface that would be nice to have for those?

[BenediktBurger]

However, what we might be able to do is to define and document a "plugin" format, where we could allow others to implement and maintain such drivers in a separate package, but allow them to add into e.g. a pymeasure.instruments.external namespace, e.g. using the "namespace package" mechanism. See https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/
I have not prototyped any of this, though.

I like the name space approach and prefer it over a mere naming convention (starting with "pymeasure_".

Is there some kind of base class/interface that would be nice to have for those?

A base class is difficult, as these instruments differ a lot. Using a template Repository (like pymodaq plug-ins template), could be useful, though.
Also, we plan to introduce more generic base classes (as mixins, see #416 ), which could serve as a template.

[seb5g]

The naming convention is useful if you want to browse automatically pypi for compatible plugins, what I did for pymodaq. Especially because plugins could be hosted under different accounts or organisations.

Once the package is installed, the entry points mechanism is really nice to discover and expose compatible objects. For instance instrument classes, ...

[BenediktBurger]

Thanks for your comment and thanks for joining the discussion.
Your experience with plug-ins helps.

[CasperSchippers]

I do not have a clear idea what the best approach would be.
One thing I do 'wory' about (but maybe, if we're shipping it of into one or multiple separate that is not our responsibility anymore?) is that you have a separate repo with no maintainer, the barrier of contributing something to that repo might be a bit higher (if that would mean that the contributer would have to 'maintain' it themselves).

[bilderbuchi]

is that you have a separate repo with no maintainer, the barrier of contributing something to that repo might be a bit higher (if that would mean that the contributer would have to 'maintain' it themselves).

Especially with the "one repo per manufacturer, in the pymeasure org" approach, I'm concerned we will end up with N (potentially >>1) repositories in our org that will become basically unmaintained and put an undue attention load on us (what we can ill afford) for a kind of driver that is kinda out of scope for pymeasure.

I'd rather we

• define an expectations (interface/guidelines) for a pymeasure plugin; maybe look at how pytest deals with this
• let other people host and maintain plugins themselves

Then, if there is a plugin with a lot of interest and a stable maintenance situation, we can always pull it into the pymeasure org later (but the other way round will be harder to justify).

OTOH, pooling developer effort in the org, and have all the permissions etc sorted out in a failsafe manner if a maintainer disappears, definitely has benefits. Potentially, we could attract more maintainers for all the pymeasure projects. Pooling drivers and development effort in one place was one of the motivations for pymeasure for a long time, after all.

So, there's arguments for both ways. I don't know what's better. 🤷
Maybe pulling #662 out of pymeasure itself, but host it in plugin form in the pymeasure organisation could serve as a good test balloon, also for the plugin design work that we would need to do anyway. Then, if there is more interest for more plugins down the line, we can re-evaluate.

[BenediktBurger]

Ok, let's move the NI driver to its own repo as a pymeasure plugin, developing the plugin system in this way.

[BenediktBurger]

From the three plugin methods, I like the namespace, as it allows to import from pymeasure.plugins without the need to write a search routine for entry points and the need to import all plug-ins instead just the needed plugin file.
I have to investigate the pros and cons.