Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[discussion] InstrumentKit #53

Closed
scasagrande opened this issue Oct 4, 2016 · 56 comments
Closed

[discussion] InstrumentKit #53

scasagrande opened this issue Oct 4, 2016 · 56 comments

Comments

@scasagrande
Copy link

Hello pymeasure devs,

I just wanted to take the time to bring a similar project to your attention, InstrumentKit, which I started several years ago with similar goals. Our code architecture actually looks surprisingly similar, so IK shouldn't be too foreign.

A few things that you will notice that IK has are handling of unitful quantities, extensive test coverage, and additional connection interfaces one can use.

I hope that we can learn from each other so that we can continue to help the instrumentation community.

@cjermain
Copy link
Member

cjermain commented Oct 5, 2016

Hey @scasagrande, thanks for mentioning InstrumentKit! We do indeed have a very similar design, which I think is a great sign. I know that there are also a number of other similar projects offering instrument communication. I think these (and PyMeasure) arise from a lack of a central project leading the way, so many scientists wind up reinventing the wheel. I think we could all benefit from a central package that covers instrument control, which would allow PyMeasure to focus on experiment execution and GUI display, instead of the instrument drivers. What do you think about this idea?

@scasagrande
Copy link
Author

I think that is a fantastic idea. Its actually one that I've tried to advocate to people in the past! I've had people ask me if I would consider including GUI display elements into IK, but I just never felt that it was a good mix to have both tied together in the same software package. I'd want to be able to develop, push, and publish them independently of each other.

Regarding your comment on "reinventing the wheel" constantly, that's actually why @cgranade and I started IK a few years ago; we got tired of watching our labmates constantly write communication code again and again. Since then I've tried to raise awareness of the project, as I meet relevant parties IRL.

Let's keep this conversation going, I like what I've read so far!

@cjermain
Copy link
Member

cjermain commented Oct 7, 2016

Yea, both @grahamrow and I had written essentially the same package for instrument control and that prompted us to combine them into the PyMeasure package. I think a separation of GUI and instrument drivers is a good idea for maintenance -- much more of a Unix style. I will plan to do a deep dive of InstrumentKit to see the comparison. @scasagrande, how flexible is your API at this point?

I think the way to make this work is by putting some effort into marketing, since people are reinventing this because they can't find the existing solutions. I think its critical to have some nice landing pages that are not the docs, which show off the features in a modern style. I think a blog that details feature updates is also a great idea, which GitHub can facilitate with Jekyll. Are you open to doing some marketing?

There are other nice packages such as Lantz that could be included as well in this discussion. @hgrecco, do you have any thoughts on how to go about putting together a common package for instrument control?

@scasagrande
Copy link
Author

I suppose it depends on what you mean by API, be it the external or internal developer ones. But hey, I'm always open to ideas and suggestions.

Re marketing I'm on board. I've been trying to get the word out, but you're right, I haven't made a website for the project (I'd lean more towards Pelican rather than Jekyll in keeping with the Python theme). I applied to speak at SciPy con and Pycon Canada this year, but sadly was rejected for both. I do bring the project up a lot at my local Python group. A few of us have also been considering writing a paper for the project.

@bilderbuchi
Copy link
Member

bilderbuchi commented May 30, 2017

Lantz development seems to have largely stopped, sadly.
Also, I can confirm the observation of many groups/people reinventing the wheel or unsuccessfully trying to "unite" people under their banner (also what lantz tried to do) - in my search for python instrument control solutions I found many different, half-finished or overly specialized packages :-( - a classical case of https://xkcd.com/927/.

Interestingly, neither pymeasure nor instrumentkit have hit my radar until now! Both look exciting - on the one hand: Procedures! GUI! On the other hand: Units! On both hands: non-dead development! 😀

@bilderbuchi
Copy link
Member

Additional thought/suggestion: Maybe it would make sense to combine both your packages into Lantz - it already has units support (pint, created by @hgrecco afaik), but is lacking "Procedures" (which pymeasure offers), and has, in my experience, better marketing/visibility. Also, merging/porting the drivers would also create a bigger driver library. I don't know if the whole thing would work API- or design-wise, though.

@scasagrande
Copy link
Author

Lantz is something that I've looked at in the past. They use pyvisa under the hood for everything, which was the exact thing I was trying to get away from. Their project doesn't seem to be designed to easily accommodate other connection libraries.

@cgranade
Copy link

By way of agreement with @scasagrande, one of the major goals of InstrumentKit was to be flexible with respect to communication layers. We didn't want to mandate the use of VISA as that's a very large dependency and as that makes a lot of assumptions about instrument connectivity that aren't appropriate for all users. For instance, we've had users successfully deploy IK on Raspberry Pi devices. That same flexibility also makes it easier for users to move instruments from USB+GPIB to Ethernet connectivity as their laboratory needs change. I think these advantages are somewhat unique to our approach, and have been important for our users. Moving forward, I think it's worth thinking about how to realize these usecases in any efforts which build off of IK.

@hgrecco
Copy link

hgrecco commented May 31, 2017

Hi there! The Lantz author here. It is a project very close to my heart that started a long, long time ago. Python 3 was brand new, the ecosystems was still building up and there was not too much out there. It fulfilled its goal and helped me and others control simple and complex experiments using Python. Like many open source projects, it's development was stopped for non-technical reasons (lack of time, other stuff to do, etc). I am really happy that other alternatives have appeared since then. At the end most of us want a vibrant and active scientific python community.

Regarding PyVISA, I am moving the development of PyVISA and its related project (pyvisa-py and pyvisa-sim) to the pyvisa organization and giving commit access to other people.

@mvddf
Copy link

mvddf commented May 31, 2017 via email

@cjermain
Copy link
Member

@mvddf, I agree with your idea for a SCPI parser. I actually started writing a YAML-based parser for this purpose a few years ago, but never pursued it that far.

The main goal of PyMeasure is to make it really easy to set up repeatable experiments with live-plotting and nice GUIs. That said, the instrument portion of the package has been largely for convenience. As I discussed with @scasagrande at the beginning of this thread, I think it would be a great idea to join InstrumentKit, Lantz and PyMeasure.instruments into a common package.

What do people think about putting in the work to join the packages? Any suggestions for moving forward with a common solution?

@mvddf
Copy link

mvddf commented May 31, 2017 via email

@scasagrande
Copy link
Author

Hey @hgrecco! I wish I chose Pint way back when I first started added units into IK. Back then Pint was pretty new, so I rolled the dice and went with a different project. Switching over is going to be a pain, but its one that I want to do.

@mvddf My work hasn't just been about copy pasting scpi strings. Those are by far the easiest! As I'm sure you know, more modern instruments that actually follow scpi have a lot of overlap between them. This is why I have generic scpi instrument classes. A lot of the work though has gone into all the other instruments: the ones without documentation, older instruments, or ones with complex packet schemes. These are the instruments that our contributors have spent most of their time on.

@cjermain What I would imagine is something like this:

  • pymeasure handles live-plotting and nice GUIs
  • Procedures are pulled out into a second repository/project so it can have its own development cycle. It looks like something other people might want to use independently of the rest of the work in pymeasure, so it might be worth exploring
  • IK handles shuffling data between Python and the instruments, using Pint to back the units :)

@bilderbuchi
Copy link
Member

bilderbuchi commented May 31, 2017

That same flexibility also makes it easier for users to move instruments from USB+GPIB to Ethernet connectivity as their laboratory needs change. I think these advantages are somewhat unique to our approach, and have been important for our users. Moving forward, I think it's worth thinking about how to realize these usecases in any efforts which build off of IK.

@cgranade If I am not misreading what you say, I think python-ivi will be an interesting/related project:

"Python IVI is a Python-based interpretation of the Interchangeable Virtual Instrument standard from the IVI foundation. [...] Python IVI can use Python VXI-11, Python USBTMC, PyVISA, pySerial and linux-gpib to connect to instruments. "

@mvddf what do you think, in your experience, about IVI or python-ivi? It seems like a great idea which could probably help with the abstraction you guys want to achieve? I don't have a deep understanding of that space, admittedly. edit: OK, this does not seem to be the case.

What do people think about putting in the work to join the packages? Any suggestions for moving forward with a common solution?

This sounds great! The division that @scasagrande suggests along the lines of GUI/presentation layer (maybe add data i/o) vs. Procedures vs. Instruments sounds reasonable. In fact, I think even having a unified/flexible instrument definitions library that absorbs instruments from elsewhere (pymeasure, lantz,...) and that other projects can easily connect to would be of great value -- so far nearly every automation project I stumbled over had its own small set of instruments defined, using their own standards, which is not a great situation and leads to lots of duplication.

From my experience in OSS projects and maintaining such a community I think it's important to unify under a common organisation (like @hgrecco tried to do with https://github.com/LabPy/ - maybe that would already be a good banner to unite under). This way it's more probable that a critical mass of maintainers is achieved to keep the ball rolling even if some devs drop out - many of the projects I have encountered seem to have started as someone's thesis, or as the specific solution for a lab/group, but have subsequently withered away when people moved on. Nowadays when I encounter an OSS project, community activity is one of the most important parameters when choosing if I will use the project.

@bilderbuchi
Copy link
Member

also, pinging @alexforencich (python-ivi dev) who migth be interested in this thread/conversation.

@mvddf
Copy link

mvddf commented May 31, 2017 via email

@scasagrande
Copy link
Author

@bilderbuchi python-ivi is something that we've looked at in the past. That project has the same shortcoming that @cgranade mentioned before regarding custom communications. It doesn't fit the needs for a lot of the people that use IK

@mvddf I'm not really sure where you're going with this. I understand that writing classes for instruments is a time consuming process. There are so many different instruments out there that people will always need to be writing more. IK attempts to help with that by providing some tools for developers to greatly cut down on the amount of time they spend on basic commands by providing "property factories". For an example see here https://github.com/Galvant/InstrumentKit/blob/master/instruments/qubitekk/mc1.py#L93 where the backing code implements all the units-, bounds-, type-, etc checking.

@bilderbuchi
Copy link
Member

bilderbuchi commented May 31, 2017

same shortcoming that @cgranade mentioned before regarding custom communications.

ah, ok; at least from the project description it had looked like that would not be the case...

@benfei
Copy link
Contributor

benfei commented Jun 22, 2017

Another nice package is Auspex. @grahamrow

@bilderbuchi
Copy link
Member

@scasagrande @cgranade @cjermain @hgrecco:
Is there something actionable we can take away from this discussion? Can we agree on a course of action to slowly converge these packages?
Here are some bulletpoints roughly summarizing the discussion above. I can turn those into issues in the relevant repositories if you want. Please indicate agreement/disagreement.

Package constellation:

  • We could all benefit from a common package that covers instrument control (i.e. InstrumentKit) @cjermain
  • Don't reinvent the wheel (again)!
  • InstrumentKit: Being flexible with respect to communication layers/connection libraries is very important (e.g. pyVISA, USB, GPIB, Ethernet, VX-11,IVI,...). IntrumentKit covers this and regards this as a core tenet.
  • PyMeasure: The main goal of PyMeasure is to make it really easy to set up repeatable experiments with live-plotting and nice GUIs
  • Lantz: Help to control simple and complex experiments using Python

Discussion:

  • A possibility to parse SCPI commands from docs and provide a command tree would be nice to have. Also to reduce the dependence on SCPI commands (@mvddf)
  • One weakness of python in this space is the time required to build up ui things as compared to labview
  • @mvddf: Most of my time is spent in copy-pasting scpi commands from user guides into code. @scasagrande: A lot of the work though has gone into all the other instruments: the ones without documentation, older instruments, or ones with complex packet schemes.
  • IVI implements a complete command set for the instrument which differs from scpi; sometimes easy, sometimes hard; no suitable/flexible python package (python-ivi not really suitable)
  • IK attempts to greatly cut down on the time spent on basic commands by providing "property factories"

Possible progress from here:

  • Marketing/visibility is important to attract users and contributors. Are you open to doing some marketing?
  • Use LabPy as an umbrella org (@hgrecco?) Is that agreeable to everyone or not desired? I think a revived Labpy could be a good fit
  • Or use Lantz as a common package for instrument control? @cjermain: "I think it would be a great idea to join InstrumentKit, Lantz and PyMeasure.instruments into a common package"
  • PyMeasure could focus on experiment execution and GUI display
  • Procedures are pulled out into a second repository/project
  • IK handles shuffling data between Python and the instruments, using Pint to back the units
  • @bilderbuchi: I think it's important to unify under a common organisation (like LabPy); achieve critical mass of maintainers; keep the ball rolling
  • To start aligning packages, IK will try to move from quantities to pint (see Quantities or Pint instrumentkit/InstrumentKit#92)

@bilderbuchi
Copy link
Member

bilderbuchi commented Jun 28, 2017

To summarize decisions that should be made/discussed:

Did I forget something?

I am not yet familiar with the code of the several packages, but I have in the past been a community manager/issue tracker maintainer/Git guy for another OSS project (https://github.com/openframeworks/openFrameworks). Therefore I can offer my assistance in coordination, bug tracking, git mangling matters if you all want to unite under a common banner and set up a focal point community for laboratory automation/instrumentation with Python. :-)

@cjermain
Copy link
Member

@bilderbuchi, thanks for putting together the summary and action points! Your work is greatly appreciated. Here are my thoughts:

Do we move the packages into one common organisation? If so, LabPy? Are people comfortable with that?

Sure. We can certainly move PyMeasure to LabPy. I made the PyMeasure organization a while ago, which is empty and is certainly an alternative home.

Do we unite into one super-package?

I don't think so. From the Unix philosophy, I think we want individual packages that do one specific task really well. From what I see there are two packages:

  1. Reproducible measurement scripts and GUIs (PyMeasure, etc.)
  2. Instrument control and data acquisition (InstrumentKit, Lantz, PyMeasure.Instruments, etc.)

Who will set up a homepage/blog under that org for marketing?

My timing is less open at this point, but I'm happy to help. We need to decide on a organization to banner under first.

As a first technical step, is it feasible/smart to introduce a compatibility layer to enable pymeasure to use IK drivers?

Good question. I have not had time to investigate this. Does anyone want to take point on leading this effort?

Can we then transfer/translate the existing pymeasure (and maybe Lantz) instrument drivers to IK?

That would be great. Is anyone willing to start this process?

Do we split out Procedures from pymeasure into its own package?

I am not clear on the advantage for doing this. Procedures are the core foundation to PyMeasure. They are more critical than the GUI module, and cover more functionality. I would be more open to splitting out the GUI section, but I don't really see the advantage on that either.

Pymeasure could then gain unit support

PR welcome. This would be a nice addition!

I am interested to hear other peoples thoughts on these questions.

@scasagrande
Copy link
Author

Sorry for the delay. I'm on vacation for Canada Day (150!) this weekend :)

Do we move the packages into one common organisation? If so, LabPy? Are people comfortable with that?

That's not something I'm confortable with until we're much futher along in this process. I think at this point it would just add organizational complexity where it is not needed. We'd have to have entire conversations around authenitcation keys, license usages, developer best practices, etc. Its not something I think we're ready for.

Do we unite into one super-package?

No, its best to keep things in seperate repos. That way each section can have its own development track and release process. This also lets people use just what they need, instead of everything all the time.

Who will set up a homepage/blog under that org for marketing?

I have plans for an IK website soon ™️

IK should probably transition from quantities to pint (instrumentkit/InstrumentKit#92)

After some of the protips I received I'm definetly switching over. My first priority is to finish some work I'm doing with fixing some inconsistencies in the api with single- vs multi-channel instruments to allow easier switching between the two. After that moving to Pint is going to be the biggest body of work moving towards a 1.0 release.

As a first technical step, is it feasible/smart to introduce a compatibility layer to enable pymeasure to use IK drivers?

I imagine the specific language that we use for our APIs aren't identical, so I probably can't just forklift in the drivers without violating some of IK's key principals. It shouldn't be too hard to manually port them over, and I've been thinking about doing so after my above mentioned sets of work.

Can we then transfer/translate the existing pymeasure (and maybe Lantz) instrument drivers to IK?

Addressed above

Do we split out Procedures from pymeasure into its own package?

I only mentioned it because it seems like people might want to use that sort of work independent of instrumentation control. It just seems like its own thing that someone might want to use for their own project.

What functionality/ideas from Lantz should we incorporate/recycle into other packages? Drivers into IK? Feats into Pymeasure? Something else?

I imagine mostly drivers, but I haven't gone through all of it.

@StefanD986
Copy link

A while back I commented on the issue that python-ivi does not (yet) make use of the IVI abstraction layer.

Since reinventing the wheel was mentioned a couple of times in this issue: Actually IVI provides a not perfect, but pretty good base for everything instrument control related. The real issue is (as mentioned) that - so far - there is no (active) project that makes use of the existing IVI driver framework.

For those not too much involved with python-ivi: The problem is that python-ivi only provides an IVI-like interface to the instruments, but it does not make use of any of the instrument driver that the instrument vendors already provide. So to use an instrument a driver has to be written, specifically for python-ivi, which is something only few people want to do. This is even more true for instruments that are not message based (like PXI instruments). Also IVI is more than just instrument drivers. The IVI configuration store is an important part of realizing the idea of "interchangeable instruments" this is not tackled by python-ivi, and it also would not make sense to reinvent this.

I have looked into python-ivi and from my perspective it is not too hard to make it a real IVI wrapper library, that makes use of the existing IVI-C or IVI-COM drivers. I actually have started implementing a wrapper for IVI-C drivers, which in the end would give the ability to use any instrument that provides an IVI-C driver with python-ivi. I hope have some more time to spend on that, and then push it to my fork of python-ivi for others to try and contribute.

So for me goal number 1 is: Provide a python wrapper for all IVI-C libraries.

@StefanD986
Copy link

Just want to add that IVI is of course not a solution for everything. There will always be instruments or specific functionality that will not be supported by IVI drivers (one of the more painful/astonishing examples: Source-Meter-Units have so far no place in the IVI standard.). But IVI has covered that quite good, by giving the option of providing instrument specific drivers, and/or IVI custom drivers, which at least can make use of the IVI configuration store and other nice IVI things.

What I want to say is: If there is the need for end-users to write drivers, then the first option to consider should be to try to build on the IVI infrastructure.

@alexforencich
Copy link

If you want that feature, then use pyivi: https://pypi.python.org/pypi/pyivi/0.0.8 . Note that you will be forced to use a standard VISA backend (NI-VISA) and it will only work on windows. This is NOT a cross-platform solution, and there is no reasonable way to make it cross-platform.

@StefanD986
Copy link

Yes, I know and have tried out pyivi, however the project is abandoned and it does not support many instrument classes. I thought about which one of the two projects - pyivi or python-ivi - provides a better foundation, and I came to the conclusion that it is more worthwhile to write a "real" IVI backend for python-ivi for two reasons:

  • From my view python-ivi looks cleaner (it has tests, the code is structured in a nice and modular way, which makes it easier to work with.)
  • As you have said, using the "real" IVI backend limits the user in most cases to windows systems[*]. Python-ivi gives the user to implement their own drivers if using windows is not an option, without using a completely different library.

[*] I think there are VISA implementations for Linux as well, but the drivers will be the real issue as they are mostly only available for windows

@alexforencich
Copy link

One of the goals of python-ivi is to be cross-platform (and probably most of these python projects). "Real" binary IVI drivers are not cross-platform. And they have to interface directly with a VISA driver. And on linux, the NI VISA software sucks. They dropped support for their own USB GPIB cables on linux as it is apparently too complicated to update the driver to support non-stone-age kernel versions.

@StefanD986
Copy link

Since (alone due to their price) the instruments are mostly used in corporate environments (where the choice of the OS is usually predefined with on anser: M$) the lacking cross-platform support is for me only a secondary problem. Also the added IVI-C backend I'd like to create would only be an additional option, it would not replace the existing drivers or pyvisa backend.

@bilderbuchi
Copy link
Member

Is this pretty detailed/specific discussion maybe better held over at the python-ivi issue tracker?

@bilderbuchi
Copy link
Member

I just found this document containing some thoughts by the Labpy contributors about desired features/structure of instrument drivers: https://github.com/LabPy/labpy-discussion/wiki/Instrument-drivers
Would be interesting to know how much of these are contained in the various libraries out there.

@MatthieuDartiailh
Copy link

Hi, everybody

From experience what you are trying to do is both very interesting and very complicated. I was involved in the common effort under the LabPy org, however thing did not turn out so well, because solving those issues is quite complex and require a lot of discussion from all involved parties. I did most on the code writing in Labpy (lantz_core) and basically gave up because I was getting no feedback from the other members (which is always a bit demotivating).
I will throw another package in the discussion https://github.com/Exopy/i3py which basically builds onto the discussion we had in Labpy (with in particular support for subsystems and channels). For the time being no drivers are implemented but you are welcome to look at the docs (in the docs folder, there are not yet on readthedocs).
I am open to discuss a common effort but I fear it may turn out like Labpy.

Also reading the some of the comments, I felt there was some dislike of PyVISA, as one of its current maintainer, I would like to know why. Some years ago, PyVISA only supported the visa library, but nowadays, PyVISA is a front end for multiple backends. In particular the pyvisa-py backend supports direct communication through Python libraries. It is not bug free but could use some knowledge and effort from some of the people that took part in this discussion (in particular for USBTMC). My point here is that as soon as an instrument support SCPI string is basically use VISA and by improving the PyVISA backend one get for free the high level capabilities from PyVISA such as binary string parsing. I believe that in such cases pyvisa-py could mostly replace the adaptors used by pymeasure.

@bilderbuchi
Copy link
Member

bilderbuchi commented Mar 20, 2018

basically gave up because I was getting no feedback from the other members (which is always a bit demotivating).
I am open to discuss a common effort but I fear it may turn out like Labpy.

Would you say this was because there were just too few developers involved in labpy? If yes, I think this is one of the key take-aways from this discussion. We need to collect/interest a big enough number of developers, so that the bus factor goes up, people can come and go without the team being reduced too much so there is no feedback/discussion, and new users are attracted to the platform. Basically, a critical mass problem. I think LabPy could (have) become that, as it's already an organisation with the right name and idea (just too few developers).

My hope/idea is that, in the end, we unite the currently N>>1 disparate efforts with 1-2 busy/seldomly active developers each (you can see this reflected in many of the packages mentioned here) into one organisation, with ~3-4 related projects, and 1.5*N developers who can support/replace each other as time/busyness commands, and therefore have overall a healthier python lab automation environment. Also reinventing the wheel is avoided, and instrument drivers can be pooled in one big pile.

I fully realise that this is a big and complex thing to achieve :-(, and probably still needs an anchor point with sustained interest and sustained opportunity to contribute (e.g. a university lab researcher or similar).

@MatthieuDartiailh
Copy link

I think you also need a hard core of developers with time enough to discuss in depth the different challenges and solutions. For example one discussion that stopped midway was the standardization of the driver interfaces, because to design such an interface one has to consider multiple instruments from different vendors and come up with a minimal common subset that makes sense. This is quite time consuming but also requires in some cases unintuitive decisions such as imposing to always use channel even if the instrument has a single output and accepting this can take some convincing (I know I was on the wrong team !)

@scasagrande
Copy link
Author

I'm just going to keep plugging away on my chosen path :) however with my recent job switch (I am now a product test engineer at an electronics manufacturing company) I just don't have time to organize or drive any of this. However I am going to be migrating a lot of our stuff to use IK so I'll still be working away at new development.

@MatthieuDartiailh
Copy link

Also you may want also to contact people from https://github.com/QCoDeS/Qcodes

@bilderbuchi
Copy link
Member

bilderbuchi commented Mar 20, 2018

Wow that looks expansive. It's crazy that there are so many different solutions/tools out there!
Your're probably right @MatthieuDartiailh, probably @jenshnielsen and @WilliamHPNielsen have something useful to add here.

@alexforencich
Copy link

So, one thing about the labpy/lantz stuff that I noticed: I wanted to help work on the core code a few times, but the metaclass stuff is a bit obscure and as a result figuring out how objects end up getting put together is not very intuitive, and after poking around for a few days I ended up setting it aside and moving on to other things. So the 'activation barrier' for new developers (and probably folks who would contribute drivers that don't require modifying core code) I think was a bit too high. As a result, I think it is a good idea to stick with a simpler paradigm that's easier to understand, debug, and use. Fancy features such as metaclasses are nice and you can do powerful things with them, but this comes with the downside that they're rather subtle and more difficult to get acquainted with and use effectively.

@WilliamHPNielsen
Copy link

Hi everyone! Thanks for mentioning us. From what I understood skimming through this long discussion, you/we want to unify all the different python-based measurement and instrument control packages into one common framework. Who could argue against the usefulness and awesomeness of that?

Having said that, I have to add that although QCoDeS has a lot of developer power, we are currently also very narrowly focused on the goal of getting tourtopological qubit project on rails. To that end, a large part of QCoDeS is even going to be proprietary (which of course no one outside our organization can see). I would personally like to keep the community part of QCoDeS thriving and useful, and I'd like to be a good citizen and contribute back to e.g. PyVISA that we use heavily. But we are currently not at a stage where we can put effort into uniting QCoDeS with other frameworks.

@pazzarpj
Copy link

Just going out on a limb here as someone who doesn't have as much stake at this layer of development.
Perhaps instead of creating the "ideal" instrumentation package, we should be using the LabPy community as a place to list all the repos of the contributors to the community in the Instrumentation space. This thread alone has shown me so many different takes on the Instrumentation layer where my original search many years ago only showed PyVisa and a very incomplete python-ivi.
This led me to write my own wrapper for PyVisa
eg. https://github.com/pazzarpj/Fixate/blob/master/src/fixate/drivers/funcgen/helper.py
https://github.com/pazzarpj/Fixate/blob/master/src/fixate/drivers/funcgen/keysight_33500b.py

Which i get is horrible compared to some of the solutions here. Which is why I'm saying if I had found a community like LabPy then, which had 4-5 wrappers listed. I could have chosen one an stuck to it and contributed instead of hacking my way through.

Once we have the list of repos, it is an easier for outsiders to see the packages available, use them and perhaps contribute.

By doing this we would perhaps reduce the additional number of these packages floating around with 1-2 users, and see some actual adoption outside the core contributors.

@MatthieuDartiailh. Thanks for your contributions to PyVisa and the community. I have really appreciated PyVisa as without it I would probably be fired from trying to implement it myself and taking way too long.

@bilderbuchi
Copy link
Member

@pazzarpj yeah, collecting the relevant packages in one place might be a useful first step.

Maybe another reduced-scope goal could be to unify the different "instrument-driver" layers floating around into one package, optimally with a featureset that is a union of what is available in the different packages (unit support, validators, testability/unit tests,...). That way at least that part of the measurement&control space is unified (GUI and sequences and automation not considered for now), and the "boring" part of writing a library, i.e. implementing drivers, would hopefully be already done. People could then integrate the package into their own thing, which would bring users and hopefuillyl contributions.
It's probably a tough task, though, to find out if there is a significant intersection/compatibility between these sometimes quite different architectures. One of these days I have to continue tallying/analysing the different packages to get some structured data to base a discussion on.

@bilderbuchi
Copy link
Member

bilderbuchi commented Mar 21, 2018

@alexforencich (you're the only visible org member) would labpy be open to hosting such an effort, and accepting more contributors/activity/discussion?
I feel like a good next step could be to continue these discussions over at https://github.com/LabPy/labpy-discussion/issues, so that we can stop spamming @cjermain's issue tracker. :D

@cjermain
Copy link
Member

@bilderbuchi, no worries. I'm glad to see the enthusiasm and openness for collaboration.

For my part, I am no longer in a research role, so I am not actively working on PyMeasure. I am maintaining the package, and have been excited to see a number of user contributions. I'm happy to help facilitate any consolidation from the PyMeasure side of things.

A few years ago we got to the discussion stage of consolidating, with the help of @hinnefe2. We put up a group, https://github.com/pyinstruments/pyinstruments, and listed a bunch of the packages. Unfortunately, that was not enough momentum to get us to work together on the same packages. I think the most important thing is to get as many dedicated developers onto the same repo. With critical mass, I think you can attract a much bigger audience.

Best of luck!

@hinnefe2
Copy link

Happy to see that people are still working on replacing labview with python :) I am also no longer in a research role, but my labmate @pbnjeff89 has taken over our collection of python instrument drivers (which we called labdrivers).

For what it's worth, I agree with @cjermain that marketing is the key to consolidation. Labs that have already heavily invested in a particular library are not going to change, but if you can a) capture all the people like me three years ago who are googling 'python instrument drivers' and b) let people know it's possible to do this with python then my guess is you could eventually get a decent community going. Having good looking documentation (eg readthedocs) and plenty of simple examples is probably a good start.

@pbnjeff89
Copy link

I didn't see a new discussion in https://github.com/LabPy/labpy-discussion/issues so I thought I'd drop by here. I'm very late to this party, but I'd like to offer whatever help I could give in between writing my thesis/doing lab work.

@bilderbuchi
Copy link
Member

Hi everybody!
I just tried to re-start my efforts in transitioning the results from this discussion over to https://github.com/LabPy/labpy-discussion/issues, which really is a better place to have these discussions. I realised this is a pretty big task for one person, so I invite everybody to assist in opening fresh issues over there (in a structured fashion), containing the key points of some aspect discussed herein. I am grateful for any help!

I only managed to plant a seed for the "less" ambitious goal of unifying the instrument driver layers (and disregarding sequencing/GUI/... for now) at the issue Unify different instrument driver solutions.

I think @hinnefe2's point about marketing being the key point to consolidation is spot on. If we manage to come up with a unified instrument driver library that contains much of the functionality floating around in insular packages, that would be a great step towards some sort of critical mass.

@robmarkcole
Copy link

Couple of other projects of interest:

  1. https://github.com/NSLS-II/bluesky powerful, complex, science crowd
  2. https://github.com/home-assistant/home-assistant I'm a regular contributor to this one. Gives control over a vast array of non-scientific instruments, e.g. webcams, esp type boards etc

I see us using a combination of pymeasure and home-assistant in our lab

@bilderbuchi
Copy link
Member

This issue has more of a discussion/usage question character, so I will move it to our new discussion forum, which is a better fit than the issue tracker.

@pymeasure pymeasure locked and limited conversation to collaborators Feb 16, 2021

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Projects
None yet
Development

No branches or pull requests