best way to pass vector data to Bluesky via yaqc-bluesky?

[nfortune]

I'm looking for advice on how to best acquire simultaneous vector data from a yaq daemon then pass it to Bluesky via yaqc-bluesky (if that is possible). I want to acquire both in and out of phase components of a lock in signal (X and Y, alternatively R and theta) simultaneously, not sequentially, then pass it to detector(s) in Bluesky.

example: For a Stanford Research Systems lock-in (such as the SR830 or SR86x), the command SNAP?1,2,3,4,9 would return the simultaneously measured values of X [V], Y [V], R [V], theta [deg], and freq [Hz], as opposed to measurements taken one at a time in sequence. SNAP?1,2 would return just X and Y. Here X and Y are the in phase and out of phase components of a sinusoidal time varying voltage (relative to a sinusoidal reference at the same frequency). R is the vector magnitude and theta is phase shift between signal and reference.

I see that the trait is-sensor indicates arrays can be passed instead of single values, and that the trait has-mapping might allow the differing units [volts, degrees, hertz] to be included as well . But BlueSky seems to want devices to consist of 1D detectors with individual values. So does that mean the X and Y measurements need to be set up as individual channels ? Reading individual channels separately would break the simultaneous measurement requirement. Alternatively, BlueSky allows "on the fly" measurements that return arrays (I think) but that isn't currently supported by yaqc-bluesky.

I understand that the yaqc-bluesky client only implements a subset of the yaq functionality, so it might be this type of vector measurement isn't possible. That too would be valuable knowledge (!) but I'm hoping there is a clear, preferable way to proceed that could be suggested.

[ksunden]

For this use case, it looks like channel is the idea that you want. yaq sensors do support multiple channels read simultaneously, just structured as a key-value pair instead of an array

is-sensor is multi channel by default. Each channel can have its own units. Even with has-mapping, each channel can have only one unit.

has-mapping is more about adding independent axes to array channels, e.g. adding wavelengths to a spectrometer. the mapping values are not usually measured values, but either statically known or calculated off of a small number of inputs.
If you were measuring e.g. 1 second of data at 1 kHz for each measure command, you could have 1000 points for each X, Y, R, theta, freq, then use a mapping which was just a linspace of 0 to 1 to say the time for each index in each of the other arrays.

Mappings are pretty flexible, but most sensors use them in a fairly simple way where they have one (usually) channel that is a 1- or 2-d array (can be more, just most _aren't) and one 1-d array for each axis of the main channel.

The story is a little more complicated if you wish to set multiple values simultaneously:

• All has-position is expected to have a single float input (with caveat for discrete positions, that they still map to a float)
• You can run two different yaq daemons for each setpoint, but that has some hurdles for some devices/use cases:
  • depending on how "simultaneous" you need it to be, they will be different calls that go over the network, and so there will be some period of time between values being set
  • if you cannot establish multiple connections to the underlying hardware, we do not have a good way of handling that usecase in yaq (we've had ideas, but never actually been unworkable, so never got completed)
    • The problem is that we currently only have infrastructure to run one kind of daemon in a given process. There are mechanisms to share e.g. serial ports between these devices, but only if they are the same code.
    • This is not a fundamental limitation of yaq, just one we have not done the configuration work to be able to specify more complicated single processes
    • For the kind of devices you are talking about, yaqd-scpi may be flexible enough to do it (at least if you don't mix read and write to the same device)
    • If this is needed, we can revisit some of our ideas/provide proofs of concept for how this might work
    • we have colloquially called this the "heterogeneous catastrophe", though we are pretty sure solutions are possible, just haven't proven actually necessary with the kinds of experiments we have been doing.

Bluesky should be able to handle just about any numeric output you give it (any nd-array)

If you have large arrays, such as cameras, it should work, but there are technical limitations which make it not the best solution perhaps (specifically sending arrays over TCP takes a fair bit of time, so some (non-yaq) cameras will write their data locally and only provide bluesky with the info it needs to find that later) From the sounds of it, this isn't your use case, but we would be willing to work towards it if we had a use case in mind which required higher throughput than sending arrays over TCP while acquiring.

Here is the complete implementation of the "System Monitor" daemon, which gives cpu/ram/diskio/uptime of the computer running the daemon, which shows something with multiple channels that are provided in a single measure command:

https://github.com/yaq-project/yaqd-system-monitor/blob/de0900d934f63360c9e4ddb5d956403b521fb343/yaqd_system_monitor/_system_monitor.py#L24

If the iterative reading of channels in the default yaqd-scpi is too slow for your experiment's definition of "simultaneous", you may have to write a daemon that gets all of the values in one query string and then repacks them as a dictionary. The core functionality would be almost the same as SCPISensor but with some different parameterizations and a different _measure method. Happy to help get you up and running on that if you need it, we may even wish to add it to yaqd-scpi as an alternative parameterization. (possibly even able to flexibly do both)

[nfortune]

Yes, I would welcome some help getting up and running!

Attached I've modeled a possible description for a lock-in-sensor (or vector sensor) on the SCPI-sensor TOML file
lock-in-sensor. In that case, an SCPI-sensor "OUTP?1" query would an string with the value for the X channel and "OUTP?2" would return the same for the Y channel.

What I really want to do, however, is create channels X and Y (or R,θ) from simultaneous reading, since sequential readings of a complex value (X + iY) introduce phase jitter not present in the actual data. To do so what I think I need to do is replace the QUERY that the SCPI daemon invokes with sequential WRITE + READ commands and then return a dictionary with the X and Y channel values already filled in:

import csv

write "SNAP?1,2"
read response_string

fieldnames = ['X', 'Y']
voltage_dict = csv.DictReader(response_string, fieldnames)
lock-in-daemon.txt

[untzag]

Great insights @nfortune. There's a lot of details here, so let's first take a look at what this looks like on the client (Bluesky) side of things. We can simulate your lockin with a fake-triggered-sensor using the following config.

[fakelockin]
port = 39000

[fakelockin.channels.x]
kind = "random-walk"
min = -1
max = 1

[fakelockin.channels.y]
kind = "random-walk"
min = -1
max = 1

This will generate two channels "x" and "y", and of course Bluesky doesn't know the difference between this fake and a real lock-in amplifier.

Once fakelockin is running, we can use yaqc-bluesky to make a Device instance that is compatible with Bluesky. Remember, the crucial methods for Bluesky to interact with a sensor are describe and read. We can call these methods ourselves.

>>> import yaqc_bluesky
>>> lockin = yaqc_bluesky.Device(39000)
>>> lockin.trigger()  # trigger starts a measurement
>>> lockin.read()
OrderedDict([('fakelockin_x', {'value': -0.10951101653121408, 'timestamp': 1691441926.4567535}), 
             ('fakelockin_y', {'value': 0.009573717711272384, 'timestamp': 1691441926.4567535})])

Now we can provide Bluesky with our lockin instance to do an experiment. Let's try the built in plan count, which just takes num data points.

>>> import bluesky
>>> from bluesky import plans as bsp
>>> from bluesky.callbacks.best_effort import BestEffortCallback
>>> RE = bluesky.RunEngine()
>>> RE.subscribe(BestEffortCallback())
>>> RE(bsp.count([lockin], num=5)


Transient Scan ID: 2     Time: 2023-08-07 16:06:00
Persistent Unique Scan ID: '1ff98c08-9f57-47f2-a225-228dd9b60529'
New stream: 'primary'
+-----------+------------+--------------+--------------+
|   seq_num |       time | fakelockin_x | fakelockin_y |
+-----------+------------+--------------+--------------+
|         1 | 16:06:01.1 |       -0.209 |        0.109 |
|         2 | 16:06:01.3 |       -0.168 |        0.074 |
|         3 | 16:06:01.4 |       -0.187 |        0.130 |
|         4 | 16:06:01.5 |       -0.190 |        0.053 |
|         5 | 16:06:01.7 |       -0.227 |        0.104 |
+-----------+------------+--------------+--------------+
generator count ['1ff98c08'] (scan num: 2)

That output table you see is there because of BestEffortCallback. You can subscribe other useful callbacks to enable other useful features, for example Serializer from suitcase.csv will allow you to dump your data to a CSV file easily.

We ran the count plan above, but there are lots of built in plans to Bluesky.

https://blueskyproject.io/bluesky/plans.html

Beyond that, Bluesky aims to make it easy for you to write your own flexible plans. There are "plan stubs" that you can compose together to design more sophisticated unique experiments. I won't go any further chatting about the client side, you can probably get a good start with the Bluesky documentation from here.

[untzag]

Now I'm going think a little bit about your actual hardware interface.

First off, a bit of context about the yaqd-scpi package. I wrote that package as a generic interface for setting and reading parameters over SCPI. It can work well for simple applications (and we do use it), but for more sophisticated hardware interaction it's probably best to just write a separate daemon. We've done this before, see for example the daemon for spectra physics millennia. That laser communicates over SCPI but the features are so complex that it wasn't possible to fit it into the "generic" mold provided by yaqd-scpi.

What I really want to do, however, is create channels X and Y (or R,θ) from simultaneous reading, since sequential readings of a complex value (X + iY) introduce phase jitter not present in the actual data. To do so what I think I need to do is replace the QUERY that the SCPI daemon invokes with sequential WRITE + READ commands and then return a dictionary with the X and Y channel values already filled in.

Exactly correct. What you want is pretty-much the same as yaqd-scpi-sensor...

https://github.com/yaq-project/yaqd-scpi/blob/main/yaqd_scpi/_scpi_sensor.py#L12

... but just change the _measure method. You suggested the SNAP command, which makes sense looking at the documentation. Using SNAP, measure might look like this (WARNING: untested code).

    async def _measure(self):
        out = {}
        self._instrument.write("SNAP?1,2,3,4")
        x, y, r, t = self._instrument.read()
        out["x"] = float(x)
        out["y"] = float(y)
        out["r"] = float(r)
        out["t"] = float(t)
        return out

Depending on your preferences, you could add other unique configuration or state to your stanford lock-in daemon.