Repository navigation
Data Acquisition
Integrating data acquisition with the sequencer.
The process of merging experimental control with data acquisition using oskar is very simple. There are two methods:
- Add an attribute to the HDF5 group for a given SQUID, which can be used to note single-value measurements (e.g., laser power/ lab temperature).
- Record a dataset (e.g., oscilloscope waveforms and camera images).
In both cases, semaphores are used to prevent read/ write conflicts of the single HDF5 data file that each run produces.
Often measurements from pressure gauges or power supplies are useful for verifying if interesting "signals" are real, or just arise from drifting experimental conditions. However, lots of precise repeat measurements of the conditions might be overkill, and a record of a single value, or the mean value, for each SQUID is often sufficient for system monitoring. Such a measurement can be recorded into the attributes (metadata) for a SQUID group using the global variable Rec.
Rec is a global variant that is wiped at the start of every SQUID acquisition period. At the end of the acquisition period any values written to the variant will be appended to the SQUID attributes with the prefix REC:. See the example rec_random.vi, which continuously generates a random number every 5 seconds. Attributes with the same name are overwritten, thus the last value to be written during a given SQUID will be the one that is ultimately stored.
Arbitrarily many measurements can be added to the SQUID group's attributes in this way, so long as each has a unique name.
VAR settings are already recorded to a SQUID's attribute record, however, REC can be used to ensure values were successfully set. For example, a piece of hardware that you control using a VAR will probably also have read functionality. By reading back the same setting controlled by the VAR and then storing it as a REC attribute you have a way to check that the hardware is actually listening to your requests.
(This functionality can be built into a VAR control, but it's slightly awkward. It's logical to read from the hardware shortly after writing to it, however, because the REC variant is wiped at the start of the acquisition period (after the VARS have been set), this is too early to add the attribute to the REC variant. The solution is to read the value from the hardware, then wait for the Sequencer.vi to begin acquisition before writing to the Rec variant. See VAR/PAUSE_RAND.vi as an example of a combined VAR / REC vi. In general, it's simpler to just have an independent REC vi that periodically queries the hardware; just be careful to avoid communication conflicts.)
Most scientific data will take some form of regularly sized array, such as oscilloscope waveforms, camera images, etc.
For each SQUID there might be many repeat measurements from the same device. In general, to record such data a program is needed to continuously acquire data, then by reading the following global variables:
- Save (Boolean. Is save selected in Sequencer.vi?)
- Acquire (Boolean. Is the Sequencer.vi state currently `acquiring data' (not, e.g., setting VARs)?
- H5 (array of paths to hdf5 file(s)
- SQUID (integer value as a string. The iterating number that uniquely identifies the current line of the running sequence.)
it's relatively easy to coordinate the saving of data in accordance with the state of the sequencer.
For instance, in the data acquisition program place an appropriate h5_[name].vi subVI inside of a case structure controlled by Save AND Running. Pass the global variables H5 and SQUID to the subVI to indicate which file and group to write the data to.
See h5_scope.vi. This also packages the relevant metadata (channel gain, time-base, etc.) into variant attributes and passes both this and the actual data to H5_1D_SGL.vi, which then writes the data to the H5 file in a dataset called OSC_# (# is replaced by the channel number), inside of the SQUID subgroup. Something equivalent to ``h5_scope.vi" could be made for each data-acquisition device, written to appropriately package the meta-data attributes.
LabVIEW semaphores are used to prevent more than one vi writing to the hdf5 file at the same time. The fake_scope.vi illustrates how this is all achieved. fake_camera.vi is an even simpler example that just uses the generic h5_2D_stacking.vi to save directly.
The format for saving data depends on its shape (e.g., 1D array, 2D image) and data type (SGL, DBL, LONG, etc.). Examples for 1D_SGl, 1D_DBL, 2D_I32, can be found in the ./tools/hdf5 folder. These vi's expect repeat measurements to be made within a SQUID. For instance, when 1D_SGL.vi is asked to write data to an existing dataset, it will be appended as a new row.
The example h5_stats.vi demonstrates how cluster data can be written to a dataset. This might be used when, rather than a time series, a range of values (possibly of differing dtypes) are recorded at once. This DataFrame structure is useful if analysis is performed in LabVIEW to determine, e.g., trigger events (time, amplitude, width), or statistical features (mean, max, min, range, FWHM).
Because VAR conditions are stored in each SQUID's attributes, and as data relating to a SQUID is stored within its subgroup, it's trivial to map data to the relevant VAR conditions. This assumes that data is recorded correctly, however, the individual data acquisition vi's are responsible for ensuring this (not the sequencer) and it's possible to get things wrong.
Warning - oskar is designed for experiments where many (>10) repeat measurements are recorded at a rate of 10 Hz to 0.3 Hz, then an experimental parameter is varied and more measurements are similarly made. For lower acquisition rates (e.g., long camera exposure times) you must ensure that data taken near the boundary of two SQUIDS is mapped to the correct one, or safely ignored if it's ambiguous.
This can be done quite easily. For instance, by checking the SQUID global variable has the same value both at the start of an exposure and at the end. Alternatively, the global variable EOS (end-of-squid) can be used to synchronise the sequencer to a data acquisition or hardware trigger. See `Tools/skip_squid.vi'. By using EOS with the acquisition time set to be unattainably long, a data acquisition device can effectively control how long the sequencer acquires data for (e.g., until 10 waveforms have been recorded).
A run will produce a file called [RID]_raw.h5, which is saved to the folder(s) `[Base(s)]/[YYYY]/[MM]/[DD]/[RID]/'.
Opening this file using HDFView will reveal a structure akin to:
The top level group is the RUN and has attributes, RID, SEQ FILE, AUTHOR, and DESCRIPTION. Each group within the RUN represents a SQUID value. This has attributes DATETIME, START, END, ACQUIRE, and all appropriate VAR settings in the form, VAR:NAME = VALUE. REC data is similarly recorded. Each SQUID group contains all of the datasets that are associated with it (IMG, OSC_0, OSC_1, and STAT_0 in the example above), which could represent repeat measurements of 1D/ 2D data from oscilloscope channels or a camera. Datasets can have their own attributes, which can be used to record, e.g., exposure time, amplifier gain, etc. HDFView has basic plotting and imaging features built in.
Note: the SQUID group names are strings. This is important if you wish to access a given group using, e.g., h5py.
The structured format of HDF5 makes batch analysis very simple. The ability to attach metadata makes it easy to associate measurements with experimental conditions and instrument settings. As HDF5 stores data in binary, it's an efficient use of HDD space, and it can be read quickly for analysis too. It's also possible to compress data within an hdf5 file, although that has not been implemented here. See the h5labview docs for more info.