Design overview

Introduction

The control software for the Spotlight setup is challenging to design. Among the sources of this difficulty are:

1. Too many hardware devices to interface with: the Euresys frame grabber, the JAI camera, the PCO camera, the Zaber translation stages, the CCS light control unit (illumination and optogenetics), the Thorlabs light control unit (blue excitation), and the Arduino. Each of these comes with its on particularities, and sometimes incompatibilities with the others.
2. Timing has to be precise: the behavior camera, the muscle camera, and the lights must be synchronized with as little latency as possible.
3. High rate of data coming in: We are recording ~1000x2000px behavior images at 350 FPS, and we want as little data loss as possible. This requires some non-trivial optimizations (and this is why the code is mostly written in C++).

On this Wiki page, I will explain the overall design decisions, particularly the use of parallel threads, shared memory, and hardware interfaces. This will hopefully make it less confusing to read the detailed API documentation.

In the Parallelism through multithreading section, I will motivate the use of multiple threads and enumerates the 10 types of threads used in the program. In the Inter-process and inter-thread communication section, I will explain how threads share access to the same information in a synchronized and consistent way. Then, in the Hardware interface section, I will explain how the program controls and accesses data from peripheral devices (i.e. cameras, Arduino, motion controller). In the Detailed description of shared data section, I will describe the detailed specifications C++ classes and structs that are shared among threads. With these in mind, in the Detailed description of worker threads, I will similarly describe the threads that implement the logics of the program. Finally, in the Control programs and Utility tools sections, I will describe the programs that the user should use to calibrate the experimental system, collect data, and post-process the data.

Parallelism through multithreading

To control the Spotlight setup, we run a number of threads in parallel. There are two main advantages for this:

First, parallelism serves as a means to encapsulate independent logics. As discussed, we have a lot of hardware and software components to handle. Implementing each in a separate thread offers a natural way of segregating them and avoids the entanglement of their states. For example, we have a thread dedicated to motion control. In this thread, we have to estimate the position of the fly in the most recently acquired behavior image, compute where the motion stages should move to in order to keep the fly centered in the field of view, and send such commands to the motion stage controller. If this logic were to be implemented in the same thread as image acquisition, we would have to to carefully synchronize image capturing with position estimation and stage movements, entangled motion control logic with image processing logic, and thus making the code overly complicated and hard to maintain.

Instead, by implementing the tracking logic in a separate thread, we can simply run an infinite loop where the thread fetches the latest image—doesn't matter how it is acquired; that's the job of another thread—and take it from there. When the desired target positions are computed, our thread can then dump the command to yet another thread—one that actually communicates with the motion controller—without having to worry about the low-level API calls to control the hardware. Of course, our thread still needs to receive the latest image and send the target stage locations, but these can be put in data holders that are much better defined (i.e. just the data, no program state, etc.; see Inter-process and inter-thread communication).

Second, running parallelizable tasks in separate threads massively enhances performance. In our previous example, aside from the two exchanges of the lates behavior frame (which needs to be synchronized; see section below), the calculation of where the fly is and the acquisition of the image can happen in parallel. By implementing them in separate threads, we allow the operating system to schedule them to run on different CPU cores, waiting for one another only when shared data have to be accessed in a consistent manner. This massively boosts the performance of the code. Additionally, for such works as compressing acquired frames and saving them to the hard drive, we can simply start a pool of multiple threads, so that the program would not lag even if data comes in faster than what a single image saving function can handle.

In the Spotlight control programs, we have the following threads:

1. Main thread
2. GUI
3. Behavior image acquiring thread
4. Behavior image saving threads (multiple)
5. Muscle image acquiring thread
6. Muscle image saving thread (multiple)
7. Motion control IO thread
8. Tracking control thread
9. Motion stage position logger thread
10. Arduino communication thread

Additionally, we have a dedicated process (slightly different from a thread) that acts as a server for the PCO camera used for muscle imaging. This will be described in Hardware interface.

Each of these threads will be explained in Detailed description of worker threads. However, before we proceed, we have to first describe how threads exchange data, what data are exchanged, and how the hardware components are controlled. These will be covered in the next three sections.

Inter-process and inter-thread communication

Shared memory

When we talk about memory in a program, we're usually referring to something abstract—variables, data structures, arrays. But under the hood, all of this data must live somewhere physical. The physical media of memory include RAM, hard drives, or external devices mapped as "files"—think a printer that "pretends" to be a regular file, and whatever you write to that file gets printed. Things in your program—variables, constants, etc.—reside at particular memory addresses. If you are familiar with programming in C/C++, a pointer is basically an integer address of the thing that it points to.

However, each program (or process) doesn't see the actual physical memory inside your computer. Instead, it sees its own private "illusion" of memory, called virtual memory. Addresses in the virtual memory are mapped to physical memory addresses by the operating system in a process called memory paging. This happens behind the scenes in while you run your program. There are several reasons for layer of indirection: for one, it's easier to manage your allocated memory if you can assume don't have to take into account what addresses are occupied by other programs. The OS moves memory around as needed without the program needing to know. Isolation and safety are another factor: virtual memories prevent a program from accidentally (or maliciously) access or corrupt the memory of another program.

More critically for our program, memory paging supports for shared memory. The OS can choose to map the same physical memory into the virtual address spaces of multiple programs. This is the foundation for efficient communication between processes.

Recall that our program consists of 10 types of worker threads and an additional process (the PCO camera server). It is now necessary to point out the differences between a thread and a process. A process is like a completely separate program running on your computer—it has its own memory and resources. A thread, on the other hand, is like a worker inside that program, doing a specific task, and all threads in the same process share the same memory (this page offers a more detailed explanation). The reason why we need a PCO server implemented as a separate process is a technical one and will be discussed later. Shared memories are handled differently between threads and between processes:

To share memory between threads, no special mechanism is needed. Threads naturally share the same address space. This means that if one thread creates a variable, another thread can directly read or modify it. In practice, we often use shared pointers (std::shared_ptr) in C++ to manage shared data safely. A shared pointer keeps track of how many threads (or parts of the program) are using a piece of data and automatically de-allocates it when no one needs it anymore.

Sharing memory between processes is a bit more complicated, since each process has its own private (i.e. virtual) memory space. To share memory across processes, we need to explicitly allocate a region of physical memory that both processes can map into their own virtual address spaces. In Linux, this is typically done using three system calls:

• shm_open(...) (shared memory open): This open a section of memory that can be opened by other processes too. The section of memory behaves like files—they can be opened in read-only, write-only, or read-and-write modes. They can be found as "files" under /dev/shm on the filesystem.
• ftruncate(...) (file truncate): Once the shared "files" have been opened, ftruncate defines the size of memory buffer. In other words, shm_open returns a file descriptor that defines the start of the memory address; ftruncate defines how large the "file" is.
• mmap(...) (memory map): Once the file descriptor is created and its size set, mmap maps the memory object into the process's own virtual memory space so it can be read and written like regular memory. In other words, mmap creates a pointer to the shared resources that you can treat as any other pointer. You can dereference it to read from or write to it.

In Hardware interface, I will explain how we use these functions to stream images from the PCO camera server to the main recording program.

It's important to note that the aforementioned methods only allow multiple threads and processes to access shared resources. To ensure that the shared data are correct and consistent, we still need additional mechanisms to synchronize the accesses. The next sub-section will discuss data consistency and synchronization.

Data consistency

While our threads mostly work independently of each other, sometimes they have to share data with one another. The exchange of information between parallel threads and processes is called inter-thread communication and, more generally, inter-process communication (IPC). The main problem in IPC is to ensure the synchrony and consistency of shared data. More specifically, we want to avoid the following two situations:

• Race conditions: These happen when two threads access and modify shared data at the same time without proper timing control. In real-time systems, this can cause confusion or loss of information.

  • Example: Imagine a shared grocery list on a whiteboard at home. One person is updating the list by adding "eggs" and "milk." Meanwhile, another person (the consumer) is heading to the store and copies the list to their phone. Without deliberate timing control—say the shopper reads the list while it's mid-update, they might only see "eggs" and miss “milk,” or even get corrupted data like "mi…" before heading out.

• Deadlocks: This happens when multiple threads are each waiting for the other to release a resource, so no one can continue.

  • Example: Imagine two cars arrive at the same intersection and their planed paths intersect. Each car tries to proceed without yielding, so the cars just block one another and we have a traffic jam. A naive solution is to implement a priority rule (e.g. the car on the right always has priority). However, this also doesn't work: If we have four cars coming in all directions, we are still deadlocked. In real life, drivers will eventually figure the situation out because humans can negotiate, make eye contact, or break the rules to get things moving. However, in computing systems, threads follow strict rules and they will wait forever unless we use more sophisticated mechanisms implemented at the operating system level.

To avoid these issues, synchronization primitives are used. These are basic building blocks that help coordinate when and how threads can access shared resources, ensuring that only one thread acts on critical data at a time, or that threads wait for the right moment to proceed. In our program, we use two of the most common synchronization primitives: the mutex (short for mutual exclusion) and the condition variable.

Mutexes are like locks on a door. When a thread wants to access shared data, it first "locks" the mutex, blocking other threads from entering the critical section until it's done and "unlocks" it. This ensures that only one thread can modify or read the shared data at a time, avoiding race conditions.

Condition variables are used to make threads wait for a certain condition to become true before continuing. Think of it like waiting for a bell to ring—one thread might be waiting for a buffer to have data, and another thread signals (rings the bell) when it has finished producing that data. This is much more efficient that the alternative of busy waiting—the receiving threads constantly checking if the condition is met in an infinite loop. Therefore, condition variables help coordinate timing between threads and prevents both unnecessary work and deadlocks.

Hardware interface

Motion control

Before controlling the motion stages programmatically, we need to install the Zaber Launcher application (see its download page here or our wiki page here). This is a convenient GUI program that allows us to set up serial connection with the hardware and modify parameters such as the maximum allowed speed. We can also do such things as manually controlling stage positions and homing the stages in Zaber Launcher.

Once the hardware is set up using Zaber Launcher, we interface with the Zaber X-MCC controller, which controls two linear translation stages, using the Zaber Motion Library. We implement a simple wrapper around the Zaber Motion Library API in recorder/src/peripherals/motionControl.cpp and motionControl.hpp. This wrapper implements methods such as moveAbsolute (move to absolute position), moveRelative (move relative to the current position), home (go to home position, which also resets its position sensing), getPosition (read out the current position), checkIfIdle (see if motion stages are stationary), and waitUntilIdle (blocks until motion stages reach their target positions).

Arduino

The Arduino circuit is a key component of the Spotlight setup. It generates trigger signals (square waves) that control the cameras (behavior camera and muscle camera) and the lights (IR illumination, blue excitation, and optogenetics). The computer on which the control software is run communicates with the Arduino to modulate the trigger signals (e.g. adjust the frequency/frame rate).

Scientific cameras often support hardware triggering, which allows precise synchronization between image acquisition and external events. Instead of relying on software timing, which can be inconsistent due to operating system delays, hardware triggers use physical electrical signals—usually 3.3V or 5V transistor-transistor logic (TTL) pulses—to tell the camera exactly when to capture a frame. Additionally, some cameras also allow us to control the exposure time with the width of the pulse so that whenever there is a rising edge, the exposure starts, and whenever there is a falling edge, the exposure stops (or the opposite, depending on the configuration). Similarly, lights can be triggered in strobing mode, where they are only on when the TTL signal is high (or low, depending on the configuration). This allows light to be turned on briefly during the camera exposure. This reduces photobleaching and heat buildup.

In our setup, the Arduino sends square wave pulses to the cameras. Each rising edge acts as a "take a picture now and switch light on" signal, and the muscle camera is triggered once every $k$ times the behavior camera is triggered. This ensures that both the behavior and muscle cameras are perfectly synchronized, even at high frame rates. Altogether, the Arduino controls six output lines:

1. Infrared illumination LED, through the CCS PD3-3024-3-EI controller
2. Optogenetics channel 1 (an additional LED), through the CCS PD3-3024-3-EI controller
3. Optogenetics channel 2 (an additional LED), through the CCS PD3-3024-3-EI controller
4. The JAI (behavior) camera, through the Euresys Coaxlink Quad G3 frame grabber
5. The PCO (muscle) camera
6. The Thorlabs (blue excitation) LED, through the Thorlabs LEDD1B controller

Note

The lights for optogenetics are not strobed. However, by supplying a constant high/low voltage to the trigger line of the light when it is supposed to be on/off, we can control these lights using the strobing functions.

A C++ program, TriggerController, is loaded on an Arduino Nano ESP32 microcontroller. This program runs in an infinite loop. In each iteration, the program reads out the current time from a high-precision hardware clock and determines whether any action should be taken on any of the lines. It also checks if the computer has demanded a change in the triggering pattern (e.g. change of exposure time or frame rate). To specify the way in which the control program on the workstation sends commands to the program running on the Arduino, we need a protocol (i.e. a syntax of possible commands and the corresponding operations). We define the following as the allowed messages (in plain ASCII text) that the workstation should may send to the Arduino:

1. >SET_BEHAVIOR_FPS <fps>\n, where <fps> is an integer.
2. >SET_SYNC_RATIO <syncRatio>\n, where syncRatio is an integer and the muscle camera is triggered once every syncRatio times the behavior is triggered. In combination of the behavior frame rate, this variable thereby defines the frame rate of the muscle camera.
3. >SET_BEHAVIOR_EXPOSURE_TIME <exposureTimeUs>\n, where exposureTimeUs is an integer and is the desired exposure time of the behavior camera in microseconds.
4. >SET_MUSCLE_EXPOSURE_TIME <exposureTimeUs>\n, where exposureTimeUs is an integer and is the desired exposure time of the muscle camera in microseconds.
5. >START_RECORDING <protocolString>\n, where <protocolString> is a string defining the schedule by which the optogenetics lights should be turned on and off and the time at which recording should stop. This will be described in two paragraphs.
6. >STOP_RECORDING\n. This stops the recording (i.e. all output lines are at a constant low).

Note that there are two design decisions here. First, all messages start with the character >. Any message received by the Arduino that does not start with > will be read back to the computer. This, combined with logging, can be useful for debugging. Second, all messages are terminated by \n (the new line character). The Arduino program only starts to parse the message when \n is encountered.

In some experiments, one might want to precisely specify a schedule by which optogenetic lights should be turned on and off, or when the recording should stop. We call this schedule an experiment protocol and specify it in the <protocolString> (see point 5 above). The syntax of this string is defined in The experiment protocol string.

In the C++ program on the computer side, we define an ArduinoCommunication interface (implemented in recorder/src/peripherals/arduinoCommunication.cpp and arduinoCommunication.hpp). This interface, implemented as a class, provides methods such as setBehaviorRecordingFPS, setSyncRatio, setBehaviorExposureTime, setMuscleExposureTime, startRecording, and stopRecording. Upon calling these methods, the appropriate commands will be sent to the Arduino microcontroller. This way, that the caller does not have to deal with formatting the commands above or low-level serial communication via USB. Under the hood, the ArduinoCommunication object manages the Arduino communication thread, which sends and receives the messages defined above in the background.

Euresys frame grabber and JAI camera

The JAI SP-5000M-CXP4 camera, which we use for behavior recording, uses the CoaXPress interface. This interface allows much faster data transfer than the typical USB interface. However, unlike USB cameras, a frame grabber is required. We use the Euresys Coaxlink Quad G3 frame grabber, which is installed inside the computer as a Peripheral Component Interconnect (PCI) card (similar to a graphics card). The camera streams data into the frame grabber, and the frame grabber streams data into the computer memory through PCI interface. We interact with the camera through the Euresys frame grabber.

The configuration of the camera is documented in eGrabber and JAI camera configuration, example mode 1. Once we configure the hardware through the eGrabber Studio GUI program, we will interact with it through a BehaviorCamera class, implemented in recorder/src/peripherals/behaviorCamera.cpp and behaviorCamera.hpp. When an object of the BehaviorCamera class is created, a sequence of configuration steps are preformed under the hood. Then, interaction with the camera can be done through methods such as start (which starts acquisition), stop (which stops acquisition), and waitForOneFrame (which blocks until a new frame is available and returns it). The low-level interaction with the Euresys API is encapsulated and agnostic to the caller.

PCO camera

Unlike the JAI camera, the "pco.panda 4.2" camera that we use for muscle imaging communicates with the computer via USB. Therefore, no frame grabber is needed.

We interface with the PCO camera through the "pco.cpp" API, though APIs at other levels such as "pco.sdk" are also provided (see Software dependencies and configurations, PCO camera software). Due to the design of the pco.cpp API, certain C++ source code, particularly those under pco.cpp/pco.camera, have to be included when we compile our Spotlight control program. This proves to be problematic due to namespace issues (common words such as BYTE and Camera are redefined in these pco.camera code). As a result, to my best attempt, any program that includes the PCO code is incompatible with both Qt (the GUI engine of our recording program) and the Euresys software. Therefore, I implemented a separate programpco-camera-server—that fetches frames from the camera and streams them to shared memory in a perpetual loop. It does absolutely nothing else and therefore is fully decoupled from the main program. Therefore, it can be compiled separately (i.e. without Qt and Euresys libraries). The main program can simply read frames through shared memory, which have been supplied by pco-camera-server. For more technical information on how shared memories work, see the Inter-process and inter-thread communication section.

In the main program, I implemented a MuscleCamera class (sources: recorder/src/peripherals/muscleCamera.cpp and muscleCamera.hpp). Similar to BehaviorCamera, this class offers a key frame waitForOneFrame, which blocks until a new frame is available, and then returns it. Under the hood is the synchronized read operation from shared memory. This way, the caller can treat MuscleCamera just like BehaviorCamera and does not need to concern themselves with the additional layer of indirection that is the PCO camera server. See PCO camera server for how the server program works.

However, there are two minor differences between MuscleCamera and BehaviorCamera. First, because the PCO camera uses a rolling shutter (this is why it has very low noise and why we use it for muscle imaging), controlling the exposure time is not as straightforward as the JAI camera, which uses a global shutter. Therefore, whereas the JAI camera opens its shutter whenever the trigger signal is high (so that we can control the exposure time by modulating the duration of each pulse sent from the Arduino), the PCO camera needs to have its exposure time separately set via the USB interface. For this, the MuscleCamera class has an additional setExposureTime method. Second, we do not have to explicitly control the start and stop of acquisition, and therefore no start nor stop method is provided. Instead, upon creation of the MuscleCamera object, a new pco-camera-server process is spawn. Upon the destruction of the object, the pco-camera-server process is terminated.

Important

Although the exposure time is directly communicated to the PCO camera, we still need to inform the Arduino of the muscle exposure time because the Arduino still controls the triggering of the blue excitation light, which has to be synchronized with the camera.

Detailed description of shared data

programState

This is a ProgramState data structure (struct) that contains:

• toQuit (atomic boolean): Whether the user has requested to quit the program, whether by clicking the "X" button on the GUI window or by interrupting the process with the SIGINT signal (e.g. by pressing Control-C in the terminal).
• isRecording (atomic boolean): Whether a recording is in session (i.e. images are being saved to disk rather than simply streamed to the GUI program).

This struct is defined in TODO.

behaviorRecordingState

This is a BehaviorRecordingState struct that contains:

• behaviorCamera (BehaviorCamera interface as previously defined): This is the single entry point to access the behavior camera.
• behaviorImageQueue (queue): A queue of frames that have been acquired by the behavior image acquisition thread, pending to be processed and saved by the behavior image saving threads.
• behaviorImageQueueMutex (mutex, a previously discussed synchronization primitive): A "lock" that ensures only one thread can access the behavior image queue at a time.
• behaviorImageQueueCondVar (condition variable, a previously discussed synchronization primitive): An object that the behavior image acquisition thread can use to "wake up" one behavior image saving thread to do the work.
• latestBehaviorFrameHolder (a custom LatestFrame class): A data holder for the most recently acquired behavior image. Other threads, namely the tracking control thread and the GUI thread, can take the latest image from this object for their respective tasks (i.e. computing where the motion stages should move to and streaming the current image in the GUI). The LatestFrame class, implemented in TODO, offers a getLatestFrameData method and a setLatestFrameData method. Internally, LatestFrame maintains its own mutex to ensure that no thread can read from it when the acquisition thread is writing to it.

This struct is implemented in TODO.

Note

Pseudo-BGR JPEG saving trick: In fact, the members of behaviorImageQueue are instances of the GroupOfThreeFrames struct. As the name suggests, the behavior images are packaged in groups of three. This is because we save every three adjacent frames as the blue, green, and red channels of a single JPEG image. The JPEG format is designed to efficiently compress three images that are structurally very similar (the three color channels of an image usually contain the same objects). We can take advantage of this design to save behavior data more compactly, as the fly doesn't move much between three adjacent frames. This trick strikes a balance between time efficiency (i.e. the need to compress image in real time) and space efficiency (fewer bytes per frame).

muscleRecordingState

The MuscleRecordingState struct is very similar to behaviorRecordingState. It contains:

• muscleCamera (MuscleCamera)
• muscleImageQueue (queue)
• muscleImageQueueMutex (mutex)
• muscleImageQueueCondVar (condition variable)
• latestBehaviorFrameHolder (LatestFrame) Refer to behaviorRecordingState for detailed explanations.

This struct is defined in TODO.

trackingControlState

This is a TrackingControlState struct that contains:

• motionControlHandlerReady (atomic boolean): Whether the motion stage controller is ready.
• latestMotionStagePosition (a custom MotionStagePosition class): The last sensed position of the motion stages. The frequency at which the current position can be read out is limited, so we update the stage position in a separate thread, and the updates are not synchronized with image acquisition.
• latestMotionStagePositionMutex (mutex, a previously discussed synchronization primitive): A "lock" that ensures no thread can read out the latest motion stage position while the motion control thread is writing to it.
• trackingOn (atomic boolean): Whether fly tracking is on. It can be handy to turn tracking off, for example, when you are manually handling the behavior stage. This can be toggled on the GUI.
• shouldOverrideTracking (atomic boolean): Whether we should override tracking. This boolean is set to true when the user has asked the motion stages to move to a specific location by clicking on the arena overview widget on the GUI, and the stages are still en route to that position. During this period, the motion stages moves independent of what the behavior camera sees.
• overridingPosX (atomic double): The X coordinate of the position that the user has asked the stages to move to (by clicking on the arena overview widget on the GUI).
• overridingPosY (atomic double): Same as above but for Y.

This struct is defined in TODO.

arduinoCommunication

The ArduinoCommunication class (not a struct!) that offers the following public methods:

• setBehaviorRecordingFPS
• setSyncRatio
• setBehaviorExposureTime
• setMuscleExposureTime
• startRecording
• stopRecording
• stopCommunication

The use of these methods is explained in Hardware interface, Arduino.

Under the hood, this class contains private members that allows the implementation of the methods above. These include, for example:

• A serial port connection that allows sending and receiving messages via USB.
• A queue of pending messages to be sent to the Arduino.
• A mutex and a condition variable that ensures only one thread can push a new message to the queue at a time.
• A background thread (the Arduino communication thread) that runs in a loop and sends any new message to the Arduino.

This class is implemented in recorder/src/peripherals/arduinoCommunication.cpp and arduinoCommunication.hpp.

saveDirectory

The SaveDirectory class (not struct!) offers the following public methods:

• setDirectory: Set a given directory as the save directory
• getDirectory: Read out the save directory
• initialize: Create the save directory and its subdirectories if they do not already exist

The class has the following private members:

• A std::filesystem::path object that keeps track of what the currently set directory is
• A mutex that ensures that the currently set directory cannot be read from and written to at the same time. This is important because multiple threads will need to read from (e.g. image saver threads) and write to (e.g. GUI) the path.

This class is implemented in TODO.

programmedRecordingStop

This is a ProgrammedStop struct that contains the following:

• numBehaviorFramesExpected (int): After this many frames have been acquired by the behavior camera, recording should stop.
• numMuscleFramesExpected (int): Same as above but for the muscle camera.
• hasEndedFlagForGUI (atomic boolean): When the behavior camera notices that it should stop recording now, it sets this flag to true. When the GUI thread sees that this flag is false, it shows that the recording has terminated in a pop-up window and toggles the flag back to false.

The numbers of behavior and muscle frames expected are set by the GUI thread based on the experiment protocol that the user has entered, particular the "stop" instruction. See The experiment protocol string for details.

The ProgrammedStop struct is implemented in TODO.

Detailed description of worker threads

Main thread

GUI

Behavior image acquiring thread

Pseudocode:

Inputs:
* behaviorRecordingState
    Shared state descriptor including (1) a pointer to the behavior camera
    (see "behavior camera interface" below), (2) a queue of images that
    have been captured by this (behavior image acquiring) thread that
    should be saved to the file system, (3)
* programState
* programmedRecordingStop

Behavior image saving threads (multiple)

Muscle image acquiring thread

Muscle image saving thread (multiple)

Motion control IO thread

Tracking control thread

Motion stage position logger thread

Arduino communication thread

Control programs

pco-camera-server

align-cameras

run-calibration-scan

run-spotlight

Utility tools

generate-calibration-board

fit-calibration-model

postprocess-recording

visualize-stage-trajectory