-
Notifications
You must be signed in to change notification settings - Fork 0
Pseudocode of CPP threads
This page contains pseudocode for the 10 types of threads and one process in the C++ codebase. One should read Design overview before reading this page.
The main recording thread is implemented in the main function in recorder/src/runSpotlight/runSpotlight_main.cpp.
Pseudocode:
Inputs:
- Command line arguments
Program:
Initialize logger
Initialize Qt GUI application
(See "Data in shared memory" page for definitions of initialized objects below)
Initialize programState
Load recorderConfig
Initialize saveDirectory
Load and initialize behaviorCamCalibrationParams
Initialize behaviorRecordingState
Initialize muscle RecordingState
Initialize trackingControlState
Start motion control IO thread from:
- motionControlRequestHandler
- recorderConfig
- trackingControlState
- programState
Start motion stage position logger thread from:
- motionStagePositionLogger
- recorderConfig
- trackingControlState
- saveDirectory
- programState
Start tracking control thread from
- trackingController
- recorderConfig
- behaviorRecordingState
- trackingControlState
- behaviorCamCalibrationParams
- programState
Initialize programmedRecordingStop
Start behavior image acquirer thread from
- behaviorImageAcquirer
- recorderConfig
- behaviorRecordingState
- programState
- programmedRecordingStop
Read numBehaviorImageSaverThreads from recorderConfig and start this many
instances of behavior image saver threads, each from:
- behaviorImageSaver
- recorderConfig
- behaviorRecordingState
- saveDirectory
- programState
muscleROI = Read muscle camera ROI from profile directory based on recorderConfig
Start muscle image acquirer thread from:
- muscleROI.imageWidth
- muscleROI.imageHeight
- muscleROI.xOffset
- muscleROI.yOffset
- recorderConfig
- profileDir
- muscleRecordingState
- programState
- programmedRecordingStop
Read numMuscleImageSaverThreads from recorderConfig and start this many
instances of muscle image saver threads, each from:
- recorderConfig
- muscleRecordingState
- saveDirectory
- programState
Initialize arduinoCommunication
Initialize GUI window
Start GUI application
(This starts an event loop and returns when user closes the window)
Wait for behavior image acquirer thread to finish
Wait for behavior image saver threads to finish
Wait for muscle image acquirer thread to finish
Wait for muscle image saver threads to finish
Wait for motion control IO thread to finish
Wait for motion stage position logger thread to finish
Wait for tracking controller thread to finish
The GUI application is implemented through callback functions—instructions of what to do when a certain event happens. The application then runs in an event loop, checking whether any event has occurred and whether any of the callback functions should be called.
This logic is implemented in the MainGUIWindow and MotionControlWidget classes in TODO. Both classes inherit from the Qt QWidget class.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- behaviorRecordingState
- muscleRecordingState
- trackingControlState
- behaviorCamCalibrationParams
- saveDirectory
- arduinoCommunication
- programState
- programmedRecordingStop
Program:
Initialize GUI widgets (text boxes, file browser, buttons, etc)
Set callback function:
When "start recording" button clicked:
Read user-set parameters
Parse experiment protocol string and populate expected
frame counts in programmedRecordingStop
Initialize save directory
Save metadata
Tell Arduino to start recording using user-set parameters
(Arduino will first stop all triggering this for a bit)
Wait 80ms for pending frames to arrive to be processed
Mark programState.isRecording as true
(All new frames that arrive *now* must be part of the
recording, not just streaming)
Set callback function:
When "stop recording" button clicked:
Tell Arduino to stop recording
Mark programState.isRecording as false
Tell Arduino to trigger at default streaming frame rates
Set timer:
Every 1 / streamingFPS seconds:
Get latest image from behaviorRecordingState
Get latest image from muscleRecordingState
Update displayed images
Initialize motion stage position overview widget
Set callback function:
When user clicks at a point on the preview window:
Take the pixel coordinates of the clicked point
Map pixel coordinates to desired physical coordinates
using behaviorCamCalibrationParams
Tell trackingControlState to override tracking and move
to the physical coordinates
Set timer:
Every 1 / motionStagePreviewUpdateFreq seconds:
Read out latest motion stage position from trackingControlState
Update position of red dot on the position preview window
Start event loop
This thread is implemented in the behaviorImageAcquirer function in TODO.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- behaviorRecordingState
- programState
- programmedRecordingStop
Program:
Read out configs from recorderConfig
behaviorCamera = Create BehaviorCamera object using said configs
Start behaviorCamera
buffer = buffer of 3 frames
WHILE not programState.toQuite:
frame = behaviorCamera.waitForOneFrame()
Load frame to behaviorRecordingState.latestBehaviorFrameHolder
IF programState.isRecording:
Attach index of this frame to the frame as metadata
Add frame to buffer
IF buffer of 3 frames is full:
Push buffer to behaviorRecordingState.behaviorImageQueue
Notify one behavior image saving thread through
behaviorRecordingState.behaviorImageQueueMutex and
behaviorRecordingState.behaviorImageQueueCondVar
Reset buffer of 3 frames
END IF
IF programmedRecordingStop.numBehaviorFramesExpected reached:
programmedRecordingStop.hasEndedFlagForGUI = true
(this will tell the GUI that recording has completed)
END IF
ENDIF
END WHILE
Stop behaviorCamera
This thread is implemented in the behaviorImageSaver function in TODO.
Multiple instances of the behavior image saving thread can be used.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- behaviorRecordingState
- saveDirectory
- programState
Program:
Set image saving parameters
WHILE not programState.toQuit:
Wait until notified by
behaviorRecordingState.behaviorImageQueueCondVar
Take one buffer of 3 images from
behaviorRecordingState.behaviorImageQueue
Save group of 3 frames to disk as a single pseudo-BGR JPEG file
Save metadata for said 3 frames
Do some statistics and logging to check if saving threads are
saving images fast enough
END WHILE
This thread is implemented in the muscleImageAcquirer function in TODO.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- imageWidth (plain integer)
- imageHeight (plain integer)
- xOffset (plain integer)
- yOffset (plain integer)
- recorderConfig
- profileDir
- muscleRecordingState
- programState
- programmedRecordingStop
Program:
Load muscle camera ROI from profileDir. This has been set by
align-cameras (see "TODO" page).
muscleCamera = Create MuscleCamera object using said ROI
(under the hood, this spawns a pco-camera-server process)
frameId = 0
WHILE not programState.toQuit:
frame = muscleCamera.waitForOneFrame()
Load frame to muscleRecordingState.latestBehaviorFrameHolder
IF programState.isRecording:
Set frame ID to frameId
frameId = frameId + 1
Push frame to muscleRecordingState.muscleImageQueue
Notify one behavior image saving thread through
behaviorRecordingState.muscleImageQueueMutex and
behaviorRecordingState.muscleImageQueueCondVar
ELSE
Reset frameId to 0 in preparation of the next recording
ENDIF
END WHILE
When the program breaks out of loop, muscleCamera goes out of scope
and automatically gets destroyed. The pco-camera-server
process gets terminatedin the destructor of muscleCamera.
This thread is implemented in the muscleImageSaver function in TODO.
Multiple instances of the muscle image saving thread can be used.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- muscleRecordingState
- saveDirectory
- programState
Program:
WHILE not programState.toQuit:
Wait until notified by
muscleRecordingState.muscleImageQueueCondVar
Take one frame from muscleRecordingState.muscleImageQueue
Save said frame and its metadata
Do some statistics and logging to check if saving threads are
saving images fast enough
END WHILE
The motion control IO thread, a.k.a. the motion control request handler thread, is implemented in the motionControlRequestHandler function in TODO. It runs in a perpetual loop in which it check if other threads have requested a command to be sent to the stage (i.e. if other threads have pushed a request to trackingControlState.requestQueue), and send it to the Zaber motion stage controller. Then, when applicable, it conveys the response from the Zaber hardware to the initiating thread. Therefore, this thread is the only one that interfaces with the motion stage hardware. All other threads (i.e. the tracking control thread and the motion stage position logger thread) go through this thread.
This thread, the tracking control thread, and the motion stage position logger thread share the following global variables. They are defined in a local namespace in the file where all three threads are implemented:
-
requestCondVar: condition variable -
requestMutex: mutex that works in conjunction withrequestCondVar -
responseCondVar: condition variable -
responseMutex: mutex that works in conjunction withresponseCondVar -
responseMap: dictionary mapping client ID (int) to response
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- trackingControlState
- programState
Program:
motionControl = Create MotionControl object
(under the hood, this establishes the connection with the
Zaber hardware controller)
WHILE not programState.toQuit:
Wait until notified by requestCondVar
myRequest = get one item from trackingControlState.requestQueue
myResponse = Create a MotionStageResponse object
clientID = hash of identifier of initiating thread as an integer
IF myRequest.requestType is GET_CURRENT_POSITION:
Get current position via Zaber API
Set myResponse.position to said position
ELSE IF myRequest.requestType is SET_TARGET_POSITION:
Move stage position to myRequest.position via Zaber API
without blocking (i.e. start motion and return
immediately without waiting for its completion)
ELSE IF myRequest.requestType is WAIT_UNTIL_IDLE:
Wait until both stages are idle via Zaber API
myResponse.isIdle = true
ELSE IF myRequest.requestType is CHECK_IF_IDLE:
myResponse.isIdle = Check if both stages are idle via Zaber API
ELSE IF myRequest.requestType is START_HOMING:
Home both stages without blocking (i.e. don't wait until complete)
END IF
responseMap[clientID] = myResponse
Notify all threads that listen on responseCondVar
END WHILE
The motion stage position logger thread is implemented in the motionStagePositionLogger function in TODO. It runs in a perpetual loop in which it asks the motion control IO thread for the most up-to-date stage position. Then, it populates the latestMotionStagePosition field of the trackingControlState struct, which is shared between several threads. Next, this thread writes the timestamp and the stage position to a log file. This thread then sleeps a little before doing the same again. The amount of time slept is determined by the update frequency configured in the "recorder_config.yaml" file.
Therefore, this thread serves two purposes:
- It updates the shared
trackingControlState.latestMotionStagePositionposition, which other threads can access when then need this information. - While recording, it writes a log of the stage positions, which is required to reconstruct the physical position of the fly in downstream post-processing.
The motion stage position logger thread is a client thread that works in conjunction with the motion control IO thread above and hey share certain variables. You should read about the motion control IO thread first.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- trackingControlState
- saveDirectory
- programState
Program:
period = 1 / logging frequency from recorderConfig
WHILE not programState.toQuit:
startTime = get precise current time
currPosition = Ask motion stage IO thread for current position
trackingControlState.latestMotionStagePosition = currPosition
IF programState.isRecording:
IF log file has not been created:
Create log file under saveDirectory
ENDIF
Append the "startTime, currPosition.x, currPosition.y" to log file
ELSE
IF log file is still open:
Close log file
END IF
END IF
Some profiling and logging - throw a warning is the logging thread
is not running fast enough
END WHILE
The tracking control thread is implemented in the trackingController function in TODO. It runs in a perpetual loop in which it gets the latest behavior image, detects the fly in the image, obtains the center-of-mass coordinates of the fly in pixel coordinates, and, based on the current stage position, calculates the fly's physical position. Then, it asks the motion control IO thread to move the stages there.
The tracking control thread is a client thread that works in conjunction with the motion control IO thread above and hey share certain variables. You should read about the motion control IO thread first.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- recorderConfig
- behaviorRecordingState
- trackingControlState
- behaviorCamCalibrationParams
- programState
Program:
Wait for the motion control IO thread to be ready
Get relevant parameters from recorderConfig
WHILE not programState.toQuit:
IF not trackingControlState.trackingOn:
(User has toggled tracking off, so do nothing)
ELSE IF trackingControlState.shouldOverrideTracking: (TODO)
(User has clicked on the motion stage overview GUI widget
and stages are en route to user-specified location)
currentPos = trackingControlState.latestMotionStagePosition
Calculate distance to user-specified location
IF said distance is within a preset threshold:
(We've reached the user-specified location)
trackingControlState.shouldOverrideTracking = false
CONTINUE
ELSE
Get user-specified overriding position again in case
it has changed in the meantime
Ask motion control IO thread to go to the new position
END IF
ELSE:
(Need to calculate where fly is and perform tracking)
image = Get latest behavior image from
behaviorRecordingState.latestBehaviorFrameHolder
stagePos = Get latest motion stage position from
trackingControlState.latestMotionStagePosition
isFound, flyPosPhysical = calculateFlyPositionAbsoluteMm(
image, stagePos, behaviorCamCalibrationParams, recorderConfig
)
currentPosPhysical = Physical position of the center of the field
of view from the current pixel position, calculated using
behaviorCamCalibrationParams.stagePosAndPixelPosToPhysicalPos
distToTarget = Distance between currentPosPhysical and flyPosPhysical
IF distToTarget >= a preset minimum distance:
dx = flyPosPhysical.x - currentPosPhysical.x
dy = flyPosPhysical.y - currentPosPhysical.y
Ask motion control IO thread to move to
(stagePos.x + dx, stagePos.y + dy)
END IF
Some profiling and logging - throw a warning is the tracking thread
is not running fast enough
END WHILE
Helper function:
FUNCTION calculateFlyPositionAbsoluteMm(
image, stagePos, behaviorCamCalibrationParams, recorderConfig
):
isFound = false
physicalPosXmm, physicalPosXmm = 0, 0
Rotate/flip image as needed to get the canonical orientation
Black out pixels that are within n mm of the physical boundary of
the arena. n is defined in recorderConfig. (This is to avoid
mistaking for bright boundary pixels for the fly).
Binarize image using a threshold set in recorderConfig
Apply morphological opening and closing with a smaller kernel defined
in recorderConfig
Find connected components (CCs) in resulting image
maxArea = number of pixels comprising the largest CC
IF maxArea > minimum set in recorderConfig:
comCol, comRow = Center-of-mass column and row of largest CC in pixels
physicalPosXmm, physicalPosXmm =
behaviorCamCalibrationParams.stagePosAndPixelPosToPhysicalPos(
stagePos.x, stagePos.y, comCol, comRow
)
isFound = true
END IF
END FUNCTION
The Arduino communication thread is embedded as a member of the ArduinoCommunication class. It is defined in the arduinoCommThreadFunc function in recorder/src/peripherals/arduinoCommunication.cpp. It is the only thread that directly communicates with the Arduino microcontroller via USB. It uses Qt's serial port API to read from and write to the Arduino.
Pseudocode:
Inputs (see "Design overview" page for definitions):
- portName (string)
- baudRate (integer): see https://en.wikipedia.org/wiki/Baud
- stopCommunication (atomic boolean)
- mutex (mutex)
- cv (condition variable)
- arduinoMessagesQueue (queue)
Program:
serialPort = Open serial port from portName at baudRate
incomingMessageBuffer = Empty string
WHILE true:
Wait until notified by cv of a new pending message,
OR until stopCommunication has been set to true
IF stopCommunication has been set to true:
BREAK
END IF
(There must be a new message that needs to be sent to Arduino)
Write pending message to Arduino
(Now, check if there's any incoming message *from* Arduino)
(This is only for logging/debugging)
IF there is incoming message:
Append incoming message to incomingMessageBuffer
IF incomingMessageBuffer now contains a complete line
Print said complete line to log
Erase the line that's just logged from incomingMessageBuffer
END IF
END IF
END WHILE
Close serialPort