Generating videos and consolidating metadata

The postprocess-recording command-line tool consolidates data collected during recording and transforms them into a more efficient, more user-friendly format. This program is written in Python and it is implemented in tools/src/spotlight_tools/scripts/postprocess_recording.py.

First, the program mergers behavior image frames into a video. During the recording, images from behavior are saved as pseudo-BGR JPEG images—each image actually contains three adjacent frames in the three color channels. This trick helps balance a trade-off between data compactness (through compression) and speed (data compression/saving must be at least as fast as data acquisition). However, once the recording is over, compression speed is no longer a constraint. Therefore, the pseudo-BGR format is no longer optimal, and it is not very user-friendly. The programs convert the behavior images into a video, where each frame in the video corresponds to an actual frame from the recording (no longer comprising three adjacent frames in the three color channels).

Second, the program merge metadata into one single file. During the recording, a separate metadata file containing the timestamp(s) at which the frame (or group of three frames in behavior recording) are recorded is saved every time an image is saved. This is because there are multiple saving threads, and saving metadata in many small files avoids the tricky situation of multiple threads trying to write to the same file, which can lead to race conditions. However, this data format is not very inode-efficient or user-friendly. Therefore, this program merges them into a single CSV file.

The postprocess-recording program accepts the following command-line arguments. This help message can also be printed by running postprocess-recording --help.

$ postprocess-recording --help
usage: postprocess-recording [-h] [OPTIONS]

Postprocess data recorded by the Spotlight setup. First, this function consolidate
metadata for each behavior frame, generating a single CSV file containing the
acquisition time (from camera's internal clock), image-received time (in UNIX epoch
time), and the estimated positions of the motion stages (their positions are logged at a
frequency typically lower than the behavior recording frequency, so some interpolation
is needed.

Second, this function merges the behavior camera frames into a single video containing
monochrome frames using the H.264 codec. Note: during recording, the behavior frames are
originally saved in 'pseudo-BGR' files. Namely, each file actually contains three
consecutive frames in the three channels. The output of this function no longer employs
this trick - each frame is just an actual monochrame frame.

For more information on the video compression parameters, see
https://trac.ffmpeg.org/wiki/Encode/H.264

╭─ options ──────────────────────────────────────────────────────────────────────╮
│ -h, --help                                                                     │
│     show this help message and exit                                            │
│ --recording-dir PATH                                                           │
│     Root directory of the recording. This is the path that you set             │
│     in the Spotlight recording GUI. (required)                                 │
│ --overwrite, --no-overwrite                                                    │
│     If True, this function will overwrite existing files.                      │
│     Otherwise, an exception is raised if the output file(s) already            │
│     exists. Default is False. (default: False)                                 │
│ --play-fps INT                                                                 │
│     Frames per second for the output video. Note that this value is            │
│     only used for *displaying* the data. The real FPS is set when              │
│     the data is collected. For example, if data is collected at 300            │
│     FPS, and `play_fps` here is set to 30, then the video will be              │
│     played at 0.1x speed when opened by a video player (even if the            │
│     video player thinks it's playing at 1x speed). This can be                 │
│     handy sometimes. Default is 30. (default: 30)                              │
│ --behavior-video-crf INT                                                       │
│     Constant Rate Factor (CRF) to be used for video compression.               │
│     CRF is a quality-based encoding method that lets users target a            │
│     specific quality level rather than bitrate. CRF values range               │
│     from 0 to 51. Lower values = higher quality and larger files,              │
│     vice versa. Generally, 0 is mathematically lossless; 1-5 are               │
│     visually lossless; 15-18 are very high quality. Default is 5. (default: 5) │
│ --behavior-video-preset STR                                                    │
│     A preset is a collection of options that will provide a certain            │
│     encoding speed to compression ratio. A slower preset will                  │
│     provide better compression (compression is quality per                     │
│     filesize). Choose from: "veryslow", "slower", "slow", "medium",            │
│     "fast", "faster", "veryfast", "superfast", "ultrafast". Default            │
│     is "slow". (default: slow)                                                 │
│ --num-frames {None}|INT                                                        │
│     If set, the video will contain only the first `num_frames`                 │
│     frames. This is useful if you want to generate a very short                │
│     video just to make sure that the data pipeline is working.                 │
│     Default is None. (default: None)                                           │
╰────────────────────────────────────────────────────────────────────────────────╯

Upon successful completion, this program saves the postprocessed data under a new "processed" subfolder under the recording directory.

Optionally, the user may Visualizing the travel path of the motion stages.