Detailed Usage Instructions
Important note: Use a minimum screen resolution of 1920x1080 for optimal viewing of the GUI window (If you change screen resolution you might have to restart Matlab for proper functioning of GUI)**
Clone the GitHub folder and then run the following file in Matlab: string_pulling_behavior_analytics.m
The software was developed with Matlab 2016b but has been tested with Matlab2019b and works fine.
Prepared by Surjeet Singh
Prepared by Arashk Ghasroddashti
The following screen will appear after you run the above-mentioned file.
Click the button "Select a Video File" to open the following window for selecting a previously opened file or open a new file.
If you select "Load New File", a new window will appear to select a new video file.
Once a video file is successfully selected, the following GUI will appear after file loading is complete.
Next to the GUI, a frames display window (FDW) will appear, showing the first 20 frames.
Select Zoom: for viewing convenience in the frames display window (FDW), draw a rectangle around the region of interest (ROI) in the frame(s).
All frames in the FDW will be updated to reflect the selected ROI, as shown below (the “Select Zoom” button will also turn green). You may alter the selected ROI by using the “Reset Zoom” button.
Set Scale: for translation of pixel measurements into real measurements (e.g., cm or mm), drag your cursor along an object or feature of known length/measure (in unit of interest, such as cm) in the frame, and specify the length/measure in the window that appears (second image below) after making this selection. Once complete, the “Set Scale” button will turn green. You may alter the selected scale by repeating this step.
Measure: once a scale has been set with the “Set Scale” button, any object or feature in a frame may be measured for a result in the same units used to set scale (e.g., cm or mm). The result will be displayed in the MATLAB Command Window (where program outputs are displayed); the first and second numbers represent pixels and length in chosen units, respectively.
Measure String Thickness: drag cursor across string to record string thickness; this is used later on for other functionalities of the software.
Rows/Cols: you may change the number of rows and columns (and, thus, total frames) displayed in the FDW for viewing convenience. The FDW is set to display four rows and five columns (20 frames) by default.
Properties: you may change the visual appearance of the FDW - alter the color of tags, frame numbers, etc.
1 – Select Epochs
Under “Start,” enter the frame number corresponding to the beginning of a string-pulling bout/event.
Under “Stop” enter the frame number corresponding to the end of the string-pulling bout/event.
You may specify multiple epochs (start/stop pairs) by clicking “Add row.” You may erase an epoch by clicking “Delete row.” Clicking on any number in the epoch table will take you to that frame in the FDW.
2 – Select Colors:
First, ensure that “Epoch” is selected under “4 – Process” under the heading “Frames.”
Then, click on each of the five body parts/objects under “2 – Select Colors” to select the color range (CR) that applies to each. A new window will appear showing a frame in your epoch.
Left click and drag to create a rectangular region containing the body part/object in question (i.e., the body part/object that you just clicked on under “2 – Select Colors”; in the case of the example image below, we wanted to create a rectangular region containing the string).
Color clusters comprising the rectangular region you selected will appear. Click on a cluster that represents the body part/object in question.
The software will extrapolate a mask based on your selection (this will appear in the same window, to the right of the original frame/selection), and the colors in the cluster you just selected will be added to the CR corresponding to the body part/object in question (e.g., string in the example images below). At the same time, a new window will appear with a number of options.
Use “Add colors” to repeat the above process and add more colors to the CR applying to the body part/object in question (e.g., string in the example images below). For each frame, it is recommended that you repeat the “Add colors” process (i.e., select colors/clusters) until the calculated mask displayed to the right covers a significant majority of the body part/object in question (see image below, which shows a satisfactory mask for the string object). Moreover, it is recommended that you repeat this entire process in a minimum of three different frames in your epoch for each body part/object. This is to ensure sufficient generality of the CR for identifying the body part/object in all frames within your epoch.
Use “Remove colors” to remove the colors in the cluster you just selected from the CR applying to the body part/object in question (e.g., string in the example images provided). That is, “Add colors” and “Remove colors” are exactly opposite in function. This feature should be used if the CR applying to the body part/object in questions become too general (i.e., when calculated mask contains body parts/objects other than the one in question) to remove colors extraneous to the body part/object in question. While removing colors, the user has to enter a threshold value (see picture below) which defines the Euclidean distance of colors/points from the chosen colors to be removed. Larger the threshold value more pixels with colors closer to the chosen color value will be removed.
To increase or decrease the number of color clusters (subdivisions) a selected region is divided into for using “Add colors” or “Remove colors,” use “Edit Number of Clusters (n),” where n represents the current number of color clusters/subdivisions that each rectangular selection will be subdivided into.
Use “Save colors” periodically to ensure that the CR being constructed is being saved.
Use “Next frame” to move on to another frame in the epoch once you are satisfied with your selections in the current frame. Again, it is recommended that you perform this process of adding/removing colors in a minimum of three different frames in your epoch for each body part/object.
Use “Skip frame” to skip color selection in the current frame.
Use “Undo” to undo the last addition or removal of colors to the CR.
Use “Exit” to exit the color selection window(s).
Once you have successfully completed specifying the CR for a body part/object, the button for that body part/object under “2 – Select Colors” will turn green.
Move on to the next body part/object under “2 – Select Colors,” and continue this process until an appropriate CR has been specified for all five body parts/objects (all five buttons should appear green once this has been completed).
The "Import" button allows the use of CRs defined previously (in the processing of another video) to be used in the processing of the current video. This feature should only be used when the videos - i.e., (1) the video whose CRs are being imported and (2) the video that is currently being processed - were filmed in substantially similar environments (same lighting, camera angle, etc.) and with the same animal. If this feature is used otherwise, the CRs being imported will be unlikely to apply to the video being processed.
3 – Find Zoom Window for Masks:
Click “Find Body Box” to automatically find the smallest box (area) that contains the animal’s body.
You may also click “Set Manually” to draw a rectangular region defining the body box. To modify the body box generated automatically, you may click “Change” to manually enter coordinates, or you may alter the margins using the inputs on the right side. When using “Change,” note that the first pair of numbers represents the x and y coordinates of the top left corner of the body box and that the second pair of numbers represents the bottom right corner of the body box; the origin is the top left corner of the frame. For example, in order to increase the width of the body box, one would decrease the first number and/or increase the third number. If margins are altered and are to your satisfaction, ensure that the “Overwrite” box is checked prior to clicking “Find Body Box” again. Selecting “Select for Display Window” will effectively replace the region defined earlier with the “Select Zoom” button with the body box found in this step (and reflect this change in the FDW).
To find head boxes, first use “Test” to view the head box for a sample frame.
Based on this, alter the margin inputs on the right (under “Head,” not “Body”) and retest accordingly until the animal’s entire head is contained in the head box.
Then, use “Find Head Boxes” to find the head boxes through the epoch.
For both body boxes and head boxes, you may click on “Play” to view a frame-by-frame video of the epoch with frames cropped to the dimensions of the body box and/or head box. This can help verify that the body and head of the animal are visible in the body boxes and head boxes, respectively, for all frames in the epoch.
4 – Process:
- Note: “Overwrite” must be selected whenever you wish to redo/reprocess something under “4 – Process.”
- Note: the functionality of the “Find Objects” subpanel is dependent on processing that must be completed under the “Mask Processing” subpanel first. However, the functionality of the “Whole-Body Assessment” subpanel is independent of both the “Mask Processing” and “Find Objects” subpanels.
Under the “Objects” heading select one of the five body parts/objects, and under the “Frames” heading select the frame(s) that you would like to process (find masks for, etc.).
- Important note: masks must be found for the body parts/objects in the order shown (string first, then body, then ears, then nose, and, finally, hands).
Under “Mask Processing,” select the processing method (KNN, Range Search, or Grid Way) you would like to employ.
Click “Find Mask” to find masks using the selected method.
Experiment with the three available methods, as one may be better suited to your dataset. If Range Search is selected as the processing method, then you may specify a “Radius” under “Specs.” This radius is the Euclidian distance from a color in 3-D space (the three dimensions being RGB); the resulting sphere is the color range that the program will identify as the body part/object being processed; such a sphere is delineated around every original color selected for the body part/object in Step “2 – Select Colors.”
Use “Erase Mask” to erase masks from the frame(s) specified under “Frames.” You must do this first if you wish to re-find masks for frames that were previously processed.
Once masks have been found, use “Find Object” to find computed masks/objects for the frame(s).
When using “Find Object” on hands, hands in the first frame in the epoch/range must be selected manually (you may do this by right-clicking on the first frame).
Furthermore, it is advisable to stop processing immediately after an incorrect label to prevent errors from propagating through the remainder of frames and affecting accuracy.
Once objects are found, this will be reflected in the FDW.
To correct an error, use “Find Manually.” You may select “Draw Region” prior to using “Find Manually” if you wish to manually draw a polygon around the body part/object of interest. If “Draw Region” is not selected, then you may simply draw a rectangle around (containing) the body part/object of interest, following which the program will identify the body part/object of interest using masks. When processing hands, you may alternatively select “Rectangle” prior to using “Find Manually;” this allows the drawing of a smaller rectangle for each hand such that each rectangles contains not a whole hand, but rather a subsection of a hand, which will define the location (coordinates) of the hand in the frame in question.
Use “Erase Object” to erase computed masks/objects from the frame(s) specified under “Frames.”
Use “Descriptive Stats” under “Find Objects” to view descriptive statistics based on computed masks/objects. The figures shown will also be saved as PDFs in the folder where the video file being analyzed is located – in the “pdfs” folder inside the “[video_filename]_processed_data” folder.
Under “Whole-Body Assessment,” use “Estimate Motion” to perform optical flow analysis on the frame(s) specified under “Frames.”
Use “Descriptive Statistics” under “Whole-Body Assessment” to view descriptive statistics based on masks (as opposed to objects).
Use “Temporal Analysis” to quantify randomness (on the basis of color) along the temporal dimension (across the selected frame(s)) for every pixel.
Use “Find PCs” and “Find ICs” to perform principle component and independent component analyses, respectively. You may adjust the number of components in the box just off to the right.
For the processes available under the “Whole-Body Assessment” and “Mask Processing” subpanels, computational/processing time can be reduced by enabling the “Reduce Image Size” option and specifying an image reduction factor; an image reduction factor of 4 is suggested as a starting point. Lower image reduction factors yield finer analysis but increase processing time as the trade-off. The lowest image reduction factor (i.e., that which produces the finest analysis and requires the greatest processing time) is 1. The image “Reduce Image Size” feature does not apply to finding objects.
Tags: for selection of the features (body parts/objects) whose tags you would like to view in the FDW.
Display Areas: when selected, displays the areas corresponding to the ears, nose, or hands in the FDW as opposed to localized dots for the body part selected under “4 – Process.” In the example image below, “Display Areas” is checked while “Hands” are selected under “4 – Process.”
Display Masks: when selected, displays masks corresponding to the ears, nose, or hands in the FDW for the body part selected under “4 – Process.” In the example image below, “Display Masks” is checked while “Body” is selected under “4 – Process.”
Display Object-Masks: when selected, displays computed masks/objects in the FDW for the body part selected under “4 – Process.” In the example image below, “Display Object-Masks” is checked while “Body” is selected under “4 – Process.”
Note: “areas” and “computed masks/objects” are the same thing except for the fact that the former does not have fill (only the perimeter line of the area is displayed).
Show Masks: in a separate window, displays masks of all body parts in selected frame.
Body, Smallest area to neglect: when finding computed masks/objects, high-density areas of the mask are identified and clustered together. If a cluster (as defined in the previous sentence) is equal to or smaller in size/area/number of constituent pixels than the value entered in this box, then it is discarded and will not be used to compute the computed mask/object.