3 ‐ Preprocessing

It is often useful to prepare your videos before analysis. Maybe you don't want to work on the entire video, or you want to compensate for a slightly tilted camera, perhaps improve contrast or something else. You can do these right when you load your video in an MgVideo. There are seven types of preprocessing steps (listed in order of execution):

• trimming: Trim contents of the video based on starttime and endtime.
• skipping: Skip every n frames, where n is determined by skip.
• fixing: Specify a fixed target number frames to extract from the video.
• rotating: Rotate the video by an angle determined by rotate.
• contrast/brightness: Adjust contrast and brightness of the video, where contrast and brightness are the level of adjustment in percentages (meaning contrast=0 will not apply any change). both values range from -100 to 100.
• cropping: Crop frames in video. If crop='auto' the module will attempt to find the area of motion, if crop='manual' we can draw the cropping rectangle over the first frame.
• grayscale: Convert the video to grayscale with specifying color=False. This will also cause all further processes called on the MgVideo to function in grayscale mode.

Trimming

You can trim the duration of the video like this:

# the numbers used for starttime and endtime represent time from the video's timeline in seconds
trimmed = musicalgestures.MgVideo('/path/to/video.avi', starttime=5, endtime=15) # take the video content between the 5th and 15th seconds of the timeline
trimmed.show() # view the result

Skipping

In order to save time, skipping every other frame, or more, in the analysis can give you a faster analysis while still getting an idea of the motion. You can for example set this by adding skip=2, to skip two frames before including a frame in the analysis, then skipping two again.

shrunken = musicalgestures.MgVideo('/path/to/video.avi', skip=2)
shrunken.show() # view the result

Fixing

In order to batch processing long video files, and files of different duration, fixing a fixed target number of frames to extract can give you a faster analysis while still getting an idea of the motion. You can for example set this by adding frames=1000, to find the skip-frame number that would result in the total number of output frames equal to that specified in the parameter. It is also possible to extract only the keyframes of the video file by setting the parameter to frames=-1.

fixed = musicalgestures.MgVideo('/path/to/video.avi', frames=1000)
fixed.show() # view the result

# Extracting only keyframes
keyframes = musicalgestures.MgVideo('/path/to/video.avi', frames=-1)
keyframes.show() # view the result

Rotating

Sometimes source videos are recorded with a slightly off horizon, or with the camera mounted sideways, therefore it is desirable to rotate the video by a few (or more) degrees. We can do this by simply specifying the angle we want to rotate with for the rotate parameter of our MgVideo:

# rotate by 90 degrees
rotated = musicalgestures.MgVideo('/path/to/video.avi', rotate=90)
rotated.show()
# or just a little bit...
rotated_a_bit = musicalgestures.MgVideo('/path/to/video.avi', rotate=5.31)
rotated_a_bit.show()

Adjusting contrast and brightness

During preprocessing you can also add (or remove) some contrast and brightness of your video.

brighter = musicalgestures.MgVideo('/path/to/video.avi', contrast=100, brightness=20) # added +100 contrast and +20 brightness (0 means unchanged)
brighter.show()

Cropping

If the video frame has big areas with no motion occurring, a lot of time could be saved if only the area with motion was used in the analysis. One useful tool developed for the pre-analysis is the crop = 'auto' input, which automatically finds the area with significant motion in the input video. The movement occurring has to be above a low threshold, as to not include irrelevant background motion from shadows, dust etc. Another mode is crop = 'manual', where you can manually mark a rectangle around your area of interest.

Automatic Cropping

auto_cropped = musicalgestures.MgVideo('/path/to/video.avi', crop='auto') # crop to the area of movement (assessed automatically)
auto_cropped.show()

Manual Cropping

manual_cropped = musicalgestures.MgVideo('/path/to/video.avi', crop='manual') # crop manually by selecting the desired area with the mouse and press "c" to crop or "r" to reset the window
manual_cropped.show()

Grayscale mode

At the final preprocessing stage we can also choose to convert the video to grayscale with specifying color=False (by default color=True). This will not only result in a grayscale version of our source video, but also informs all future processes to work in grayscale mode. The technical benefit of this can be slightly shorter processing times at most processes, since in grayscale mode we process a single color channel per frame (instead of three channels).

gray = musicalgestures.MgVideo('/path/to/video.avi', color=False) # convert to grayscale
gray.show()

Summary of preprocessing modules

As we have seen, we can optionally apply six types of preprocessing to the video we load into our MgVideo, they are (in order of execution):

• trim: Trim contents of the video based on starttime and endtime.
• skip: Skip every n frames, where n is determined by skip.
• rotate: Rotate the video by an angle determined by rotate.
• cb: Adjust contrast and brightness of the video, where contrast and brightness are the level of adjustment in percentages (meaning contrast=0 will not apply any change). both values range from -100 to 100.
• crop: Crop frames in video. If crop='auto' the module will attempt to find the area of motion, if crop='manual' we can draw the cropping rectangle over the first frame.
• grayscale: Convert the video to grayscale with specifying color=False. This will also cause all further processes called on the MgVideo to function in grayscale mode.

Keep everything

Notice that although we can optionally apply up to four preprocessing modules to our source video, normally we only keep the final result. If you would like to keep the results of all modules, set keep_all=True.

preprocessed_video = musicalgestures.MgVideo('/path/to/video.avi', starttime=5, endtime=15, skip=3, rotate=90, contrast=100, brightness=20, crop='auto', color=False, keep_all=True)

This will output six new video files:

• video_trim.avi
• video_trim_skip.avi
• video_trim_skip_rot.avi
• video_trim_skip_rot_cb.avi
• video_trim_skip_rot_cb_crop.avi
• video_trim_skip_rot_cb_crop_gray.avi