Automated Categorization of Drosophila(Fruit Fly) Learning and Memory Behaviors using Video Analysis.
C++ C Shell Matlab Python
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
fly-tools
scripts
README.markdown

README.markdown

Automated Categorization of Drosophila Learning and Memory Behaviors using Video Analysis

For more information about this project, goto:

https://www.cs.drexel.edu/~david/Papers/Md_Reza_MS_Thesis.pdf

This repository contains all of the source codes used for the Fly Tracking project.

Scripts

You will find a few scripts to automate the processing of the data:

  • ./process-video-beta
  • ./batch-process

Tools

You will find the utilities in fly-tools. You will need to 'make' them which is easy to do. If you have trouble building, please make sure that you have gsl and ImageMagick++, MagickWand, OpenCV and CvBlob libraries and headers installed.

$ cd fly-data
$ make

The programs that will be built are located here:

  • ./fly-tools/FilterFlyMask
  • ./fly-tools/FlyTracking
  • ./fly-tools/mask-generator
  • ./fly-tools/standard-deviation
  • ./fly-tools/derive-background
  • ./fly-tools/filter-mask

Dependencies

Make sure you have these libraries and headers installed:

  • OpenCv
  • cvBlob
  • MagickWand
  • ImageMagick++

You will also need MPlayer to extract the PNGs.

Usage

Basically the fly tracking process takes several steps to complete. All of these steps can be automated with the process-video-beta script available in the scripts directory.

  1. Cropping and Extracting Cell Images
  2. Deriving Cell Backgrounds
  3. Generating Cell Masks
  4. Filtering Cell Masks
  5. Fly Tracking

Step 1: Cropping and Extracting Cell Images

Since each video has several fly cells at one time, we need to crop each cell into it's own set. To do so, extract one frame of the video and then use gimp to determine the x, y, height and width of each cell (the select tool is very handy for this). You should write each cell into a file in this format:

height:width:x:y

You should save this in a file (make sure to name it properly. For the purpose of this explaination I am going to be working on video "Group1"

Group1First10MinCropFile will look like this:

180:172:119:116
181:173:195:120
179:170:370:114

After you have completed that, you will want to run the extraction step of the process-video-beta script. To get a more detailed look at the arguments accepted by the processing script, run it with the -h flag.

process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -1

The -d flag indicates the folder you want to extract these files to, and will be used as the directory structure for the rest of the steps. The -1 flag tells it to run the first out of five steps.

This will take some time, and require a lot of CPU and disk space.

Step 2: Deriving Cell Backgrounds

The second step of the process is deriving the background image from the video. By doing this, we can subtract the background from the current frame to generate the mask. When subtracted the parts left over are the fly, while things in the background won't show up.

process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -2

This will derive the backgrounds for all of the Cells in the video seperately. After this tool is done running, it will output the Background.png into the mask folder. $data-root/$settype$setnumber/Background.png, for example it might be in data/First10MinSet2/Background.png.

Verify this image. Sometimes when the flies do not move the background might include the fly! If there are any left of flie parts, use GIMP to airbrush over that section with an appropriate color (guess what the background would be "close" too).

Step 3: Generating Cell Masks

The third step of the process is generating the Masks from the background and the cropped images.

process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -3

This step will output the masks in $data-root/$settype$setnumber/Masks/

Step 4: Filtering Cell Masks

The fourth step is the filtering step. It's important because it makes sure the masks do not have stray objects in them (which will mess up the final tracking sequence)

process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -4

The filtered masks are in $data-root/$settype$setnumber/Filtered/final/

Step 5: Fly Tracking

This is the final step of the image processing. It will generate results in the $data-root/$settype$setnumber/Final/. There will be PNGs showing the tracking results (refer to the paper to see what the polygons and lines refer too), a few files that indicate the standard deviation and mean of certain fly behaviors, and a "stat" file, which contains debug and statistical information as well.

This process willl take a long time beceause it iterates through each images in a set (about 18,000 images).

process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -4

After this is done running you should review the pngs to make sure that the fly tracking was performed correctlyAfter this is done running you should review the pngs to make sure that the fly tracking was performed correctly.

Into the Beyond

Now that we've done the hard work of image processing the rest is up to you! The data is all there, now you need to do manually generate the feature vectors, and you need to run our simple k-means clustering.

The K-Means clustering requires Java to run and our Least Squares Solution CCI calculator was written in Matlab and requires it to run.