Motion correction on large datasets

Here is a few tips for dealing with large datasets:

• Do not load the file in memory: You can run the motion correction routines normcorre/normcorre_batch and apply_shifts by not loading the file in memory and instead giving as an input the path to the file location. The supported filetypes are .tif, .h5/.hdf5, and memory mapped .mat files. The apply_shifts function can also be used with miniscope .avi files for 1p data.

• Save the output directly with loading it in memory: You can save the result of the motion correction directly as a .tif or .h5 file by changing the options.output_type entry to .tif or .h5, respectively. Make sure you also change the entries options.tiff_filename or options.h5_filename to specify the name of the output file. Memory mapped output is also supported but it's not recommended since it can be much slower.

• Do not convert to double: If you still wish to load the file in memory, do not modify the file datatype. uint8 takes 1 byte in RAM per entry, uint16 2, single 4, and double 8. See also this nice entry. If you want to perform arithmetic operations (e.g., high pass filtering for 1p motion correction) conversion to single (and not to double) precision is typically sufficient.

• For 1p data motion correction: If you're using the package for 1p data motion correction, you can save the high pass filtered version of the movie directly in the hard drive, operating on a small chunk at a time. The proceed by motion correcting the high pass filtered movie and applying the shifts to the original movie, without loading any of the two files in memory. This process is demonstrated in the script demo_1p_low_RAM.m. Note that the choice of chunk_size=500 in this script is rather conservative and will cause speed slowdowns since the reader will always start reading the file from the beginning to get the next chunk of data. It is recommended that you increase this number significantly to a value that your machine can handle (e.g., chunk_size=2500).