Skip to content

orlandi/netcal

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

52 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

logo

NETCAL: An interactive platform for large-scale, NETwork and population dynamics analysis of CALcium imaging recordings

NETCAL is a MATLAB-built, dedicated software platform to record, manage and analyze high-speed high-resolution calcium imaging experiments. Its ease of use, interactive graphical interface and exhaustive documentation is aimed to wet-lab researchers, but it will also meet the needs of any experienced data scientist through its plugin and scripting system. We have developed a large set of tools and incorporated state-of-the-art algorithms and toolboxes for large-scale analysis of network and population dynamics. Analyses include: automated cell detection (both static and dynamic); trace and population sorting through machine learning, clustering and pattern recognition; bursting dynamics; spike detection; network inference (from functional networks to causal relations); and many more. Several of these tools are also available in real-time, e.g. cells and spikes can be monitored during the actual recording, giving the researcher extensive feedback on the progress of the experiment.

Installation instructions

  • You can find here the latest installation instructions.
  • Install additional MATLAB toolboxes: Go to the installDependencies folder and execute the mltbx files (or drag and drop in MATLAB). NOTE: In MATLAB r2017a+ you might need to download the GUI Layout Toolbox from the Add On Explorer yourself.
  • Change default MAT-Files version: Go to MATLAB Preferences -> General -> MAT-Files to v7.3
  • To be able to export figures to pdf and eps you need ghostscript installed.
  • To be able to use any of the python modules you need python installed and available to MATLAB

Requeriments

  • Matlab 9.1+ (2015a or higher) - 2017a or higher recommended. Seriouslly, upgrade.

MATLAB Tolboxes

  • Bioinformatics Toolbox (4.5.1) [bioinfo]
  • Curve Fitting Toolbox (3.5.1) [curve fit]
  • Econometrics Toolbox (3.2) [econ]
  • Global Optimization Toolbox (3.3.1) [globaloptim]
  • System Identification Toolbox (9.2) [ident]
  • Image Processing Toolbox (9.2) [images]
  • Neural Network Toolbox (8.3) [nnet]
  • Statistics and Machine Learning Toolbox (10.0) [stats]
  • Computer Vision System Toolbox (6.2) [vision]

Change Log

[8.6.0] - 2018-10-08

Added

  • Tutorials / Documentation section
  • Ability to change the default average image (useful for ROI detection)
  • ROI detection using a contour image

[8.5.0] - 2018-09-29

Added

  • Working on full compatibility with MATLAB R2018b

[8.4.5] - 2018-09-27

Attention

If you are upgradint to MATLAB R2018b make sure to update the GUI Layout Toolbox too! (Version 2.3.2 minimum)

Added

  • Support for MATLAB R201b (experimental)

Fixed

  • Findjobj required an update tow ork wiht r2018b
  • Spatial kernel on CellSort ROI detection was only being applied on the second dimension

[8.4.4] - 2018-09-26

Fixed

  • Leftover errors on the denoiser when using too large blocks

[8.4.3] - 2018-09-26

Modified

  • Small changes to the denoiser output

[8.4.1] - 2018-09-10

Modified

  • Added warnings in case the drift correction returns an unquared mask

[8.4.0] - 2018-09-08

ATTENTION

Hey! Due to some mailing issues, if you ever sent me a question/request to javierorlandi@javierorlandi.com or *@itsnetcal.com and never got answered. Please send it again to javiergorlandi@gmail.com or javier.orlandigomez@ucalgary.ca . Thanks! (I believe those mails have not been functioning properly for the last year)

Added

  • Several options to automatically import ROI from files and experiments to multiple experiments at once
  • Redesigned export movie pipeline function (so it also works ok with drifft corrected movies)
  • Added big tiff export support
  • Pipeline mode: help window to provide a small description of any selected available function
  • Most pipeline statistics functions are now available on the single and batch mode menus
  • Several pipeline functions added to the main menus
  • Option to export inferred networks to GEPHI (with any additional network statistic you might think of)

Fixed

  • Error when ROI with and without weights coexisted
  • Error when trying to plot fluorescence burst statistics from the bursts GUI

[8.3.0] - 2018-09-07

Added

  • Complete overhaul of the drift correction. Now it should be compatible with the denoiser and cell sort
  • Option to automatically add custom pipeline functions (any m file inside the plugins/pipeline folder is a candidate)

Modified

  • AutoLevels is now based on (iterative) accurate percentile calculations. Not on fixed pixel values

Fixed

  • Many issues with 32 bit images

[8.2.8] - 2018-09-05

Added

  • More features to the drift correction (mostly for in vivo recordings)

Modified

  • Quit accelerator is now Ctrl+Q And Clear Log is Ctrl+X

[8.2.7] - 2018-09-04

Added

  • Drift correction preprocessing feature. For samples that move around: badly fixed coverslips, human intervention, in vivo recordings, etc. You can find it in preprocessing -> drif correction. Uses MATLAB image registration features. For now only rigid body and translation transformations. Once run, it will automatically be applied everytime a frame is captured. So should be compatible with any preexisting function. However, consider it an EXPERIMENTAL feature.

[8.2.6] - 2018-09-04

Modified

  • Added verbose output to fast ICA analysis

Fixed

  • Autolevels function not working on 32 bit images

[8.2.5] - 2018-09-03

Fixed

  • Updater issues

[8.2.4] - 2018-09-03

Modified

  • Statistics plots now run much faster (when multiple statistics are selected)

Fixed

  • Bug when UnivarScatter would enter in an infinite loop
  • Bug when trying to arrange inexistent figures

[8.2.3] - 2018-08-31

Fixed

  • FastICA was only available on the development branch

[8.2.2] - 2018-08-30

Fixed

  • New .TIF or .TIFF could not be added from the load Experiment menu

[8.2.1] - 2018-08-29

Added

  • Plot statistics now shows the # experiment being processed
  • Clustering coefficient measures for directed networks
  • Bunch of new network statistics measures
  • Treatment based network statistics

Modified

  • Now any network plots requires an active network to be set
  • New option to set the active network (adjacency matrix) based on some inference measure

Fixed

  • Tif files without metadata are now being loaded correctly
  • DenoiseRecording was using the transposed image in some instances
  • Small bug fixes

8.2.0 - 2018-08-14

Added

  • Network inference: fast estimator of Generalized Transfer Entropy with significance estimators (fully working)
  • Many network measures (from the brain connectivity toolbox)
  • Full Brain Connectivity Toolbox
  • sigstar allows negative p values (will display # instead of *)
  • Burst statistics. Added high and low bursting rate measures (those that recruit above and below 20% of the population respectively)

8.1.3 - 2018-07-20

Added

  • Global Firing Rate to spike statistics plots
  • Option to automatically export data to csv when generating statistics figures
  • Option to plot several burst spike statistics within the same pipeline call
  • Hardcoded transparency effects on UnivarScatter bars (to fix exporting issues)
  • Option to not use weights when displaying ROIs with weights
  • Option to circumvent pipeline errors when using updated functions.

Modified

  • Right click ROI sorting is now accessible when viewing the average of current ROI within the view Traces window (not only when viewing the movie anymore).

Fixed

  • Multigap subplot had transposed output
  • Issue when cloning experiments with missing fields
  • Small bug fixes

[8.1.2] - 2018-07-08

Added

  • New view recording plugin (2) for fast viewing of recordings (specially for HIS files). Allows drag and drop of the movies and switching between different movies within the same window

Fixed

  • Several files were not being uploaded to the repository due to a faulty .gitignore
  • Several small bug fixes

[8.1.1] - 2018-06-26

Added

  • Univarscatter module to plot statistics functions

8.1.0 - 2018-06-26

Added

  • New features for visualizing defining and sorting schmitt-based inference
  • Statistics plots from schmitt inference features
  • Ability to define more threshold on schmitt inference (minum sizes and durations)
  • More options to measure differences across treatments

Fixed

  • Bug when importing experiments from existing projects with duplicate names would result in the existing experiments being overriden by the new one
  • Improved renaming of existing files when renaming or importing experiments
  • Substantial speed-up when processing statistics across many experiments

8.0.7 - 2018-06-20

Added

  • Option to directly compute p-values on the treatment statistics for each experiment pair (ttest2, Mann-Whitney, Kolmogorov-Smirnov)
  • minimum event size and duration for schmitt-based inference
  • Several new observables for schmitt-based inference

Modified

  • Length of patterns during selection in the pattern-based classification is now limited to the sampling range

Fixed

  • Warning while loading experiments with bigtiff and TIFF_LONG8 datatypes

8.0.6 - 2018-06-18

Added

  • Extra feature list on schmitt trigger based inference
  • Visualization of schmit events on the view traces menu

Fixed

  • Error when identifying similarities in traces if only 1 trace was present
  • Several typos

[8.0.5] - 2018-06-18

Added

  • Transfered many of the new plot statistics options to the treatment version. Exports, significance, etc.

Fixed

  • Improved support of big tiff files
  • Speed up on statistics plots when a large number of labels exist

8.0.4 - 2018-06-11

Added

  • Option to show the standard error of the mean on all plot Statistics functions
  • Speed-up of plot Statistics functions when many labels are present

[8.0.3] - 2018-06-07

Fixed

  • Issue when the project export would get the wrong file list if any experiment was missing some of the bigFields
  • Shortened project names in the recent projects list
  • Error on the denoiser when the number of ICA components was lower than the PCA ones (due to degeneracies)
  • Removed the 'no changes were made' after closing GUI windows
  • Fixed a bunch of issues on the plotStatistics function when the number of groups was bigger than 1
  • Fixed errors due to the Holm-Bonferroni correction on several fringe cases

8.0.2 - 2018-06-06

Fixed

  • Issue when the experiments were not being correctly saved after importing a windows project on a nix system on pipeline mode

8.0.1 - 2018-06-06

Fixed

  • recent projects list not displyaing correctly when using samba shares. Now it should work seamlessly on win and nix systems

[8.0.0] - 2018-06-06

Added

  • Full implementation of CellSort-like (Mukamel 2009) activity-based ROI detection. Selects ROIs after a PCA/ICA dimensionality reduction. Implemented within the denoiser and the pipeline -> fluorescence -> cellsort. Guide coming soon!
  • Pattern count classifier now also creates intersection groups (if pattern overlapping is allowed). Example: you can now end up with traces that belong to both neuron and glia populations in a new subgroup
  • New project export feature. You can export the project to a new folder with only a subset of experiments and a subset dditional data (traces, denoisedData, etc) for reduced project size. Useful to share with collaborators
  • On the traces window you can now sort traces by some ROI properties. Size, position, etc.
  • Option to disable saving the experiment on some figure plots. This will result in a speed up on the pipeline when you are only interested in just plotting stuff.
  • Added fast mode for trace extraction (not recommended, only useful for cellsort ROI detection).
  • Ability to view the PCA and ICA temporal components on the traces menu (after denoising).

Modified

  • Several ROI visualization options now work with weighted ROIs. Weighted ROIs only appeear in the cellsort ROI detection (for now).
  • Simplified the saveProject file.
  • Lots of changes to the denoiser. Now ICA is always performed after PCA.

Fixes

  • Lots of small bug fixes

7.4.1 - 2018-04-16

Added

  • Batch change of framerate for multiple experiments
  • Batch change of multiple experiment handles
  • Batch change of number of frames for multiple experiments
  • Unpaired two-tailed t-test statistic

Fixed

  • Refine ROI threshold was using old progress bar code
  • Movie export profile was not being correctly assigned in some cases
  • Rate-based spike conflict fixing was asking for unecessary variables

7.4.0 - 2018-04-03

Added

  • Option to remember selected FPS on avi and btf files
  • Option to move one ROI at a time (or random) on Spike Inference Training

Modified

  • Option to redo ROI index list is now on the Modify ROI menu
  • loadTraces now tells you what is loading (sometimes)
  • Default range of preprocessing temporal subset (from 600s to inf)

Fixed

  • Bug when training with OASIS failed due to the ncbar not being present

[7.3.4] - 2018-04-03

Fixed

  • Small bug fixes

[7.3.3] - 2018-03-07

Added

  • Additional options can now be passed to the raster plot through an external .m file. Will be rolling this feature to all plots in the near future

[7.3.2] - 2018-03-06

Added

  • Continuous version checker (every hour)

Modified

  • Raster plot function can now plot multiple groups together (using different colors)

7.3.1 - 2018-03-06

Added

  • New spike-based group classifier (identify ISI). Will group spikes with similar ISI statistics (uses two-tailed KS as similarity measure)

Fixed

  • Small bug fixes

7.3.0 - 2018-03-05

Added

  • Support for mj2 files
  • Options to detect spike conflicts based on rate

Modified

  • Major rework on statistics plots. New significance options and distributions features
  • Overhaul of export movie features
  • Color of log headers

Fixed

  • Many small bug fixes

[7.2.6] - 2018-02-16

Modified

  • Added options for spike train exports

[7.2.5] - 2018-02-06

Added

  • X-based burst detection

7.2.4 - 2018-02-05

Added

  • Suprise-based burst detection

7.2.3 - 2018-02-02

Added

  • Semi-automatic burst detector based on ISI_N

[7.2.2] - 2018-02-01

Modified

  • ISI_N burst detection: now each burst is plotted in a different color
  • ISI_N burst detection explorer: now shows the estimated positon of maximums and minimums of the distribution

Fixed

  • New options were not being stored within the experiments after a succesful parallel pipeline run

[7.2.2] - 2018-01-31

Fixed

  • Small bug fixes on KCL analys and ISI_N burst detectors

[7.2.1] - 2018-01-30

Added

  • Option to store traces with single precision (EXPERIMENTAL)

7.2.0 - 2018-01-30

Added

  • Bivariate gaussian kernel density estimation functions
  • nhist for prettier histograms
  • Option to plot multiple statistics simultaneously (spikes and KCl for now)
  • Option to tile (most) pipeline-generated figures

Modified

  • Completely redesigned several options on acute treatments analyses

Fixed

  • Export KCl features now works with the new data

Bugs

  • PCA analysis with KCl features is now broken

7.1.10 - 2018-01-29

Added

  • spike bursts statistics plots now have the option to plot all statistics simultaneously

Modified

  • Forced option to copy files when renaming experiments

7.1.9 - 2018-01-25

Added

  • Option to show signififcance level (stars) on statistics plots. Mann-Whitney and Kolmogorov-Smirnov for now.

Fixed

  • Bug when importing ROIs from an external file would result on their centers being transposed (has no effect on anything other than exporting ROI positions). Added pipeline function to recalculate centers based on the actual pixel values.

7.1.8 - 2018-01-24

Added

  • Pipeline function to create a new population based on firing neurons (those with a minimum number of spikes)

Fixed

  • Jitter option was always on on the ISI_N burst detector

7.1.7 - 2018-01-24

Modified

  • Changed ROI factor name for event in statistics plots
  • Changed amplitude for number of spikes (and a normalized option) for spike-based burst statistics

Fixed

  • Bug when loading/saving pipelines was not working if they contained deep structures with table-like lists

7.1.6 - 2018-01-23

Added

Modified

  • Previous spike burst detector is now called rate detector. Also added option to plot (or not) its results

7.1.5 - 2018-01-23

Added

  • Peeling spike inference now has several new options to allow more customization (as in the original paper)
  • Cartesian coordiantes export of ROI centers
  • Export ROI coordinates button to the view ROI GUI

Modified

  • ROI centers export on the pipeline now has its own section (ROI)
  • Spike inference modes are now in the same order as in the training window

7.1.4 - 2018-01-22

Modified

  • Moved Trace Fixer to the end of the View menu to avoid user missclicks

7.1.3 - 2018-01-22

Fixed

  • Spike inference: Schmitt trigger should correctly display detected events and thresholds in the training window

7.1.2 - 2018-01-22

Added

  • exportROIcenters: pipeline function to export ROI centers of a given group (groups: exports)

7.1.1 - 2018-01-22

Added

  • viewROI now accepts power spectrum images
  • peak detector to detect peaks in traces (will be used to refine pattern-based classifiers)
  • module to detect dying cells (those with step-like changes in fluorescence levels)
  • view traces available in batch mode
  • Several new features on the KCl analyis module
  • Spike-based burst detection method
  • Power-spectrum based preprocessing (obtain different average images based on power-spectrum bands)
  • More spike features (spikes in bursts)
  • More options for MLspike inference
  • Cross-correlation measures
  • New raster plots with average activity on top
  • Options to multiply or divide all frames using an external file (to correct uneven light fields)

Modified

  • Updated CaImAn repository
  • Updated CellSort repository and fully integrated it within NETCAL (GUI pending)
  • Schmitt trigger now admits point to point thresholds
  • Substantial speed up on automatic ROI detection (quick_dev and split modules)

Fixed

  • Renamed brick montage to avoid conflicts
  • Some files where not being copied when importing/cloning exsting .EXP files
  • Some statistics were not being correctly exported

Removed

  • Several consistency checks on trace fixer

Known Bugs

  • There is an ongoing bug where several pipeline functions are not being correctly loaded from their json files

7.0.0 - 2017-11-12

Added

  • Open beta release

Known Bugs

  • There is an ongoing bug where several pipeline functions are not being correctly loaded from their json files

[6.3.0] - 2017-11-03