Allow user to set an explicit skip flag for custom regionprops and marker extraction functionality in notebook 1

[alex-l-kong]

What is the purpose of this PR?

Closes #875. Since PyTorch/Tensorflow optimizations to running these extractions will require more changes to the Docker than anticipated, for now we add the functionality to skip these computations.

How did you implement your changes

A flag, skip_extraction, can be set for generate_cell_table, which gets propagated to the internal extractions, allowing them to avoid running these expensive processes if so desired by the user.

Remaining issues

We do need to maintain the 'cell_size', 'label', 'coords', and 'centroid' features from regionprops. The first 3 are needed for indexing and 'centroid-0' and 'centroid-1' (generated from 'centroid') is needed for spatial LDA. This means there will still be the bottleneck of running regionprops_table, since I believe the way it works is it generates all the regionprops features and then subsets on them.

Please comment if there are additional base features we should keep regardless of if skip_extraction is set.

[alex-l-kong]

Tested with the skip_extraction flag and it runs about 70-80% times faster. The number of cells to assign is an unavoidable bottleneck, but without signal extraction on the example dataset it takes on our worst FOVs 40s (it's about 2-3x slower without skip_extraction).

[ngreenwald]

Skipping the extraction step makes sense for testing purposes to see where the bottleneck is, but for in terms of actual user setting, isn't relevant. The only reason to generate the cell table is to have the extracted intensity values.

The parameter that would be helpful for the user is the option to skip the custom regionprops functions, which are quite slow.

My understanding is that all the features in REGIONPROPS_BASE don't contribute materially to the runtime. Is that correct? If so, then having a skip flag which sets REGIONPROPS_SINGLE_COMP and REGIONPROPS_MULTI_COMP to None would be sufficient. If it's the case that certain features in REGIONPROPS_BASE are also slow to compute, then the skip flag should also remove any of those slow ones.

[alex-l-kong]

@ngreenwald fixed up the pipeline to only remove regionprops_single_comp and regionprops_multi_comp and leave marker extraction intact.

By removing certain regionprops base calculations (all but label, centroid, and coord), we can trim the single compartment props retrieval time by several orders (up to 10x for our more expensive FOVs). So I'd say the skip_extraction step should also set regionprops_base to just these features in addition to setting regionprops_single_comp and regionprops_multi_comp to [].

[ngreenwald]

Got it. And what if you remove centroid and coord as coord as well?

[alex-l-kong]

Got it. And what if you remove centroid and coord as coord as well?

No significant difference, a couple tenths of a second difference at most.

[ngreenwald]

great, sounds good

[ngreenwald]

The documentation needs to be updated to reflect the behavior of skip extraction. Will you also add a flag in notebook1 with a description where the user can toggle this on or off?

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ngreenwald]

Looks good!