compute_pixel_cluster_channel_avg kills the kernel

[ngreenwald]

Please refer to our FAQ and look at our known issues before opening a bug report.

Describe the bug
When running `apply_pixel_meta_cluster_remapping', the kernel often dies when re-computing the channel averages. Given that this is being computed across all of the FOVs, is this something that will be prohibitive for analyzing larger cohorts? It won't matter how much we subset if this calculation needs to be computed from all the underlying data?

[alex-l-kong]

Depending on what we find, this may be related to #518.

[alex-l-kong]

@ngreenwald I'll be doing some memprofing on this one to see if there's a bottleneck in the way this is being computed.

[alex-l-kong]

@ngreenwald yeah, we do have a massive memory problem on our hand. I used the pympler memory profiler to check out each iteration of compute_pixel_cluster_channel_avg and here's what we're getting on Candace's dataset about 25 FOVs in:

It's understandable that the amount of pandas.DataFrame memory goes up per iteration: we are appending after all. However, there should not be this much memory usage: for a 10x10 SOM and 30 FOVs, there should only be at most 3000 rows of data, and we don't have any large blob objects or the like either.

The problem may be we need to read in the full pixel data frame for each FOV to compute the sums and counts, and there are issues clearing the memory per iteration.

Regardless of if this should crash a 30+GB Docker instance or not, this is a lot more memory than this function is expected to use. I'll be taking a deeper dive into compute_pixel_cluster_channel_avg.

[alex-l-kong]

@ngreenwald any status updates on these memory-related issues? Now that we've updated FlowSOM to not run MapDataToCodes when SOM is called, I think we can close this one as they solved the internal memory issues.