corrects randomization implementation in spatial_analysis_utils

[ackagel]

Purpose

Corrects the randomization strategy in the spatial analysis code.

Implementation

Random sampling happens without replacement on each individual bootstrap iteration. Previously, sampling with replacement was performed to bootstrap in one iteration per comparison. It was thought this was faster and on average the same as the former strategy, but as it turns out, neither is the case.

Quick benchmarks for old/new strategy:

Old implementation

bootstrap_num = 5

bootstrap_num = 10

New implementation

bootstrap_num = 5

bootstrap_num = 10

[ackagel]

@cliu72

Tagged you for review here since we had talked about correcting the randomization strategy earlier (as in months ago, but just imagine it was sooner....)

[ngreenwald]

If this is the way that it was originally done and it's a tiny bit faster, makes sense to revert back

[alex-l-kong]

Looks good to me, but how much of a slowdown are we looking at here? I remember the tests would often time out on a smaller dataset using this method.

[cliu72]

Looks great!

[ackagel]

@alex-l-kong the "old old" version was the same algorithm as this implementation, but it recomputed the binarized distance matrix on each bootstrap iteration, along with some other needless 'recomputes'.

Awhile ago, I made the mistake of correcting that while also changing the algorithm. I thought the algo change made it faster, while preserving the correct randomization strategy. Both of those conclusions were incorrect, and it turns out only the removal of the 'recomputes' led to the speed up, while the algorithm change actually slowed it down due to a memory bottleneck.

In other words, for n=num_cells this corrected implementation is O(n^2) in memory, and O(r) in 'python loops', while the previous method is O(r * n^2) in memory, and O(1) in 'python loops'. Because of the high number of cells, the memory allocation for the previous method ends up being slower than the O(r) 'python loops' in the corrected method.

tl;dr past Adam did bad and should feel bad. also python loops are less evil than excessive memory allocations.