Benchmark scattering mask data vs transmitting pickled mask factories

[uellue]

Pickled mask factories can be very slow, depending on their complexity #232.

We should compare scattering mask data with sending factories for typical examples as soon as we have the cluster running and a performance benchmarking suite #198.

[sk1p]

I think an important aspect of this is actually moving the mask container into a per-job-overhead from the current per-task-overhead. Then the question becomes: is it still significant, compared to the whole job? We have to weigh supporting jobs with many masks and datasets with large frames versus being friendly to the API-user with regards to pickle overhead.

From my small micro-benchmark in #235, with ~30ms per frame, I think sending the function over still has benefits, especially for applications like our airy masks example, which has ~126 masks.

[SahanWickramage]

I would like to contribute to this issue.

[sk1p]

I would like to contribute to this issue.

Sure! I guess a first step would be to understand the ApplyMasksUDF, understand the different types of masks, and understand the difference between sending the mask data with the task vs. sending a factory function. A good practical application to look at is the radial fourier analysis - this possibly generates large stacks of masks, which can be configured to be sparse or dense.

From that, you can think about a good benchmark setup and see how the two options compare.