You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmark scattering mask data vs transmitting pickled mask factories #238
Pickled mask factories can be very slow, depending on their complexity #232.
We should compare scattering mask data with sending factories for typical examples as soon as we have the cluster running and a performance benchmarking suite #198.
I think an important aspect of this is actually moving the mask container into a per-job-overhead from the current per-task-overhead. Then the question becomes: is it still significant, compared to the whole job? We have to weigh supporting jobs with many masks and datasets with large frames versus being friendly to the API-user with regards to pickle overhead.
From my small micro-benchmark in #235, with ~30ms per frame, I think sending the function over still has benefits, especially for applications like our airy masks example, which has ~126 masks.
Sure! I guess a first step would be to understand the ApplyMasksUDF, understand the different types of masks, and understand the difference between sending the mask data with the task vs. sending a factory function. A good practical application to look at is the radial fourier analysis - this possibly generates large stacks of masks, which can be configured to be sparse or dense.
From that, you can think about a good benchmark setup and see how the two options compare.
Activity
sk1p commentedon Jan 8, 2019
I think an important aspect of this is actually moving the mask container into a per-job-overhead from the current per-task-overhead. Then the question becomes: is it still significant, compared to the whole job? We have to weigh supporting jobs with many masks and datasets with large frames versus being friendly to the API-user with regards to pickle overhead.
From my small micro-benchmark in #235, with ~30ms per frame, I think sending the function over still has benefits, especially for applications like our airy masks example, which has ~126 masks.
SahanWickramage commentedon Apr 9, 2020
I would like to contribute to this issue.
sk1p commentedon Apr 10, 2020
Sure! I guess a first step would be to understand the
ApplyMasksUDF, understand the different types of masks, and understand the difference between sending the mask data with the task vs. sending a factory function. A good practical application to look at is the radial fourier analysis - this possibly generates large stacks of masks, which can be configured to be sparse or dense.From that, you can think about a good benchmark setup and see how the two options compare.