Speed up fitting in HyperSpy

[francisco-dlp]

Fitting in HyperSpy is slow, or at least it is not as fast as it could be. For multidimensional datasets parallesation can help in some cases and is in the works (see #242). Another complementary approach is to speed up function evaluation. One way to do this would be to use numexpr. Any other ideas?

[dnjohnstone]

This is a particular problem I'm facing right now with trying to fit many 2D Gaussians in a stack of images... i.e. ~40 2D Gaussians in ~5000 images.

I have some code written in C that parallelises over the individual peaks and is really a lot faster than anything I can write in HyperSpy right now. I guess there are two points here, first we could perhaps achieve a speed up by implementing some parts of the optimisation in C but I guess this might go against some of the wider HyperSpy principles?

Secondly, an option to parallelise over components in the model may be of particular benefit in cases (like mine, and often atomic resolution images) where the peaks in each spectrum/image are quite well spaced. In short fitting 4 variables 40 times is probably a better bet than trying to fit 160 in one go.

Any thoughts or points that need doing would be appreciated to guide doing that!

Any thoughts or

[francisco-dlp]

#573 partially solves this issue as, when using numexpr, the speed should be very close to C code. However a new Expression2D component will be needed for Model2D. Hopefully it'll require only minor changes.

Regarding parallelization, if you mention that you have ~5000 images, then I think that running the fit of each image in parallel (as implemented in #242) instead of parallelizing the fit of the individual images should provide a similar or better boost in speed without compromising accuracy. Actually, I might be wrong about this but, if for parallelizing the fit on an individual image you assume that the contribution of all other peaks is negligible, then you may not need to fit at all, as just estimating the parameters of the 2D Gaussians analytically might be good enough for the purpose.

[tjof2]

How much of this is fixed/solved with #1101?

[francisco-dlp]

#573, #1101 and #1321 should fully address this.

[tjof2]

#573, #1101 and #1321 should fully address this.

• New Expression component #573 and NEW: SAMFire #1101 are merged
• NEW: Linear fitting #1321 was closed in favour of New: Linear fitting 2.0 #1462, which is still open Linear fitting #2422