baseline detection fails if the histogram built by ionic_current_stats contains many zeros

[shadowk29]

If the histogram built by ionic current stats contains a large number of zeros, fitting will fail and give enormous values for perr. The solution is simple: weight the residuals by the y values. You can see an explanation here:

http://math.stackexchange.com/questions/1771660/analytical-solution-to-nonlinear-least-squares-problem

I'm not 100% sure how to fit this intro curve_fit, however. They provide a sigma parameter, which weights data points as 1/sigma^2, so it get y-weighting you would have to provide sigma=1/sqrt(y), which is undefined for y=0. I think the proper solution would be to use scipy.optimize.minimize directly and write a custom residuals function.

[abalijepalli]

Does this become an issue with sparse data? What steps will reproduce this problem?

[shadowk29]

This is mainly an issue with the baseline detection changing in Ticket 69 branch. If the bounds set on minBaseline and maxBaseline are too big, the fitting algorithm fails. I need to work out a way to allow fitting to work even for large bounds since the drift can be significant.

To reproduce it, try building a histogram with an x range of more than 20 standard deviations or so, so that the tail is full of zeros. Fitting will fail and perr will be orders of magnitude larger than popt.

[abalijepalli]

It sounds like when the bounds are large, fitting doesn't converge due to some combination of initial guesses being off and other factors. Also, since the baseline moves a lot, you need to set limit to 0, rather than 0.5 or -0.5.

To get around this, could we weight the histogram with the counts in each bin like you suggested, and simply add a small epsilon value to the weights to prevent divide by zero errors?

[shadowk29]

That might work. I'll try it for my next data set and get back to you.

[shadowk29]

Setting sigma=1/np.sqrt(y+1e-10) seems to work well and allows for very large window sizes while still getting accurate fits.

[abalijepalli]

That's great, we should integrate it into devel-1.0 with a PR.

[shadowk29]

Currently I have it on the ticket69 branch, but the changes there broke a few things when the reorg happened. Let me fix things and I'll sumbit a PR there, and that branch should be ready to merge into devel-1.0 after that.

[shadowk29]

Covered by pull request #83