Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fitting with data including y-error #80

Open
anir92 opened this issue Dec 5, 2020 · 5 comments
Open

Fitting with data including y-error #80

anir92 opened this issue Dec 5, 2020 · 5 comments

Comments

@anir92
Copy link

anir92 commented Dec 5, 2020

My data involves y-error. How do I incorporate the error in y values in the fitting? Should I take --> weight = 1/y_error ?
The problem arises when my data set contains few y values and corresponding y_error values equal to zero. If y_error = 0, then corresponding weight is coming to be infinity. How to fit the data in this case ?

@cjekel
Copy link
Owner

cjekel commented Dec 6, 2020

How are you getting those errors where you have some data with and without error?

You could just do a very large weight to the values which have 0 error. You could play with a couple different values to see if it effects the results. I would say a couple orders of magnitude larger than the other errors

Linking weighted least squares for reference.

@anir92
Copy link
Author

anir92 commented Dec 8, 2020

Thanks for the suggestion. I have looked deep into the data which actually comes from a measurement of a CCD detecting photons. I now understand that the detector was OFF during those instances (so both y (counts) =0 and y_err=0). So, I am either going to discard those data points or rebin the data set.

@anir92
Copy link
Author

anir92 commented Dec 8, 2020

I have another query. Can you please inform how "Standard_error" is calculating the error on the breakpoints ? The description says "Calculate the standard errors for each beta parameter determined from the piecewise linear fit". Thanks in advance.

@cjekel
Copy link
Owner

cjekel commented Dec 9, 2020

https://jekel.me/piecewise_linear_fit_py/pwlf.html#pwlf.PiecewiseLinFit.standard_errors

One approximation that is commonly used for non-linear models is the 'delta method' or finite differences. I basically evaluate how sensitive the model is to the breakpoint locations. It is technically a first order taylor series expansion of the non-linear model.

This is the the paper that details the method https://mae.ufl.edu/nkim/Papers/paper54.pdf

Here is another reference on the delta method https://stats.idre.ucla.edu/r/faq/how-can-i-estimate-the-standard-error-of-transformed-regression-parameters-in-r-using-the-delta-method/

@cjekel
Copy link
Owner

cjekel commented Dec 9, 2020

I guess I should add that the delta method is only used when method='non-linear'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants