Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-39733: Specify nan_policy="omit" in model fit #268

Merged
merged 1 commit into from Jul 12, 2023
Merged

DM-39733: Specify nan_policy="omit" in model fit #268

merged 1 commit into from Jul 12, 2023

Conversation

timj
Copy link
Member

@timj timj commented Jun 20, 2023

This was deprecated in lmfit 0.9.8 and removed in 2019.

lmfit/lmfit-py#565

@timj
Copy link
Member Author

timj commented Jun 20, 2023

The tests currently assume that nan_policy="raise".

@erykoff
Copy link
Contributor

erykoff commented Jun 23, 2023

The problem is that nan_policy="omit" doesn't work with the fancy way we are using lmfit with multiple arrays and indices. I believe the right solution is to do what I did here, and manually filter out any infinite values first and set their weights to 0. (This is basically what lmfit does prior to any fitting, but using masks in a way that appears to be better tested with pandas than numpy). But I would like @isullivan to confirm that I haven't done anything crazy here.

@erykoff erykoff requested a review from isullivan June 23, 2023 01:12
isullivan
isullivan approved these changes Jul 11, 2023
View reviewed changes
Copy link
Contributor

@isullivan isullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK, with a couple of minor suggestions.

python/lsst/ip/diffim/dipoleFitTask.py Outdated
@@ -748,6 +750,11 @@ def dipoleModelFunctor(x, flux, xcenPos, ycenPos, xcenNeg, ycenNeg, fluxNeg=None
if np.any(~mask):
weights[~mask] = 0.

# Filter out any nans, and make the weights 0.
nans = (~np.isfinite(z) | ~np.isfinite(weights))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are only worried about NaNs, I think np.isnan(z) | np.isnan(weights) would be more clear.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess infinities could be legal? I've changed it.

python/lsst/ip/diffim/dipoleFitTask.py Outdated
@@ -748,6 +750,11 @@ def dipoleModelFunctor(x, flux, xcenPos, ycenPos, xcenNeg, ycenNeg, fluxNeg=None
if np.any(~mask):
weights[~mask] = 0.

# Filter out any nans, and make the weights 0.
nans = (~np.isfinite(z) | ~np.isfinite(weights))
z[nans] = 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try to avoid setting image values to zero, in case there is a non-zero offset. Perhaps z[nans] = np.nanmedian(z)

@erykoff
Manually filter nans prior to fitting.
fac7b07 
The lmfit nan_policy needs all the variables to be aligned, but here we are
fitting with the "x" grid as the independent variable, but "z" has a different
shape.  But the nan_policy="omit" option in lmfit simply does a filtering prior
to doing any fit, so here we zero out any nans and ensure the weights are 0 for
these values.
@erykoff erykoff merged commit a500d10 into main Jul 12, 2023
2 checks passed
@erykoff erykoff deleted the tickets/DM-39733 branch July 12, 2023 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erykoff erykoff erykoff left review comments

@isullivan isullivan isullivan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@timj @erykoff @isullivan