Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic arithmetic methods #541

Merged
merged 15 commits into from Dec 19, 2022
Merged

Conversation

wtbarnes
Copy link
Member

@wtbarnes wtbarnes commented Aug 26, 2022

This PR adds basic support for arithmetic methods on NDCube: add, subtract, multiply, divide. It explicitly does not support operations between NDCube objects (or more precisely anything that is an NDData instance).

There was also some discussion about to what extent we can support operations with Dask arrays. There is no issue if our NDCube is already backed by a Dask array. However, if the other object that we are performing the operation with is a Dask array, it will be eagerly evaluated because of the use of Quantity. This is probably not necessary and we should be checking possible other operands in a slightly different way.

TODOs

  • Fix the tests
  • Gallery example
  • Changelog
  • Create issues for supporting pow and rtruediv
  • Create "downstream" issues on sunpy
  • Test for NotImplementedError
  • Scale uncertainties as part of multiplication

@pep8speaks
Copy link

Hello @wtbarnes! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 817:36: W291 trailing whitespace

Line 869:80: E501 line too long (86 > 79 characters)
Line 894:80: E501 line too long (86 > 79 characters)
Line 910:80: E501 line too long (86 > 79 characters)
Line 928:80: E501 line too long (90 > 79 characters)

Line 19:80: E501 line too long (93 > 79 characters)
Line 20:80: E501 line too long (96 > 79 characters)
Line 25:80: E501 line too long (97 > 79 characters)
Line 26:80: E501 line too long (96 > 79 characters)
Line 27:80: E501 line too long (96 > 79 characters)
Line 28:80: E501 line too long (93 > 79 characters)

@DanRyanIrish
Copy link
Member

DanRyanIrish commented Sep 2, 2022

@wtbarnes: Summarising our discussion on the sunpy call:

  • Current implementation of casting input to a Quantity forces input to be read into memory and not supportive of dask.
  • Therefore, we should start by only supporting Quantities and non-unit objects, e.g. floats, arrays, etc. So the new implementation should be as follows:
    • Check if input has a unit using hasatrr(input, "unit")
    • If not, apply operation assuming input is a zero-uncertainty unitless scalar/array
    • If it does have a unit, check if input is a Quantity using isinstance.
    • If not, raise NotImplementedError
    • If so, confirm the unit is compatiable with the NDCube's unit. If not, raise Error.
    • Work out scaling factor between units.
    • Apply operation to data with input value * scaling factor. This should mean that the resulting data is still in the units of the original NDCube.
  • In the future we can work out ways of being less restrictive the types we support. But this should account for many users' needs without painting ourselves into a corner and so is worth implementing.
  • In particular, we should be able to support operations with NDData objects so long as they don't have a WCS, i.e. obj.wcs is None. See attempt to implement error propagation as part of Implement Superpixel on NDCube #450.

Feel free to push back if you feel this doesn't represent what we agreed.

@DanRyanIrish DanRyanIrish added this to the 2.1 milestone Sep 11, 2022
@DanRyanIrish
Copy link
Member

Any movement or planned movement on this @wtbarnes?

@wtbarnes
Copy link
Member Author

Sorry I have not had any time to revisit this since the discussion you summarized above. I should have some time this week or next to make the refactor suggested in the above comment.

@wtbarnes wtbarnes marked this pull request as ready for review September 22, 2022 05:17
@wtbarnes
Copy link
Member Author

The logic here is not very clean and could probably be refactored a bit to avoid some repetition, but I believe this accomplishes what is laid out above. Currently, I'm not explicitly excluding NDCube or NDData instances, but they're implicitly caught by the "has-unit-but-is-not-a-quantity" case.

Copy link
Member

@DanRyanIrish DanRyanIrish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great @wtbarnes. I haven't reviewed the tests but have left a few comments and one discussion point (about NDCube vs NDCubeBase). This PR isn't the end of the story but is a big leap forward and I believe isn't far off now.

The way you've filtered out the NDCube and NDData instances is a good way to go.

ndcube/utils/decorators.py Outdated Show resolved Hide resolved
changelog/541.feature.rst Outdated Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/tests/test_ndcube.py Show resolved Hide resolved
ndcube/ndcube.py Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/ndcube.py Show resolved Hide resolved
Cadair
Cadair previously requested changes Sep 26, 2022
Copy link
Member

@Cadair Cadair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still need to review this properly, but I want to keep the behaviour of NDCube and Map the same (in all things) at the moment, so if we are going to deviate from map for NotImplemeted vs NotImplementedError then I would want a PR open to sunpy core with the same change before we merge this one.

@ayshih
Copy link
Member

ayshih commented Sep 26, 2022

To be clear, the deviation from Map is not NotImplemented versus NotImplementedError, because Map does not return either for the situation under discussion. If there are uncertainties on the data, Map doesn't care at all, and in fact discards any uncertainties in its arithmetic operations.

ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
@DanRyanIrish
Copy link
Member

DanRyanIrish commented Dec 1, 2022

After discussion with @ayshih we identified the following remaining issues.

  • In fact, unsupported type combinations for the operations should return NotImplemented, not NotImplementedError. NotImplemented allows other types to override the operators to handle NDCubes which we should allow while NotImplementedError does not. Moreover, this would not require a fix in sunpy to maintain alignment with Map.
  • A point to highlight that does not require a change to this PR: There is divergence between this PR and Map in that M̀ap always drops uncertainties when any operation is applied. This PR does not do that. This is a shortcoming of Map that would be addressed by rebasing onto NDCube. Map suddenly not dropping uncertainties should cause no issues as any code written for Maps without uncertainties should not break if there are uncertainties.

I will leave a few code suggestions in inline comments that should address the above issues.

ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
ndcube/ndcube.py Show resolved Hide resolved
ndcube/ndcube.py Outdated Show resolved Hide resolved
@wtbarnes
Copy link
Member Author

Just one minor correction in a comment, but otherwise I think this is ready to go

Co-authored-by: Will Barnes <will.t.barnes@gmail.com>
@DanRyanIrish DanRyanIrish dismissed Cadair’s stale review December 19, 2022 10:57

Issues raised in review have been resolved

@DanRyanIrish DanRyanIrish merged commit d7112bc into sunpy:main Dec 19, 2022
@DanRyanIrish
Copy link
Member

Thanks so much @wtbarnes for this!!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants