Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow numpy shape functions on ShapedNDArray subclasses #10337

Merged
merged 12 commits into from Jun 6, 2020

Conversation

mhvk
Copy link
Contributor

@mhvk mhvk commented May 9, 2020

Partially addresses #8610, though for now only the simplest numpy functions are supported on all ShapedNDArray subclasses (representations, coordinates, and times):

        np.moveaxis, np.rollaxis,
        np.atleast_1d, np.atleast_2d, np.atleast_3d, np.expand_dims,
        np.broadcast_to, np.flip, np.fliplr, np.flipud, np.rot90,
        np.roll, np.delete,

For instance,

t = Time(np.arange(50000, 50005), format='mjd')
np.roll(t, 2)
# <Time object: scale='utc' format='mjd' value=[50003. 50004. 50000. 50001. 50002.]>

EDIT: while testing, noticed that some functions already worked; e.g., np.moveaxis and np.rollaxis (and np.where though it returns an object ndarray).

@mhvk mhvk added this to the v4.2 milestone May 9, 2020
@mhvk mhvk added Enhancement and removed wcs labels May 9, 2020
@mhvk
Copy link
Contributor Author

mhvk commented May 9, 2020

Wow, astrobot is good with adding labels! Note that I also moved ShapedLikeNDArray out of misc.py, as I felt it made much more sense to have it in its own module (as it is becoming quite large, and probably could use its own documentation). If need be, I can separate that out.

@mhvk mhvk requested review from astrofrog, eteq and taldcroft and removed request for astrofrog May 9, 2020 21:37
@mhvk mhvk force-pushed the shaped-array-function branch 2 times, most recently from 8b1d13c to b18e906 Compare May 10, 2020 20:56
@mhvk mhvk force-pushed the shaped-array-function branch 2 times, most recently from d65d0b0 to c9dff84 Compare June 2, 2020 00:31
@mhvk mhvk requested a review from adrn June 2, 2020 00:35
@mhvk
Copy link
Contributor Author

mhvk commented Jun 2, 2020

Looking at the PR again, I felt one commit made it harder to review than necessary, so I moved that to its own PR (#10425). For this one, the only real change is in ShapedLikeNDArray - the rest is all tests to make sure it actually works. It complements the array-like shape methods provided with support for numpy functions that affect the shape.

One thing to do in follow-up is to properly implement np.where - right now, I kept it as it was, which means it works but just returns an object array.

@eteq, @taldcroft, @astrofrog - review would be most appreciated, it is not as bad as it looks!

Copy link
Member

@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a first pass, looks generally good. I don't see any tests of the new methods for Time, so those should be added unless I'm missing something.

astropy/time/tests/test_methods.py Show resolved Hide resolved
from contextlib import contextmanager
from collections import defaultdict, OrderedDict

from astropy.utils.decorators import deprecated
# For backwards compatibility, import the shape-related stuff.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about these symbol definitions for back-compatibility? Are there other uses of ShapedLikeNDArray outside astropy core?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether it is used outside the core, and even if it was, it would hopefully have been imported from astropy.utils directly, though it is listed in the API at https://docs.astropy.org/en/latest/utils/index.html#id14. Maybe it makes sense to not copy it over for now, and then possibly re-introduce it if it causes any problems?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But at least I shouldn't leave commented-out code - I've now simply remove it.

CHANGES.rst Show resolved Hide resolved
astropy/utils/shapes.py Show resolved Hide resolved
@mhvk
Copy link
Contributor Author

mhvk commented Jun 2, 2020

OK, comments addressed! I added more tests using Time - they just mimic what was added in test_representation_methods, but it makes sense to have them for the different sub-packages.

@mhvk
Copy link
Contributor Author

mhvk commented Jun 2, 2020

Note: numpy-dev travis failure is unrelated; see #10432

Copy link
Member

@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my perspective. For the coordinates stuff I have given it a "looks about right" level of review w/r/t tests and basic functionality.

@astrofrog astrofrog removed their request for review June 5, 2020 13:37
@mhvk
Copy link
Contributor Author

mhvk commented Jun 5, 2020

@taldcroft - thanks! I realized I had not tested SkyCoord and also not added documentation, so decided I should remedy that. The latter is very short, let me know if you want to have a look (last commit).

Copy link
Member

@adrn adrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't take an extremely close look at the tests, but did look closely at the code and text, and this LGTM (modulo one very minor docs suggestion)!

`~numpy.ndarray` instances. Similarly, on ``numpy`` version 1.17 or later,
corresponding functions as well as others that affect the shape, such as
`~numpy.at_least1d` and `~numpy.rollaxis`, work as expected.
(The relevant functions have to be explicitly enabled; let us know if a
Copy link
Member

@adrn adrn Jun 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(The relevant functions have to be explicitly enabled; let us know if a
(The relevant functions have to be explicitly enabled in the ``astropy`` source code; let us know if a

To clarify that the user doesn't need to enable anything: something has to be changed in the astropy code itself!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I'll do it in all the places where I added this... (and fix the docs build at the same time)

@mhvk
Copy link
Contributor Author

mhvk commented Jun 6, 2020

Thanks, @taldcroft and @adrn for the reviews. Finally got all the documentation tests to pass (really need an .rst checker for emacs...), and since you both signed off on complementary parts, I'll merge.

@mhvk mhvk merged commit 2023292 into astropy:master Jun 6, 2020
@mhvk mhvk deleted the shaped-array-function branch June 6, 2020 20:39
@pllim
Copy link
Member

pllim commented Jun 8, 2020

Does the error I see in #10471 look familiar to you?

@pllim
Copy link
Member

pllim commented Jun 8, 2020

Hmm, looks like the tests in TestManipulation were skipped on Windows because they are marked as remote data. That would explain why CI was green and that maybe I am not crazy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants