More MaskedNDArray failures with Astropy 5.0

[dhomeier]

Problem description

Found a couple more test failures with astropy 5.0rc2 that I missed previously (likely were present with 5.0rc1 as well, but I did not test with remote-data). All passing with 4.3.1 in the same environment; I have not looked into details, but they all seem in some way related to handling MaskedNDArray (rather than unmasked NDData or np.ma.array) now.

The first test e.g. could be fixed by substituting
assert np.isnan(getattr(lc.flux, 'data', lc.flux)).any() # ensure the test data has nan in flux

Example

poetry run pytest --remote-data

______________________________________________________________________ test_preprocess_lc ______________________________________________________________________

    @pytest.mark.remote_data
    def test_preprocess_lc():
        """Test to ensure the lightcurve is pre-processed before applying BLS for correctness and consistent output"""
        from lightkurve.interact_bls import _preprocess_lc_for_bls
    
        lc = KeplerLightCurve.read(KEPLER10)
>       assert np.isnan(lc.flux).any()  # ensure the test data has nan in flux
E       assert MaskedNDArray(False)
E        +  where MaskedNDArray(False) = <bound method MaskedNDArray.any of MaskedNDArray([  ———, False, False, ..., False, False, False])>()
E        +    where <bound method MaskedNDArray.any of MaskedNDArray([  ———, False, False, ..., False, False, False])> = MaskedNDArray([  ———, False, False, ..., False, False, False]).any
E        +      where MaskedNDArray([  ———, False, False, ..., False, False, False]) = <ufunc 'isnan'>(<MaskedQuantity [      ———, 575100.6 , 575187.25, ..., 575059.7 ,\n                 575059.3 , 574996.6 ] electron / s>)
E        +        where <ufunc 'isnan'> = np.isnan
E        +        and   <MaskedQuantity [      ———, 575100.6 , 575187.25, ..., 575059.7 ,\n                 575059.3 , 574996.6 ] electron / s> = <KeplerLightCurve length=1011 LABEL="KIC 11904151" QUARTER=4 AUTHOR=Kepler FLUX_ORIGIN=pdcsap_flux>\n       time       ...58  5.7505931e+05 ...  6.4287381e-03  4.3517463e-02\n373.21904601071583  5.7499662e+05 ...  6.4082127e-03  4.3601818e-02.flux

tests/test_interact_bls.py:85: AssertionError
____________________________________________________________________ test_lightcurve_plots _____________________________________________________________________

    @pytest.mark.remote_data
    def test_lightcurve_plots():
        """Sanity check to verify that lightcurve plotting works"""
        for lc in [KeplerLightCurve.read(TABBY_Q8), TessLightCurve.read(TESS_SIM)]:
            lc.plot()
            lc.scatter()
>           lc.errorbar()

tests/test_lightcurve.py:416: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1999: in errorbar
    return self._create_plot(method="errorbar", **kwargs)
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1849: in _create_plot
    ax.errorbar(
/opt/lib/python3.9/site-packages/matplotlib/__init__.py:1412: in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
/opt/lib/python3.9/site-packages/matplotlib/axes/_axes.py:3415: in errorbar
    barcols.append(lines_func(
/opt/lib/python3.9/site-packages/matplotlib/__init__.py:1412: in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
/opt/lib/python3.9/site-packages/matplotlib/axes/_axes.py:1125: in vlines
    x, ymin, ymax = cbook._combine_masks(x, ymin, ymax)
/opt/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:1032: in _combine_masks
    x = safe_masked_invalid(x)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = MaskedNDArray([            ———, 258310.04830265, 258301.8428278 , ...,
               258854.41510582, 258792.36798096, 258795.49073887]), copy = False

    def safe_masked_invalid(x, copy=False):
        x = np.array(x, subok=True, copy=copy)
        if not x.dtype.isnative:
            # If we have already made a copy, do the byteswap in place, else make a
            # copy with the byte order swapped.
            x = x.byteswap(inplace=copy).newbyteorder('N')  # Swap to native order.
        try:
            xm = np.ma.masked_invalid(x, copy=False)
>           xm.shrink_mask()
E           AttributeError: 'MaskedNDArray' object has no attribute 'shrink_mask'

/opt/lib/python3.9/site-packages/matplotlib/cbook/__init__.py:708: AttributeError
___________________________________________________________________ test_lightcurve_scatter ____________________________________________________________________

    @pytest.mark.remote_data
    def test_lightcurve_scatter():
        """Sanity check to verify that lightcurve scatter plotting works"""
        lc = KeplerLightCurve.read(KEPLER10)
>       lc = lc.flatten()

tests/test_lightcurve.py:434: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
    mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = MaskedNDArray([  ———,  True,  True, ...,  True,  True,  True]), ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True,  True,  True, ...,  True,  True,  True]), MaskedNDArray([  ———,  True,  True, ...,  True,  True,  True])), kwargs = {}
out = (array([ True,  True,  True, ...,  True,  True,  True]),), out_unmasked = (array([ True,  True,  True, ...,  True,  True,  True]),), out_mask = None
out_masks = (None,), d = array([ True,  True,  True, ...,  True,  True,  True]), m = None

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        out = kwargs.pop('out', None)
        out_unmasked = None
        out_mask = None
        if out is not None:
            out_unmasked, out_masks = self._get_data_and_masks(*out)
            for d, m in zip(out_unmasked, out_masks):
                if m is None:
                    # TODO: allow writing to unmasked output if nothing is masked?
                    if d is not None:
>                       raise TypeError('cannot write to unmasked output')
E                       TypeError: cannot write to unmasked output

/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError
_______________________________________________________________________ test_cdpp_tabby ________________________________________________________________________

    @pytest.mark.remote_data
    def test_cdpp_tabby():
        """Compare the cdpp noise metric against the pipeline value."""
        lc = KeplerLightCurve.read(TABBY_Q8)
        # Tabby's star shows dips after cadence 1000 which increase the cdpp
        lc2 = LightCurve(time=lc.time[:1000], flux=lc.flux[:1000])
>       assert np.abs(lc2.estimate_cdpp().value - lc.cdpp6_0) < 30

tests/test_lightcurve.py:486: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:1573: in estimate_cdpp
    detrended_lc = self.flatten(
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
    mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = MaskedNDArray([  ———,  True,  True,  True,  True,  True,  True,  True,
                True,  True,  True,  True,  Tru...True,  True,  True,  True,  True,  True,  True,
                True,  True,  True,  True,  True,  True,  True,  True])
ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...rue,  True,  True,  True,  True,  True,  True,
                True,  True,  True,  True,  True,  True,  True,  True]))
kwargs = {}
out = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...rue,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True]),)
out_unmasked = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...rue,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True]),)
out_mask = None, out_masks = (None,)
d = array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,... True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True])
m = None

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        out = kwargs.pop('out', None)
        out_unmasked = None
        out_mask = None
        if out is not None:
            out_unmasked, out_masks = self._get_data_and_masks(*out)
            for d, m in zip(out_unmasked, out_masks):
                if m is None:
                    # TODO: allow writing to unmasked output if nothing is masked?
                    if d is not None:
>                       raise TypeError('cannot write to unmasked output')
E                       TypeError: cannot write to unmasked output

/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError
_____________________________________________________________________ test_regression_346 ______________________________________________________________________

    @pytest.mark.remote_data  # due to test (K2_C08 is remote)
    def test_regression_346():
        """Regression test for https://github.com/lightkurve/lightkurve/issues/346"""
        # This previously triggered an IndexError:
        with warnings.catch_warnings():  # KeplerLightCurveFile is deprecated
            warnings.simplefilter("ignore", LightkurveDeprecationWarning)
            from lightkurve import KeplerLightCurveFile
    
>           KeplerLightCurveFile(
                K2_C08
            ).PDCSAP_FLUX.remove_nans().to_corrector().correct().estimate_cdpp()

tests/test_lightcurve.py:1107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
    ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
               0.72696675, 0.79226021]), MaskedND...68, 1.09270334,
               1.17217677, 1.25490099, 1.3383802 , 1.41997026, 1.50608308,
               1.59696806]))
kwargs = {}
relevant_args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
               0.72696675, 0.79226021]), MaskedND...68, 1.09270334,
               1.17217677, 1.25490099, 1.3383802 , 1.41997026, 1.50608308,
               1.59696806]))

>   ???
E   TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]

<__array_function__ internals>:5: TypeError
_________________________________________________________________ test_CBVCorrector_retrieval __________________________________________________________________

    @pytest.mark.remote_data
    def test_CBVCorrector_retrieval():
        """Tests CBVCorrector by retrieving some sample Kepler/TESS light curves
        and correcting them
        """
    
        # ***
        # A good TESS example of both over- and under-fitting
        # The "over-fitted" curve looks better to the eye, but eyes can be deceiving!
        lc = search_lightcurve(
            "TIC 357126143", mission="tess", author="spoc", sector=10
        ).download(flux_column="sap_flux")
        cbvCorrector = CBVCorrector(lc)
        assert isinstance(cbvCorrector, CBVCorrector)
    
        cbv_type = ["SingleScale", "Spike"]
        cbv_indices = [np.arange(1, 9), "ALL"]
    
        # Gaussian Prior correction
>       lc = cbvCorrector.correct_gaussian_prior(
            cbv_type=cbv_type, cbv_indices=cbv_indices, alpha=1e-2
        )

tests/correctors/test_cbvcorrector.py:452: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/correctors/cbvcorrector.py:278: in correct_gaussian_prior
    self.correct_regressioncorrector(self.design_matrix_collection,
../../opt/lib/python3.9/site-packages/lightkurve/correctors/cbvcorrector.py:501: in correct_regressioncorrector
    return super(CBVCorrector, self).correct(design_matrix_collection, **kwargs)
../../opt/lib/python3.9/site-packages/lightkurve/correctors/regressioncorrector.py:257: in correct
    self.outlier_mask |= sigma_clip(residuals, sigma=sigma).mask
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:835: in sigma_clip
    return sigclip(data, axis=axis, masked=masked,
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:638: in __call__
    return self._sigmaclip_noaxis(data, masked=masked,
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:420: in _sigmaclip_noaxis
    self._compute_bounds(filtered_data, axis=None)
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:302: in _compute_bounds
    self._max_value = self._cenfunc_parsed(data, axis=axis)
/opt/lib/python3.9/site-packages/astropy/stats/sigma_clipping.py:69: in _nanmedian
    return array.__array_wrap__(bottleneck.nanmedian(array, axis=axis))
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:563: in __array_wrap__
    return self._new_view(obj)
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:746: in _new_view
    view.__array_finalize__(self)
/opt/lib/python3.9/site-packages/astropy/units/quantity.py:540: in __array_finalize__
    super_array_finalize(obj)
/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:557: in __array_finalize__
    self._set_mask(getattr(obj, '_mask', False))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <MaskedQuantity 72.30110983 electron / s>, mask = array([False, False, False, ..., False, False, False]), copy = False

    def _set_mask(self, mask, copy=False):
        self_dtype = getattr(self, 'dtype', None)
        mask_dtype = (np.ma.make_mask_descr(self_dtype)
                      if self_dtype and self_dtype.names else np.dtype('?'))
        ma = np.asanyarray(mask, dtype=mask_dtype)
        if ma.shape != self.shape:
            # This will fail (correctly) if not broadcastable.
            self._mask = np.empty(self.shape, dtype=mask_dtype)
>           self._mask[...] = ma
E           ValueError: could not broadcast input array from shape (17692,) into shape ()

/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:226: ValueError
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
Started querying MAST for observations with the exact target_name='357126143'.
MAST found 1 observations. Now querying MAST for the corresponding data products.
MAST found 1 matching data products.
File found in local cache.
---------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
DEBUG    lightkurve.search:search.py:1121 Started querying MAST for observations with the exact target_name='357126143'.
DEBUG    lightkurve.search:search.py:939 MAST found 1 observations. Now querying MAST for the corresponding data products.
DEBUG    lightkurve.search:search.py:1001 MAST found 1 matching data products.
DEBUG    lightkurve.search:search.py:281 File found in local cache.
______________________ test_remote_data[https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits] _______________________

path = 'https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits'

    @pytest.mark.remote_data
    @pytest.mark.parametrize("path", [K2_C08])
    def test_remote_data(path):
        """Can we correct a simple K2 light curve?"""
        lc = KeplerLightCurve.read(path, quality_bitmask=None)
        sff = SFFCorrector(lc.remove_nans())
>       sff.correct(windows=10, bins=5, timescale=0.5)

tests/correctors/test_sffcorrector.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
    ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
               0.72696675, 0.79226021]), MaskedND...35,
               1.13949553, 1.2075622 , 1.27598666, 1.3474425 , 1.41329164,
               1.47335664, 1.53116095]))
kwargs = {}
relevant_args = (MaskedNDArray([0.88009785, 0.99366021, 1.11913694, ..., 0.63925245,
               0.72696675, 0.79226021]), MaskedND...35,
               1.13949553, 1.2075622 , 1.27598666, 1.3474425 , 1.41329164,
               1.47335664, 1.53116095]))

>   ???
E   TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]

<__array_function__ internals>:5: TypeError
____________________________________________________________________ test_sff_nan_centroids ____________________________________________________________________

    @pytest.mark.remote_data
    def test_sff_nan_centroids():
        """Regression test for #827: SFF failed if light curve contained
        NaNs in its `centroid_col` or `centroid_row` columns."""
        lc = search_lightcurve("EPIC 211083408", author="K2").download()
        # This previously raised a ValueError:
>       lc[200:500].remove_nans().to_corrector("sff").correct()

tests/correctors/test_sffcorrector.py:237: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/correctors/sffcorrector.py:173: in correct
    ar[~np.in1d(ar, ar[a:b])] = 0
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (MaskedNDArray([1.74682912, 1.97256351, 2.73136771, 2.73306766, 2.7346456 ,
               2.73633047, 2.73865168, 2.7...33047, 2.73865168, 2.74123547, 2.74447127, 2.74615955,
               2.73309043, 2.71391645, 2.72493336, 2.72557768]))
kwargs = {}
relevant_args = (MaskedNDArray([1.74682912, 1.97256351, 2.73136771, 2.73306766, 2.7346456 ,
               2.73633047, 2.73865168, 2.7...33047, 2.73865168, 2.74123547, 2.74447127, 2.74615955,
               2.73309043, 2.71391645, 2.72493336, 2.72557768]))

>   ???
E   TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<class 'astropy.utils.masked.core.MaskedNDArray'>]

<__array_function__ internals>:5: TypeError
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
Started querying MAST for observations with the exact target_name='ktwo211083408'.
MAST found 1 observations. Now querying MAST for the corresponding data products.
MAST found 1 matching data products.
File found in local cache.
---------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
DEBUG    lightkurve.search:search.py:1121 Started querying MAST for observations with the exact target_name='ktwo211083408'.
DEBUG    lightkurve.search:search.py:939 MAST found 1 observations. Now querying MAST for the corresponding data products.
DEBUG    lightkurve.search:search.py:1001 MAST found 1 matching data products.
DEBUG    lightkurve.search:search.py:281 File found in local cache.
____________________________________________________________________ test_asteroseismology _____________________________________________________________________

    @pytest.mark.remote_data
    def test_asteroseismology():
        datalist = search_lightcurve("KIC11615890")
        data = datalist.download_all()
>       lc = data[0].normalize().flatten()

tests/seismology/test_butler.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../opt/lib/python3.9/site-packages/lightkurve/lightcurve.py:813: in flatten
    mask &= np.isfinite(self.flux)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = MaskedNDArray([ True,  True,  True,  True,  True,  True,  True,  True,
                True,  True,  True,  True,  Tru...rue,  True,  True,  True,
                True,  True,  True,  True,  True,  True,  True,  True,
                True])
ufunc = <ufunc 'bitwise_and'>, method = '__call__'
inputs = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...ue,  True,  True,  True,
                True,  True,  True,  True,  True,  True,  True,  True,
                True]))
kwargs = {}
out = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...rue,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True]),)
out_unmasked = (array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True...rue,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True]),)
out_mask = None, out_masks = (None,)
d = array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,... True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True])
m = None

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        out = kwargs.pop('out', None)
        out_unmasked = None
        out_mask = None
        if out is not None:
            out_unmasked, out_masks = self._get_data_and_masks(*out)
            for d, m in zip(out_unmasked, out_masks):
                if m is None:
                    # TODO: allow writing to unmasked output if nothing is masked?
                    if d is not None:
>                       raise TypeError('cannot write to unmasked output')
E                       TypeError: cannot write to unmasked output

/opt/lib/python3.9/site-packages/astropy/utils/masked/core.py:659: TypeError

[skipping warnings]
=================================================================== short test summary info ====================================================================
FAILED tests/test_interact_bls.py::test_preprocess_lc - assert MaskedNDArray(False)
FAILED tests/test_lightcurve.py::test_lightcurve_plots - AttributeError: 'MaskedNDArray' object has no attribute 'shrink_mask'
FAILED tests/test_lightcurve.py::test_lightcurve_scatter - TypeError: cannot write to unmasked output
FAILED tests/test_lightcurve.py::test_cdpp_tabby - TypeError: cannot write to unmasked output
FAILED tests/test_lightcurve.py::test_regression_346 - TypeError: no implementation found for 'numpy.in1d' on types that implement __array_function__: [<clas...
FAILED tests/correctors/test_cbvcorrector.py::test_CBVCorrector_retrieval - ValueError: could not broadcast input array from shape (17692,) into shape ()
FAILED tests/correctors/test_sffcorrector.py::test_remote_data[https://archive.stsci.edu/missions/k2/lightcurves/c8/220100000/39000/ktwo220139473-c08_llc.fits]
FAILED tests/correctors/test_sffcorrector.py::test_sff_nan_centroids - TypeError: no implementation found for 'numpy.in1d' on types that implement __array_fu...
FAILED tests/seismology/test_butler.py::test_asteroseismology - TypeError: cannot write to unmasked output
======================================== 9 failed, 315 passed, 26 skipped, 4 xfailed, 325 warnings in 435.04s (0:07:15) ========================================

Environment

Platform: macOS-10.14.6-x86_64-i386-64bit
Python 3.9.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
Astropy: 5.0rc2
Matplotlib: 3.5.0rc1
Freetype: 2.6.1
rootdir: ~/git/lightkurve, configfile: pyproject.toml, testpaths: tests, src
plugins: filter-subpackage-0.1.1, arraydiff-0.3, xdist-2.1.0, astropy-header-0.1.2, cov-2.10.1, flaky-3.7.0, remotedata-0.3.2, mock-3.3.1, asdf-2.8.1, forked-1.3.0, openfiles-0.5.0, mpl-0.11, expect-1.1.0, hypothesis-6.14.2, doctestplus-0.10.1, flake8-1.0.6

lightkurve version: 2.0.12dev
Installation method: git main @ 70d1c4c

[barentsen]

Thanks for reporting this @dhomeier! It looks like 9 separate tests failed in total. We'll probably have to debug one by one.

[astrofrog]

I'm just going to loop @mhvk into this issue as I wonder if some of these are issues with the new masked quantity class?

[barentsen]

Several of the failing tests can be narrowed down to the TypeError demonstrated by the example below. It's not immediately obvious to me whether this is a bug or a feature to keep me safe. Thoughts?

>>> import numpy as np
>>> from astropy.utils.masked import Masked
>>> notmasked = np.zeros(3, dtype=bool)
>>> masked = Masked(notmasked)
>>> notmasked &= masked
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/x4/y38zs5fj4szf7xj3354kv74m0000gn/T/ipykernel_23575/1983679778.py in <module>
      3 notmasked = np.zeros(3, dtype=bool)
      4 masked = Masked(notmasked)
----> 5 notmasked &= masked

~/Library/Caches/pypoetry/virtualenvs/lightkurve-HqDeGDfc-py3.8/lib/python3.8/site-packages/astropy/utils/masked/core.py in __array_ufunc__(self, ufunc, method, *inputs, **kwargs)
    657                     # TODO: allow writing to unmasked output if nothing is masked?
    658                     if d is not None:
--> 659                         raise TypeError('cannot write to unmasked output')
    660                 elif out_mask is None:
    661                     out_mask = m

TypeError: cannot write to unmasked output

[dhomeier]

I think it's a decision to err on the safe side since the output could not be written to unmasked without discarding the mask. As the comment states, there is no exception yet even for masked.mask.any() == False.

I think for this to work in its current state, mask in LightCurve.flatten would need to be initialised as Masked(np.ones(len(self.time), dtype=bool)) if astropy is at 5.x/has utils.masked.Masked, perhaps conditional on self.flux being of masked type. This would go beyond just adapting the tests of course, but should resolve 3 of the failures, and will probably be necessary to work safely with the new MaskedNDData.

[dhomeier]

numpy.in1d looks like another ufunc yet lacking support, @mhvk.

[mhvk]

@barentsen - indeed, I was looking and thinking similarly. As @dhomeier notes, Masked prefers not to discard the mask, unlike numpy's MaskedArray, which does discard the mask in those operations. For that piece of code, I think there are two options:

1. Let numpy eat the mask by doing mask[~np.isfinite(self.flux)] = False (this might actually be faster...)
2. Use flux = self.flux.filled(np.nan) and then work with flux from thereon.

p.s. Just to be sure: is the whole np.isfinite call needed? Shouldn't all non-finite values already be masked? If so, one can just use the mask, i.e., something like mask &= self.flux.mask if hasattr(self.flux, 'mask') else np.isfinite(self.flux)

EDIT: I think the mask as a pure ndarray, as you have it in flatten, is most logical, so would try to keep that!

[mhvk]

On np.in1d - oops - see https://github.com/astropy/astropy/blob/b740594dcc9f8aacae0c14a5774e09f0732d1aef/astropy/utils/masked/function_helpers.py#L140-L142

[barentsen]

p.s. Just to be sure: is the whole np.isfinite call needed? Shouldn't all non-finite values already be masked?

Excellent question! Of course most of this code was written in a world where MaskedQuantity didn't exist yet, hence lots of np.isfinite() everywhere. I expect we'll be able to resolve these failing tests with trivial fixes for now however, stay tuned!

[mhvk]

Great! Feel free to ping if there are issues. I'll look into np.in1d, since it would be nice to support that and the other set-like functions -- but I may have to think a bit exactly what a masked entry means for some of the arguments.

[barentsen]

@mhvk I noticed the following issue while debugging one of the tests: astropy/astropy#12481

(It's easy enough to work around this, but it looked like a genuine albeit obscure issue to me.)

[astrofrog]

@mhvk @barentsen - just to check, should I consider the issues here blockers for the astropy 5.0 release or can I go ahead and release and we fix these in 5.0.1?

[barentsen]

@astrofrog I haven't seen anything that should block a 5.0 release from my perspective!