ENH: Return array scalars for skew and kurtosis

[Kai-Striega]

This PR returns array scalars for the mstats version of skew and kurtosis and adds a test that the returned value is indeed a scalar value.

Reference issue

fixes #12548

What does this implement/fix?

The masked skew and kurtosis function have branches where the resultant masked array is not converted to an array scalar leading to a masked array being returned as below:

>>> kurtosis([1,5,4,6,nan], fisher=False, nan_policy="omit")
masked_array(data=2.,  #  Correct value but a masked array
             mask=False,
       fill_value=1e+20)

This PR adds an addition step so that, in the case of a 1d array, a scalar value is returned.

>>> kurtosis([1,5,4,6,nan], fisher=False, nan_policy="omit")
2.0

Additional information

I'm not sure whether this is the correct approach; and haven't worked in the mstats modules enough to know if there is a better approach. I'd appreciate if someone with more experience could give their opinion on whether there is a better way to handle it.

[Kai-Striega]

@WarrenWeckesser I was wondering if you could take a quick look at this? The solution was to changevals -> vals + 0. As vals is a masked array it will convert to a scalar if the number of dimensions is 0. It's passing (the failing tests are due to memory errors) but I'm not sure if it had been better to just use an if statement.

[sethtroisi]

should the docstring at L2280 should be updated?

    Returns
    -------
    skewness : ndarray
        The skewness of values along an axis, returning 0 where all values are
        equal.

it seems in this case it's returning a scalar instead of a ndarray

[sethtroisi]

In [30]: skew([[1,5,4],[5,1,3],[4,7,nan]], nan_policy="omit") + 0   
Out[30]: 
masked_array(data=[-0.52800498, -0.38180177,  0.        ],
             mask=False,
       fill_value=1e+20)

Will the new code handle this case? maybe vals.data is what we actually want?

[Kai-Striega]

Edit Yes I think the docstring should be updated, but let's get the rest of the discussion done before I make any more changes.

I'm not sure what we want in that case. To me the problem wasn't from the masked array - it is that an array is being returned at all when there is only a single dimension. Like other ufuncs. I don't have a strong preference either way but I'd like to be consistent

[sethtroisi]

vals.data seems to correctly be a scalar vs ndarray. spitballing it looks like

vals = vals.data
if fisher:
    return vals - 3
else:
    return vals

Seems to work

[Kai-Striega]

I've just run the above and it still gives a float value for fisher and an array (not a masked array) for the regular version.

>>> import numpy as np
>>> from scipy.stats import kurtosis
>>> kurtosis([1, 5, 4, 6, np.nan], nan_policy="omit")
-1.0
>>> kurtosis([1, 5, 4, 6, np.nan], fisher=False, nan_policy="omit")
array(2.)

Would we expect both results to be a float instead? i.e.

vals = vals.data
if fisher:
    return vals - 3
else:
    return vals + 0

[rlucas7]

Suggested change

	s = stats.skew([1, 5, 4, 6, np.nan], nan_policy='omit')
	s = stats.skew([1, 5, 4, 6, np.nan], fisher=True, nan_policy='omit')

[rlucas7]

This suggestion is primarily to make explicit that both options for fisher arg are tested

[Kai-Striega]

@rlucas7 sorry about taking so long to reply - I've been away for a long time, but I'm trying to get more involved again now. stats.skew doesn't actually accept a fisher kwarg, only kurtosis. Perhaps this should be changed but that should be another PR.

[rlucas7]

@Kai-Striega thanks for this changeset

This looks like it fixes the issue and makes masked arrays consistent in shape.

I left 1 suggestion inline. The rest of the changes seem ok.

Also can you add something to the docstring to indicate that the skewness is -3, 0 depending on fisher Truthyness when n=1.

Something like:

When `a` has 1 record skew returns a -3 or 0 if fisher is true or false, respectively.
When `a` is empty skew returns a `nan`. 

and similarly for kurtosis?

Other than those small docstring changes this seems to be good.

[rlucas7]

This suggestion is primarily to make explicit that both options for fisher arg are tested

[mdhaber]

@Kai-Striega I don't think this is needed now that the decorator has been applied to skew and kurtosis.

Now in main:

import numpy as np
from scipy.stats import kurtosis
kurtosis([1, 5, 4, 6, np.nan], fisher=False, nan_policy="omit")  # 2.0

Please re-open if you think the masked versions themselves should be fixed, but I think we're working toward deprecating mstats.