Nan Changes

[SteveAIS]

Adding changes to support nanmean, nammin, nanmax, etc.

[boazmohar]

@stevevarner Hi,
I am not sure this will accomplish what I think you intend. There is a difference in how numpy treats NaN in + and - operations and with np.nanmean. see https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst

For example:

import numpy as np
a = np.array([1, 2, 3, float('NaN'), 5, 6])
b = np.array([1, 2, float('NaN'), 5, 6, 7])
c = numpy.stack((a,b))
numpy.nanmean(c, axis=0)
Out[1]: array([ 1. ,  2. ,  3. ,  5. ,  5.5,  6.5])
# versus:
(a+b)/2
Out[2]: array([ 1. ,  2. ,  nan,  nan,  5.5,  6.5])

Am I missing something?

[SteveAIS]

We are wanting it to disregard all NaN values in the mean.

[boazmohar]

That would mean that thunder images and series .nanmean would behave differently in local and spark modes.
Right?

[techchrj]

No....they would behave the same. numpy.nanmean ignores the NaN values just as what was implemented.

[boazmohar]

Hi,
I am trying to add tests to bolt to check your new functions (As in https://github.com/bolt-project/bolt/blob/master/test/spark/test_spark_functional.py#L66)

from numpy import arange, float32
from bolt import array

x = arange(2*3*4).reshape(2, 3, 4).astype(float32)
b = array(x, sc, axis=(0,))
b.nanmean()

gives the error:

 File "/Users/moharb/Documents/Repos/bolt/bolt/spark/statcounter.py", line 95, in merge
    delta[isnan(value)] = 0
TypeError: 'numpy.float64' object does not support item assignment

Any idea why?

Thanks,
Boaz

[freeman-lab]

Had an offline chat with @boazmohar about some of this. Given that the goal of bolt is to stay as close to the numpy API as possible, we don't think we should add new methods here like nancount and nansampleVariance etc., but keeping nanmax nanmin nansum nanmean nanstd and nanvariance would be great.

For implementation, nanmax nanmin and nansum should probably be implemented with a reduce the way max and min and sum are now, and only nanmean nanstd and nanvariance should be done with the stats counter.

[boazmohar]

@freeman-lab @jwittenbach looking over code, there is a python 2/3 problem with specifying longs for counters in stats counter. http://python3porting.com/differences.html (see long)
I think it makes sense to specify long in python 2, would this code be acceptable:

import sys
if sys.version_info < (3,):
    long = int