[Python] Add option to not consider NaN to be null when converting to an integer Arrow type

[asfimport]

Follow-on work to ARROW-2135

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #2750
• GitHub Pull Request #4682
• GitHub PR

_{Note: This issue was originally created as ARROW-2298. Please see the migration documentation for further details.}

[asfimport]

Jakub Okoński:
This is a major blocker for my usecase. Can you please advise how to implement this?

 

Is the conceptual solution as simple as "when we see a nullable integer field in schema, and numpy type for that field is float64, we treat NaNs as nulls and cast everything else to the integer type"?

[asfimport]

Wes McKinney / @wesm:
Can you give an example of the code you want to write and the desired behavior? I don't remember the context of this issue well enough

[asfimport]

Jakub Okoński:
I would like for this to work:

 

In [1]: import pyarrow

In [2]: import pandas as pd

In [3]: df = pd.DataFrame(\{'a': [None, 1, 2, 3, None]})

In [4]: df.dtypes
Out[4]:
a    float64
dtype: object

In [5]: schema = pyarrow.schema([pyarrow.field(name='a', type=pyarrow.int64(), nullable=True)])

In [6]: pyarrow.Table.from_pandas(df, schema=schema, preserve_index=False)
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-6-c779f0c872bb> in <module>()
----> 1 pyarrow.Table.from_pandas(df, schema=schema, preserve_index=False)

...

ArrowInvalid: ('Floating point value truncated', 'Conversion failed for column a with type float64')

[asfimport]

Wes McKinney / @wesm:
Seems reasonable to me.

[asfimport]

Jakub Okoński:
Is the usecase reasonable, or my conceptual solution? Perhaps both? :)

I think that not every 64-bit signed integer has a representation in np.float64, but this caveat is something pandas users would (IMO) be prepared to live with.

[asfimport]

Wes McKinney / @wesm:
When safe=True we should verify that the values are in the representable range of integers. Otherwise we perform an unsafe cast if the value is non-null

[asfimport]

Jakub Okoński:
Sounds great, but I must admit I'm not intimately familiar with floating point numbers.

 

How could one detect problems with this conversion?

 

In [2]: np.array([2**64-1], dtype=np.uint64)
Out[2]: array([18446744073709551615], dtype=uint64)

In [3]: np.array([2**64-1], dtype=np.uint64).astype(np.float64)
Out[3]: array([1.84467441e+19])

In [4]: np.array([2**64-1], dtype=np.uint64).astype(np.float64).astype(np.uint64)
Out[4]: array([0], dtype=uint64)
 

Is this something you can implement quickly? If not, can you give me some pointers so that I can implement it in pyarrow?

[asfimport]

Wes McKinney / @wesm:
Check out these functions: https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/helpers.cc#L350. There's a float64 and float32 version. We would need to implement a version of these within the cast kernels

[asfimport]

Jakub Okoński:
I've pushed a PR addressing this, but it is just my first attempt - here

[asfimport]

Jakub Okoński:
I've looked into this a bit more and I realized that Arrow already supports what I want to do with pandas nulls and overflows, I just need to set safe=False when doing the conversion.

 

Still, I would like to add another parameter to CastOptions, called allow_float_mantissa_overflow. This one would check for out of range values and return errors.

 

This however, won't be very useful unless we expand the safe parameter in the Python API, so that users could specify, for example:

 

pyarrow.Table.from_pandas(df, schema=schema, safe=True, safe_allow_float_mantissa_overflow=False) - this invocation would work with truncated floats, int overflow, truncated timestamps, but it would reject floating point values outside of the allowed range (+/- 1 << 53 for float64, +/- 1 << 24 for float32).

 

@wesm What do you think of this approach? Would it expose too much to the user?

[asfimport]

Wes McKinney / @wesm:
I'm unable to get to this for 0.12. Will have to wait for 0.13

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~farnoy] For me, the example you show above works:

In [33]: schema = pa.schema([)a.field(name='a', type=pa.int64(), nullable=True)])                                                                                                                    

In [34]: pa.Table.from_pandas(df, schema=schema, preserve_index=False)                                                                                                                                         
Out[34]: 
pyarrow.Table
a: int64
metadata
--------
{b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": [{"name":'
            b' "a", "field_name": "a", "pandas_type": "int64", "numpy_type": "'
            b'float64", "metadata": null}], "creator": {"library": "pyarrow", '
            b'"version": "0.13.1.dev313+g997226a9"}, "pandas_version": "0.24.2'
            b'"}'}

In [35]: table = _                                                                                                                                                                                                  

In [36]: table.column('a')                                                                                                                                                                                          
Out[36]: 
<Column name='a' type=DataType(int64)>
[
  [
    null,
    1,
    2,
    3,
    null
  ]
]

this is because in Table.from_pandas we assume data are coming from pandas and allow the above.

Using just the array API, you can see that with (converting float numpy array to integer arrow array):

In [41]: pa.array(np.array([1, 2, np.nan], dtype=float), type=pa.int64())                                                                                                                                           
...
ArrowInvalid: Floating point value truncated

In [42]: pa.array(np.array([1, 2, np.nan], dtype=float), type=pa.int64(), from_pandas=True)                                                                                                                         
Out[42]: 
<pyarrow.lib.Int64Array object at 0x7feaeea36548>
[
  1,
  2,
  null
]

Does that satisfy your use case?

It might not help with for very big integers that cannot be represented properly as floats (that will still raise an error about values being truncated), but I think if you are coming from pandas, that use case will not be very frequent, exactly because pandas cannot properly represent that itself.

[asfimport]

Jakub Okoński:
@jorisvandenbossche Is there a difference in your case between using np.nan and None?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am not sure I fully understand your question. I am passing a numpy array to pa.array, and creating a numpy array with np.nan or None (np.array([1, 2, None], dtype=float) or np.array([1, 2, np.nan], dtype=float)) give the same float64 numpy array.

Also if you have a pandas object like pd.Series([1, 2, None]) this will actually be stored as a float array with np.nan.

[asfimport]

Wes McKinney / @wesm:
Whatever issues there were back in October seem to be resolved now. I'm adding some unit tests to assert that that is the case