[Python] Handle more kinds of null sentinel objects from pandas 0.x

[asfimport]

Follow-on work to ARROW-707. See https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/lib.pyx#L193 and discussion in #554

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Pandas - Parquet - Pandas roundtrip causes out of bounds timestamp (fixes)
• [Python] Conversion when mixing NaT and datetime objects not working (duplicates)
• [Python] scalar pd.NaT incorrectly parsed in conversion from Python (duplicates)
• [Python] ArrowInvalid error raised when deserialising pandas with pd.NaT values in object column (causes)
• [Python] Conversion when mixing NaT and datetime objects not working (relates to)

PRs and other links:

• GitHub Pull Request #7537

_{Note: This issue was originally created as ARROW-842. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
This might wait for more general tooling around NumPy scalar types. See also ARROW-1646

[asfimport]

Wes McKinney / @wesm:
Removing this from a Fix Version until it can be demonstrated that it's a problem

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
pd.NaT is one that is not recognized, but, only in lists / object dtype:

In [25]: pa.array([pd.Timestamp("2012-01-01"), pd.NaT, None, np.nan, float('nan')])                                                                           
Out[25]: 
<pyarrow.lib.TimestampArray object at 0x7f27703b82c8>
[
  1325376000000000,
  -62135596800000000,
  null,
  null,
  null
]

If you actually pass a pandas Series object with the proper dtype such as datetime64[ns], then those missing values are handled correctly:

In [32]: pa.array(pd.Series([pd.Timestamp("2012-01-01"), pd.NaT]))                                                                                            
Out[32]: 
<pyarrow.lib.TimestampArray object at 0x7f2752662b38>
[
  1325376000000000000,
  null
]

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Updated example from ARROW-6305 that shows the NaT problem if it is passed as a scalar:

In [6]: pa.array([pd.Timestamp("2012-01-01"), pd.NaT]) 
Out[6]: 
<pyarrow.lib.TimestampArray object at 0x7f46c8368780>
[
  2012-01-01 00:00:00.000000,
  0001-01-01 00:00:00.000000
]

[asfimport]

Jakub Okoński:
Hey folks, I hit the same problem with pd.NaT being converted to 0001-01-01. Is there any PR to address it? Is this planned to be fixed for the next release?

[asfimport]

Wes McKinney / @wesm:
You're more than welcome to submit a PR to detect the NaT value

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The next pandas release is also adding a pd.NA object. That's another null sentinel that we could learn to recognize

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 7537
#7537