[Python] Non-nullable schema fields not checked in conversions from pandas

[asfimport]

If you provide a schema with nullable=False but pass a DataFrame which in fact has nulls it appears the schema is ignored? I would expect an error here.

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"a":[1.2, 2.1, pd.np.NaN]})
schema = pa.schema([pa.field("a", pa.float64(), nullable=False)])
table = pa.Table.from_pandas(df, schema=schema)
table[0]

<pyarrow.lib.Column object at 0x7f213bf2fb70>
chunk 0: <pyarrow.lib.DoubleArray object at 0x7f213bf20ea8>
[
  1.2,
  2.1,
  NA
]

Reporter: Matthew Gilbert
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #4683

_{Note: This issue was originally created as ARROW-2136. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
We have not yet implemented handling for non-nullable fields in most of the pandas conversions. Probably the simplest thing to do will be to leave the current conversion code as is and then raise an exception if a non-nullable field turns out to have nulls post-conversion to Arrow

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I have a PR for ARROW-5169 (#4397), which tries to use this nullability of the passed schema, but I should check how my PR interacts with passing data with nulls.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
You can also run into this when using pa.Table.from_arrays

In [2]: schema = pa.schema([pa.field("a", pa.float64(), nullable=False)])

In [4]: table = pa.Table.from_arrays([pa.array([1.5, None])], schema=schema) 

In [5]: table.schema
Out[5]: a: double

In [6]: table.schema.field_by_name('a')
Out[6]: pyarrow.Field<a: double not null>

In [7]: table.column('a')
Out[7]: 
<Column name='a' type=DataType(double)>
[
  [
    1.5,
    null
  ]
]

Under the hood, this function is doing Column(field, array), and the Column constructor is assuming the field datatype matches the array's datatype.

There is a Column::ValidateData(), but looking at the implementation, that only checks that the chunks' types equal the field type, and does not check any metadata such as nullability (although the doc comment says "Verify that the column's array data is consistent with the passed field's metadata")

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
For Table.from_pandas, in the end, the actual conversion is done column by column with pa.array.

So the question for me is, do we want to bake this in into pa.array (eg add a nullable=True default keyword to pa.array)? So that the underlying conversion raises when it cannot satisfy a nullable=False.

Alternatively, we could also rather easily verify in pandas_compat.dataframe_to_arrays (the function that calls pa.array for each column) that the arrays null_count does not conflict with the schema fields' metadata on this. Raising while converting in pa.array could of course raise earlier.

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 4683
#4683