[Python] Parquet segfaults if coercing ns timestamps and writing 96-bit timestamps

[asfimport]

If you try to write a PyArrow table containing nanosecond-resolution timestamps to Parquet using coerce_timestamps and use_deprecated_int96_timestamps=True, the Arrow library will segfault.

The crash doesn't happen if you don't coerce the timestamp resolution or if you don't use 96-bit timestamps.

 

 

To Reproduce:

 

 
import datetime

import pyarrow
from pyarrow import parquet

schema = pyarrow.schema([
    pyarrow.field('last_updated', pyarrow.timestamp('ns')),
])

data = [
    pyarrow.array([datetime.datetime.now()], pyarrow.timestamp('ns')),
]

table = pyarrow.Table.from_arrays(data, ['last_updated'])

with open('test_file.parquet', 'wb') as fdesc:
    parquet.write_table(table, fdesc,
                        coerce_timestamps='us',  # 'ms' works too
                        use_deprecated_int96_timestamps=True)

 

See attached file for the crash report.

 

Environment: OS: Mac OS X 10.13.2
Python: 3.6.4
PyArrow: 0.8.0
Reporter: Diego Argueta / @dargueta
Assignee: Joshua Storck / @joshuastorck

Related issues:

• [Python] SegFault in pyarrow.parquet.write_table with specific options (Blocked)

Original Issue Attachments:

• crash-report.txt

Externally tracked issue: #1498

_{Note: This issue was originally created as ARROW-2020. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@cpcloud or @xhochy are you able to take a look at this?

[asfimport]

Antoine Pitrou / @pitrou:
This seems to have been fixed by changeset 4a8a102d508fc31570584b3618291103a901e6b0 (ARROW-969).

[asfimport]

Uwe Korn / @xhochy:
@pitrou The changeset looks unrelated. On master I still get the segementation fault.

[asfimport]

Antoine Pitrou / @pitrou:
Ok. Here the changeset does "fix" the crash somehow, but it still produces bogus results.

This issue might be related to ARROW-2026, in that when you pass coerce_timestamps, write_table seems to save the timestamps as int64 rather than int96.

[asfimport]

Wes McKinney / @wesm:
Moving this off 0.9.0

[asfimport]

Joshua Storck / @joshuastorck:
This is a duplicate of ARROW-2082, which is resolve with this PR. @wesm or @cpcloud, could you please mark this as a duplicate? I don't seem to have permissions.

[asfimport]

Antoine Pitrou / @pitrou:
Well apparently you do :-)