[Python] write csv decimal cast error

[asfimport]

Hi, when try to write table with any field in Decimal128 type, arrow raises with this message:

In [136]: ds.write_dataset(table, "data", format="csv")
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
Cell In [136], line 1
----> 1 ds.write_dataset(table, "data", format="csv")

File c:\users\documents\projects\.venv\lib\site-packages\pyarrow\dataset.py:930, in write_dataset(data, base_dir, basename_template, format, partitioning, partitioning_flavor, schema, filesystem, file_options, use_threads, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, file_visitor, existing_data_behavior, create_dir)
    927         raise ValueError("Cannot specify a schema when writing a Scanner")
    928     scanner = data
--> 930 _filesystemdataset_write(
    931     scanner, base_dir, basename_template, filesystem, partitioning,
    932     file_options, max_partitions, file_visitor, existing_data_behavior,
    933     max_open_files, max_rows_per_file,
    934     min_rows_per_group, max_rows_per_group, create_dir
    935 )

File c:\users\documents\projects\.venv\lib\site-packages\pyarrow\_dataset.pyx:2737, in pyarrow._dataset._filesystemdataset_write()

File c:\users\documents\projects\.venv\lib\site-packages\pyarrow\error.pxi:121, in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from decimal128(21, 15) to utf8 using function cast_string

my data is:

In [137]: table
Out[137]: 
pyarrow.Table
col1: int64
col2: double
col3: decimal128(21, 15)
col4: string
----
col1: [[1,2,3,0]]
col2: [[2.7,0,3.24,3]]
col3: [[-304236.460000000000000,0.E-15,0.E-15,0.E-15]]
col4: [["primera","segunda","tercera","cuarta"]]

 

Thanks in advance.

Reporter: Alejandro Marco Ramos
Assignee: Miles Granger / @milesgranger

Related issues:

• [C++] CSV Writer: Unsupported cast from decimal to utf8 (relates to)

PRs and other links:

• GitHub Pull Request #14525

_{Note: This issue was originally created as ARROW-17774. Please see the migration documentation for further details.}

[asfimport]

Miles Granger / @milesgranger:
Indeed and unfortunate. :(
Basically a duplicate of ARROW-17458 (as experienced in C++) but of course experienced here in Python.

Small reproducible example:

import pyarrow as pa
import pyarrow.dataset as ds

table = pa.table({'col1': pa.array([1, 2], pa.decimal128(21, 15))})
ds.write_dataset(table, "data.csv", format="csv")
...
ArrowNotImplementedError: Unsupported cast from decimal128(21, 15) to utf8 using function cast_string

[asfimport]

Weston Pace / @westonpace:
The underlying issues appears to have been solved. I just checked that this indeed works with the CSV writer:

import decimal

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.csv as csv

decimals = pa.array([decimal.Decimal(1), decimal.Decimal(0.3)])
small_decimals = pc.cast(decimals, pa.decimal256(12, 6), safe=False)
table = pa.Table.from_pydict({"rownum": [1, 2], "decimal": small_decimals})
csv.write_csv(table, '/tmp/foo.csv')

with open('/tmp/foo.csv') as f:
    print(f.read())
# "rownum","decimal"
# 1,1.000000
# 2,0.299999

Do we want to add any pyarrow tests? Or just close this.

[asfimport]

Miles Granger / @milesgranger:
Thanks @westonpace, good timing. :)
I added a PR #14525 with some simple checks, let me know if you have suggestions there or if you don't think it's worth it I'm fine just closing this and the PR. (y)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 14525
#14525