BUG: read_csv na_values parameters not working with pyarrow engine.

[Happily-Coding]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Since na_values parameter is set, '-' and '?' should be interpreted as na values, but aren't.
df_with_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])

print(df_with_pyarrow_engine.head(5))

Issue Description

When using engine='pyarrow' & na_values at the same time, on pandas read_csv method, additional na_values will not be detected correctly.

Expected Behavior

import pandas as pd

# Since na_values parameter is set, '-' and '?' will be interpreted as na values.
df_without_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    #engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])

print(df_without_pyarrow_engine.head(5))

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Spanish_Argentina.1252

pandas : 2.0.1
numpy : 1.23.5
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.1.2
Cython : None
pytest : None
hypothesis : None
sphinx : 6.1.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.11.0
pandas_datareader: None
bs4 : 4.11.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.4.0
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : 2.3.0
pyqt5 : None

[lithomas1]

Hi, do you know what pandas commit you were working off on on the main branch (pasting pd.show_versions() should be fine)?

I remember patching something similar to this pretty recently, and this runs fine for me on main, and raises an error on 2.0.1.

import pandas as pd
df_with_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])

df_without_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    #engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])
pd.testing.assert_frame_equal(df_with_pyarrow_engine, df_without_pyarrow_engine)

[Happily-Coding]

Hi, do you know what pandas commit you were working off on on the main branch (pasting pd.show_versions() should be fine)?

I remember patching something similar to this pretty recently, and this runs fine for me on main, and raises an error on 2.0.1.

import pandas as pd
df_with_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])

df_without_pyarrow_engine = pd.read_csv(
    'https://gist.githubusercontent.com/ycui1/dd33483219aeec871b9137a1e7925235/raw/6ae3e4e51b4807313068244dd315eb5e0a989827/pandas_missing_values_dataset.csv',
    #engine='pyarrow',
    dtype_backend='pyarrow',
    na_values=['-','?'])
pd.testing.assert_frame_equal(df_with_pyarrow_engine, df_without_pyarrow_engine)

Hey there. Thanks for the reply. The output is actually on the issue :). I'm working with commit : 37ea63d . I can see there are new commits after that, but no new releases, so it might've been fixed in those commits if you are using those instead.

I'l see if i can upgrade to the version on github and try again.

[lithomas1]

Ah, I was just confused since you ticked off the box that said that you tried on main.

You can actually install the nightly builds following this page https://pandas.pydata.org/docs/dev/getting_started/install.html#installing-the-development-version-of-pandas.

Let me know how it goes.
(If all goes well, I'll try to backport the change from main to the next 2.0.x release).

[Happily-Coding]

I just updated to the latest commit on github using:
pip install git+https://github.com/pandas-dev/pandas.git@824fc8304e6f6f202ee169c7118242374654a4a9
And its working properly on that commit!
Thanks a lot for the fix and for the replies !!!

Let me know if i should close the issue or wait for you to try backporting :)

INSTALLED VERSIONS

commit : 824fc83
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Spanish_Argentina.1252

pandas : 2.1.0.dev0+736.g824fc8304
numpy : 1.23.5
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.1.2
Cython : None
pytest : None
hypothesis : None
sphinx : 6.1.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.11.0
pandas_datareader: None
bs4 : 4.11.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.4.0
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : 2.3.0
pyqt5 : None

[lithomas1]

Sorry for the slow update, but I think the fix is now going to be backported to 2.0.2, which should be coming out sometime in the next week.

[RichardDally]

Hello @lithomas1, any update on the back port please ? :)

[lithomas1]

Sorry, I forgot to update but the back port was rejected.
(It depended on another PR that wasn't backported).

2.1 will be out in a month anyways, so hopefully the wait isn't too long.

[RichardDally]

Sorry, I forgot to update but the back port was rejected.

(It depended on another PR that wasn't backported).

2.1 will be out in a month anyways, so hopefully the wait isn't too long.

Thanks for the update.