BUG: read_csv converts quoted strings to floats with quoting=csv.QUOTE_NONNUMERIC

[bonsairobo]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import csv
import io
import pandas as pd

source = io.StringIO('"col"\n"1"')

df = pd.read_csv(source, quoting=csv.QUOTE_NONNUMERIC)

# Expect a string, but it parsed into a float.
assert df['col'][0] == '1'

Problem description

In the documentation of csv.QUOTE_NONNUMERIC, it says:

Instructs the reader to convert all non-quoted fields to type float.

Yet pandas.read_csv is converting my quoted value into a float. The above test case fails.

Expected Output

The code sample should pass the assertion.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.8.3.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.2
setuptools : 46.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[bonsairobo]

I've confirmed this is a pandas bug, not a csv bug.

import csv
import io

source = io.StringIO('"col"\n"1"')

reader = csv.reader(source, quoting=csv.QUOTE_NONNUMERIC)
rows = [r for r in reader]

assert rows[1] == ['1']

[rmsmani]

Hi @simonjayhawkins
I can take it up for fixing, as am new to this repo contribution I need some help from where to start with

[simonjayhawkins]

Thanks @rmsmani

the contributing docs are here

https://pandas.pydata.org/docs/dev/development/contributing.html

[psybers]

Is there any update on this bug? I am hitting it in a very weird way: specifically the string "None" is breaking in my string columns. For example:

import pandas as pd
import io
source = io.StringIO('"col"\n"Some"\n"None"\n"All"')
df = pd.read_csv(source)
print(df)

gives the output:

    col
0  Some
1   NaN
2   All

So the "None" string is being converted to float, which is NaN. Even if you specify the dtypes with string on that column, you get this behavior.

[thomas-vincent]

@psybers
On a side note, pandas v1.5.3 seemed to be correct:

    col
0  Some
1  None
2   All

Now in pandas > 2, all bugs related to csv reading can be avoided if you know the types in advance. Then you use the converters argument of pd.read_csv.

If you want to force strings in your example :

import pandas as pd
import io
source = io.StringIO('"col"\n"Some"\n"None"\n"All"')
df = pd.read_csv(source, converters={0:lambda v: v})