BUG: csv newline \n \r\n in Window with opened file

[noklam]

• I have checked that this issue has not already been reported.
  Related issue: Saving and reloading a csv on Windows with a \n-character in a field replaces \n with \r\n. #22497, the issues looks related, but it is 2 years old and does not have any update.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

https://github.com/noklam/fsspec_pandas_bug_demo # if you need to download the data

# Your code here
import pandas as pd
import fsspec
from fsspec.implementations.local import LocalFileSystem
fs = LocalFileSystem()
df = pd.read_csv('iris.csv')

with fs.open('iris_fsspec.csv', 'w') as f:
    df.to_csv(f)

df.to_csv('iris_classic_pandas.csv')

Problem description

The output csv behavior is inconsistent

Expected Output

Expected to have identical output.

Output of pd.show_versions()

[paste the output of pd.show_versions() here leaving a blank line after the details tag]
OS: Window 10
INSTALLED VERSIONS

commit : 67a3d42
python : 3.8.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.17763
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : English_Hong Kong SAR.1252

pandas : 1.1.4
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1
setuptools : 46.3.1.post20200515
Cython : None
pytest : None
hypothesis : None
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.15.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 0.6.3
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.17.1
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.17
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[twoertwein]

thanks @noklam for your report!

As a potential temporary workaround, do you mind testing

with fs.open('iris_fsspec.csv', 'wb') as f:
    df.to_csv(f)

[noklam]

@twoertwein It will throws error instead.

$ python main.py
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    df.to_csv(f)
  File "C:\Users\lrcno\Anaconda3\lib\site-packages\pandas\core\generic.py", line 3170, in to_csv
    formatter.save()
  File "C:\Users\lrcno\Anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 206, in save
    self._save()
  File "C:\Users\lrcno\Anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 314, in _save
    self._save_header()
  File "C:\Users\lrcno\Anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 283, in _save_header
    writer.writerow(encoded_labels)
TypeError: a bytes-like object is required, not 'str

[twoertwein]

yes, sorry - binary handles will be supported in 1.2. I feel that this workaround might give you the same result as without the fsspec but I currently do not know why you are getting more newlines than necessary (maybe pandas and fsspec are inserting new lines independently of each other).

This issue doesn't seem to happen on Linux.

[noklam]

#10018

this old issue seems to describing the problem. the careiage return somehow get converted

[twoertwein]

Is there \r in your iris.csv? The issue #10018 seems to be a result of having \r as part of your data.

[noklam]

@twoertwein No.

>>> f = open('iris.csv')
>>> f.readline()
'sepal_length,sepal_width,petal_length,petal_width,species\n'
>>> f.readline()
'5.1,3.5,1.4,0.2,setosa\n'

I updated the issue as I found the problem is not specific for fsspec

This suffers from the same problem.

with open("iris_pyopen.csv", "w") as f:
    df.to_csv(f)

[noklam]

https://github.com/pandas-dev/pandas/blob/master/pandas/core/generic.py#L3220-L3224

Pandas doc actually mention if opening a file, it should be opened with newline='', any idea why is that ?

This seems fix the problem
with open("iris_pyopen.csv", "w" ,newline='') as f:
df.to_csv(f)

[ivanovmg]

https://github.com/pandas-dev/pandas/blob/master/pandas/core/generic.py#L3220-L3224

Pandas doc actually mention if opening a file, it should be opened with newline='', any idea why is that ?

This seems fix the problem
with open("iris_pyopen.csv", "w" ,newline='') as f:
df.to_csv(f)

Probably it comes from this: https://docs.python.org/3/library/csv.html#csv.writer
Check the footnotes there.