to_csv to Google Cloud Storage ignores compression

[VelizarVESSELINOV]

Code Sample, a copy-pastable example

dtf.to_csv('gs://mybucket/test1.csv.gz')
dtf.to_csv('gs://mybucket/test2.csv.gz', compression='gzip')

Problem description

Compression is not working for a file located on gs://

Expected Output

Working as a local version.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.8.1.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0
numpy : 1.18.5
pytz : 2019.3
dateutil : 2.8.1
pip : 20.2.1
setuptools : 41.2.0
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.13.0
pandas_datareader: 0.9.0
bs4 : None
bottleneck : None
fsspec : 0.8.0
fastparquet : None
gcsfs : 0.6.2
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : 3.0.4
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 0.7.10
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[twoertwein]

thank you for your bug report! I have a feeling that this might already work on master (#35129 might cover your case, if pandas internally convert 'gs://...' to a binary file object).

Can you confirm whether it works on master? I will need to first figure out google cloud storage before I can test this.

[VelizarVESSELINOV]

Tested with

commit           : 3c87b019b63ebff3c56acb6402fba5bc6f99671f
pandas           : 1.2.0.dev0+79.g3c87b019b

still not working.

[twoertwein]

Thanks for testing!

I got the google cloud storage working: df.to_csv("gs://mybucket/test2.csv.gz", compression="gzip", mode="wb") should work at the moment on master. I think I understand why the others don't work.

[VelizarVESSELINOV]

@twoertwein confirm it is working with mode='wb', it will be great if it is not needed to add compression and mode arguments so can work agnostically for local or cloud storage file path

[VelizarVESSELINOV]

@jreback Any chance to be added this fix to the 1.1.1 milestone? Waiting until the end of November vs waiting one week is a game-changer.

[twoertwein]

if 1.1.1 is the goal: #35681 depends on other PRs that would need to be back-ported too!

[jreback]

no 1.1.x are for regressions

[VelizarVESSELINOV]

@jreback a quick fact check: in pandas 1.1.x and 1.0.x release notes, there are bugs fixes inclusion, not only regressions fixes.

As a pandas user, don't appreciate a package maintainer that is happy to keep bugs unsolved for many months.

[jreback]

@VelizarVESSELINOV well that is going to persuade

pandas is a very complex package

the more backports the more possibly of broken-ness