BUG/ENH: compression for google cloud storage in to_csv #35681

twoertwein · 2020-08-12T01:38:04Z

closes to_csv to Google Cloud Storage ignores compression #35677, closes to_csv to Google Cloud Storage ignores encoding #26124, and closes read_csv from Google Cloud Storage ignores encoding #32392
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

By inferring the compression before converting the path to a file object df.to_csv("gs://mybucket/test2.csv.gz", compression="infer", mode="wb") works. By wrapping fsspec file-objects in a TextIOWrapper df.to_csv("gs://mybucket/test2.csv", mode="wb") works as well. Path-like objects that are internally converted to file-like objects (in get_filepath_or_buffer) are now always opened in binary mode (unless text mode is explicitly requested) and the potentially changed mode is returned (no need to specify mode="wb" for google cloud files). As long as the google file is opened in binary mode (which is now always the case), we also honor the requested encoding.

This PR also fixes Zip compression for file objects not having a name.

pandas/tests/io/test_gcs.py

pandas/io/common.py

pandas/_libs/parsers.pyx

pandas/io/common.py

twoertwein · 2020-08-14T02:22:33Z

it seems that only the windows py37 machine on azure actually tests the google cloud interface?!

EDIT: and one linux machine on travis

pandas/io/json/_json.py

pandas/io/orc.py

twoertwein · 2020-08-26T03:14:15Z

pandas/io/common.py

@@ -162,13 +165,13 @@ def is_fsspec_url(url: FilePathOrBuffer) -> bool:
    )


-def get_filepath_or_buffer(
+def get_filepath_or_buffer(  # type: ignore[assignment]


my local mypy needs that for line 170 and 172 but the CI mypy needs it apparently at that line (TypeVars cannot have default values, could be fixed with @overload)

twoertwein · 2020-08-26T03:17:51Z

pandas/io/common.py

@@ -583,12 +638,15 @@ def __init__(
        self.archive_name = archive_name
        kwargs_zip: Dict[str, Any] = {"compression": zipfile.ZIP_DEFLATED}
        kwargs_zip.update(kwargs)
-        super().__init__(file, mode, **kwargs_zip)
+        super().__init__(file, mode, **kwargs_zip)  # type: ignore[arg-type]


complains about file being IOBase but we cannot have an assert not isisntance(file, IOBase) since io.StringIO inherits from IOBase

pandas/io/json/_json.py

twoertwein · 2020-08-26T03:28:09Z

pandas/io/stata.py

            fname, mode="wb", compression=compression, storage_options=storage_options,
        )
-        f, _ = get_handle(path_or_buf, "wb", compression=compression, is_text=False)
-        return f, True, compression
+        f, _ = get_handle(


I assume the auxiliary file handles should be closed as well? I think that only happens for compressed files. But that would require yet another interface change.

twoertwein · 2020-08-26T03:39:52Z

Typing of filepath_or_buffer is a mess. I also added IOBase to FilePathOrBuffer (io.BytesIO and io.StringIO do not seem to be covered by isinstance(..., typing.IO)).

I ended up using a dataclass for IOargs, NamedTuple does not support TypeVars in it. I use TypeVars for encoding (stays None/str) and mode (stays None/str or it changes from None to str).

If IOargs was retrieved inside an if-branch, I 'unpack' it (using the tuppe names) into the previous variables, otherwise I use ioargs. instead of the previous variable names.

jreback

looks really good. a couple of comments.

pandas/io/common.py

pandas/io/feather_format.py

pandas/io/json/_json.py

jreback · 2020-08-27T02:38:39Z

pandas/io/stata.py

+            compression=ioargs.compression,
+            is_text=False,
+        )
+        return f, True, ioargs.compression


might need a slight refactoring but can you use IOargs here?

it returns similar information but in this case f is always BinaryIO (according to the current type annotations). If we were to use IOargs, we would give it a broader type.

I think the following options would make sense for this function:

inline it (it is used only one time and it is a private function)

simplify it (the first if-block is imho not necessary, can be handled by the elif-block)

simplification from (2) and move the function to io/common and use it more than once: the pattern to convert the compression string/dict to a dict, calling get_filepath_or_buffer, and then calling get_handle is present in many to_* (and read_*) functions.

I'm tempted to go for the second option in this PR and leave the third option for a future PR.

I will not touch that for now: I don't understand the difference between self._output_file and self._file in StataWriter. Option (2) would affect these two variables. Having had only a brief look at StataWriter, it seems that this class distinguishes between writing to a compressed file and writing to a buffer (get_handle should take care of that, after that you should be able to treat them the same?).

yeah this can all addressed later. this writer might need some restructuing.

pandas/tests/io/test_common.py

jreback · 2020-08-27T02:40:20Z

pandas/tests/io/test_gcs.py

-def test_read_csv_gcs(monkeypatch):
+@pytest.fixture
+def gcs_buffer(monkeypatch):
+    """Emulate GCS using a binary buffer."""
    from fsspec import AbstractFileSystem, registry

    registry.target.clear()  # noqa  # remove state



looks fine, does this make it easier / cleaner?

@alimcmaster1 commented one some duplication across the GCS tests. I'm not 100% sure how the monkeypatch part works, the CI seems to be happy about it. Double/triple checking that I didn't invalidated these tests would be good.

jreback · 2020-08-27T02:41:48Z

@WillAyd @TomAugspurger @simonjayhawkins if any comments; also using Generic protocols now so if you guys can review types (not to get bogged down of course.....)

get_handle: fsspec file objects need to be wrapped get_filepath_or_buffer: path-like objects that are internally converted to file-like objects are opened in binary mode; named tuple _BytesZipFile: work with filename-less objects

…gnore statements (mypy will compile about filepath_or_buffer)

…es; refine type for filepath_or_buffer

jreback

lgtm @twoertwein

thanks for the patch!

jreback · 2020-09-03T03:05:02Z

pandas/io/stata.py

+            compression=ioargs.compression,
+            is_text=False,
+        )
+        return f, True, ioargs.compression


yeah this can all addressed later. this writer might need some restructuing.

…5681)

twoertwein marked this pull request as draft August 12, 2020 01:38

twoertwein force-pushed the google_storage branch from 27e881d to 33d2f44 Compare August 12, 2020 02:48

VelizarVESSELINOV reviewed Aug 12, 2020

View reviewed changes

pandas/tests/io/test_gcs.py Outdated Show resolved Hide resolved

VelizarVESSELINOV reviewed Aug 12, 2020

View reviewed changes

pandas/tests/io/test_gcs.py Outdated Show resolved Hide resolved

twoertwein force-pushed the google_storage branch 2 times, most recently from c73915e to 1fe2c39 Compare August 12, 2020 13:39

twoertwein commented Aug 12, 2020

View reviewed changes

pandas/io/common.py Show resolved Hide resolved

twoertwein force-pushed the google_storage branch 4 times, most recently from 4b5272d to ca1800b Compare August 13, 2020 13:32

twoertwein commented Aug 13, 2020

View reviewed changes

pandas/_libs/parsers.pyx Outdated Show resolved Hide resolved

twoertwein force-pushed the google_storage branch from 378d37f to 0362178 Compare August 13, 2020 19:44

twoertwein marked this pull request as ready for review August 13, 2020 22:03

twoertwein force-pushed the google_storage branch 4 times, most recently from 78925fd to a3d012c Compare August 13, 2020 23:56

jreback added Compat pandas objects compatability with Numpy or Python functions IO Google labels Aug 14, 2020

jreback added this to the 1.2 milestone Aug 14, 2020

VelizarVESSELINOV reviewed Aug 14, 2020

View reviewed changes

pandas/io/common.py Outdated Show resolved Hide resolved

twoertwein mentioned this pull request Aug 14, 2020

to_csv to Google Cloud Storage ignores compression #35677

Closed

twoertwein force-pushed the google_storage branch 4 times, most recently from b2a4eb9 to 479b7bf Compare August 15, 2020 14:46

twoertwein mentioned this pull request Aug 15, 2020

BUG/ENH: to_pickle/read_pickle support compression for file ojects #35736

Merged

5 tasks

twoertwein force-pushed the google_storage branch from d78fb7f to 7a686cc Compare August 17, 2020 12:32

twoertwein force-pushed the google_storage branch from 3af5df6 to 18927c0 Compare August 25, 2020 21:08

twoertwein commented Aug 25, 2020

View reviewed changes

pandas/io/json/_json.py Show resolved Hide resolved

twoertwein commented Aug 25, 2020

View reviewed changes

pandas/io/orc.py Outdated Show resolved Hide resolved

twoertwein force-pushed the google_storage branch 2 times, most recently from e7248e4 to 7b03c2a Compare August 26, 2020 02:52

twoertwein commented Aug 26, 2020

View reviewed changes

pandas/io/json/_json.py Outdated Show resolved Hide resolved

twoertwein commented Aug 26, 2020

View reviewed changes

jreback reviewed Aug 27, 2020

View reviewed changes

twoertwein force-pushed the google_storage branch 8 times, most recently from 1063e3f to 90c4e01 Compare August 30, 2020 14:47

twoertwein added 3 commits August 31, 2020 16:14

bind input type of encding and mode with the returned type; removed i…

935fc4b

…gnore statements (mypy will compile about filepath_or_buffer)

use named tuple; remove some unused variables; closed some file handl…

475e8e8

…es; refine type for filepath_or_buffer

twoertwein force-pushed the google_storage branch 2 times, most recently from 1357c4a to 475e8e8 Compare August 31, 2020 23:18

jreback approved these changes Sep 3, 2020

View reviewed changes

jreback merged commit 361166f into pandas-dev:master Sep 3, 2020

twoertwein deleted the google_storage branch September 3, 2020 15:56

arw2019 mentioned this pull request Sep 23, 2020

TST: DataFrame.to_parquet accepts pathlib.Path with partition_cols defined #36491

Merged

4 tasks

kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020

BUG/ENH: compression for google cloud storage in to_csv (pandas-dev#3…

327707b

…5681)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG/ENH: compression for google cloud storage in to_csv #35681

BUG/ENH: compression for google cloud storage in to_csv #35681

twoertwein commented Aug 12, 2020 •

edited

Loading

twoertwein commented Aug 14, 2020 •

edited

Loading

twoertwein Aug 26, 2020

twoertwein Aug 26, 2020

twoertwein Aug 26, 2020

twoertwein commented Aug 26, 2020

jreback left a comment

jreback Aug 27, 2020

twoertwein Aug 27, 2020

twoertwein Aug 31, 2020

twoertwein Aug 31, 2020

jreback Sep 3, 2020

jreback Aug 27, 2020

twoertwein Aug 27, 2020

jreback commented Aug 27, 2020

jreback left a comment

jreback Sep 3, 2020

BUG/ENH: compression for google cloud storage in to_csv #35681

BUG/ENH: compression for google cloud storage in to_csv #35681

Conversation

twoertwein commented Aug 12, 2020 • edited Loading

twoertwein commented Aug 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twoertwein commented Aug 26, 2020

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 27, 2020

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twoertwein commented Aug 12, 2020 •

edited

Loading

twoertwein commented Aug 14, 2020 •

edited

Loading