ENH: DataFrame.to_parquet() returns bytes if path_or_buf not provided #37129
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jreback
requested changes
Oct 16, 2020
pandas/io/parquet.py
Outdated
|
|
||
| if path is None: | ||
| path = io.BytesIO() | ||
|
|
||
| return impl.write( |
There was a problem hiding this comment.
you also need to return path.getvalue() no? (of path was None in the first place).
pandas/tests/io/test_parquet.py
Outdated
| def test_to_bytes_without_path_or_buf_provided(self): | ||
| # GH 37105 | ||
| df = pd.DataFrame() | ||
| df.to_parquet() |
There was a problem hiding this comment.
check one of the test dataframes, and test that the round-trip works as well, alt you can write it to a file and check that its the ssame.
arw2019
commented
Oct 16, 2020
pandas/io/parquet.py
Outdated
|
|
||
| if path is None: | ||
| path = io.BytesIO() | ||
|
|
||
| return impl.write( |
pandas/tests/io/test_parquet.py
Outdated
| def test_to_bytes_without_path_or_buf_provided(self): | ||
| # GH 37105 | ||
| df = pd.DataFrame() | ||
| df.to_parquet() |
jreback
requested changes
Oct 17, 2020
doc/source/whatsnew/v1.2.0.rst
Outdated
| @@ -194,6 +194,7 @@ Other enhancements | |||
| - Added :meth:`Rolling.sem()` and :meth:`Expanding.sem()` to compute the standard error of mean (:issue:`26476`). | |||
| - :meth:`Rolling.var()` and :meth:`Rolling.std()` use Kahan summation and Welfords Method to avoid numerical issues (:issue:`37051`) | |||
| - :meth:`DataFrame.plot` now recognizes ``xlabel`` and ``ylabel`` arguments for plots of type ``scatter`` and ``hexbin`` (:issue:`37001`) | |||
| - :meth:`DataFrame.to_parquet` now writes to ``io.Bytes`` when no ``path`` argument is passed (:issue:`37105`) | |||
There was a problem hiding this comment.
no this returns a bytes object
pandas/io/parquet.py
Outdated
| If a string, it will be used as Root Directory path | ||
| when writing a partitioned dataset. By file-like object, | ||
| we refer to objects with a write() method, such as a file handle | ||
| (e.g. via builtin open function) or io.BytesIO. The engine | ||
| fastparquet does not accept file-like objects. | ||
| fastparquet does not accept file-like objects. If path is None, | ||
| frame is written to an io.BytesIO object and a bytes object with |
There was a problem hiding this comment.
revise, the io.Bytes is a detail that is not important; its the return of bytes that is.
pandas/io/parquet.py
Outdated
| fastparquet does not accept file-like objects. | ||
| fastparquet does not accept file-like objects. If path is None, | ||
| frame is written to an io.BytesIO object and a bytes object with | ||
| the contents of the buffer is returned. |
There was a problem hiding this comment.
add a versionchanged 1.2 (return bytes)
pandas/tests/io/test_parquet.py
Outdated
| # GH 37105 | ||
|
|
||
| buf = df_full.to_parquet() | ||
|
|
arw2019
force-pushed
the
GH37105
branch
4 times, most recently
from
1518ad8
to
0bd0f47
Compare
jreback
approved these changes
Oct 20, 2020
|
lgtm. can you merge master and ping on green just to make sure. @jorisvandenbossche if any comments. |
pandas/tests/io/test_parquet.py
Outdated
|
|
||
| with tm.ensure_clean() as path: | ||
| with open(path, "wb") as f: | ||
| f.write(buf) |
There was a problem hiding this comment.
I would maybe rather test that you can directly read the bytes again (by wrapping it in a BytesIO?) instead of writing the bytes to a file.
jorisvandenbossche
approved these changes
Oct 21, 2020
|
Thanks @arw2019 ! |
JulianWgs
pushed a commit
to JulianWgs/pandas
that referenced
this pull request
Oct 26, 2020
kesmit13
pushed a commit
to kesmit13/pandas
that referenced
this pull request
Nov 2, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffNot sure if this is an API breaking change. Prior to this patch
pathwas a required positional argument but it becomes optional here. It is now consistent with, for example, the csv writer.