Skip to content

Commit

Permalink
Backport PR #43447: REGR: SpooledTemporaryFile support in read_csv (#…
Browse files Browse the repository at this point in the history
…43488)

Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com>
  • Loading branch information
meeseeksmachine and twoertwein committed Sep 10, 2021
1 parent 5d6e352 commit 8b55e4f
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 2 deletions.
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Fixed regressions
- Fixed regression in :meth:`DataFrame.corr` where Kendall correlation would produce incorrect results for columns with repeated values (:issue:`43401`)
- Fixed regression in :meth:`DataFrame.groupby` where aggregation on columns with object types dropped results on those columns (:issue:`42395`, :issue:`43108`)
- Fixed regression in :meth:`Series.fillna` raising ``TypeError`` when filling ``float`` ``Series`` with list-like fill value having a dtype which couldn't cast lostlessly (like ``float32`` filled with ``float64``) (:issue:`43424`)
- Fixed regression in :func:`read_csv` throwing an ``AttributeError`` when the file handle is an ``tempfile.SpooledTemporaryFile`` object (:issue:`43439`)
-

.. ---------------------------------------------------------------------------
Expand Down
12 changes: 10 additions & 2 deletions pandas/io/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
)
import mmap
import os
import tempfile
from typing import (
IO,
Any,
Expand Down Expand Up @@ -943,8 +944,15 @@ def _is_binary_mode(handle: FilePathOrBuffer, mode: str) -> bool:
if "t" in mode or "b" in mode:
return "b" in mode

# classes that expect string but have 'b' in mode
text_classes = (codecs.StreamWriter, codecs.StreamReader, codecs.StreamReaderWriter)
# exceptions
text_classes = (
# classes that expect string but have 'b' in mode
codecs.StreamWriter,
codecs.StreamReader,
codecs.StreamReaderWriter,
# cannot be wrapped in TextIOWrapper GH43439
tempfile.SpooledTemporaryFile,
)
if issubclass(type(handle), text_classes):
return False

Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/io/parser/test_encoding.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,3 +237,16 @@ def test_encoding_memory_map(all_parsers, encoding):
expected.to_csv(file, index=False, encoding=encoding)
df = parser.read_csv(file, encoding=encoding, memory_map=True)
tm.assert_frame_equal(df, expected)


def test_not_readable(all_parsers):
# GH43439
parser = all_parsers
if parser.engine in ("python", "pyarrow"):
pytest.skip("SpooledTemporaryFile does only work with the c-engine")
with tempfile.SpooledTemporaryFile() as handle:
handle.write(b"abcd")
handle.seek(0)
df = parser.read_csv(handle)
expected = DataFrame([], columns=["abcd"])
tm.assert_frame_equal(df, expected)

0 comments on commit 8b55e4f

Please sign in to comment.