We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using pyathena's results-to-pandas conversion feature breaks pandas to_csv function.
to_csv
Minimum reproducible example:
>>> from pyathena.pandas.util import as_pandas >>> import pandas as pd >>> df = pd.DataFrame({"foo": [1,2,3]}) >>> df.to_csv("s3://...") Traceback (most recent call last): File "/Users/gshand/code/adhoc/pandas_pyathena/test.py", line 5, in <module> df.to_csv("s3://...") File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pandas/core/generic.py", line 3902, in to_csv return DataFrameRenderer(formatter).to_csv( File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pandas/io/formats/format.py", line 1152, in to_csv csv_formatter.save() File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pandas/io/formats/csvs.py", line 247, in save with get_handle( File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pandas/io/common.py", line 142, in __exit__ self.close() File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pandas/io/common.py", line 134, in close handle.close() File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pyathena/filesystem/s3.py", line 520, in close super(S3File, self).close() File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/fsspec/spec.py", line 1944, in close self.flush(force=True) File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/fsspec/spec.py", line 1810, in flush self._initiate_upload() File "/Users/gshand/code/adhoc/pandas_pyathena/.venv/lib/python3.10/site-packages/pyathena/filesystem/s3.py", line 524, in _initiate_upload raise NotImplementedError # pragma: no cover NotImplementedError
Removing the as_pandas import allows the script to work as intended.
as_pandas
Expected Behavior: pyathena doesn't hijack pandas' default s3fs library and to_csv continues to work as normal.
Versions:
pandas==2.1.1 pyathena==3.0.8
The text was updated successfully, but these errors were encountered:
Please do not register the same issue. #465
Sorry, something went wrong.
Apologies, I did search before posting but didn't find that.
No branches or pull requests
Using pyathena's results-to-pandas conversion feature breaks pandas
to_csv
function.Minimum reproducible example:
Removing the
as_pandas
import allows the script to work as intended.Expected Behavior: pyathena doesn't hijack pandas' default s3fs library and
to_csv
continues to work as normal.Versions:
The text was updated successfully, but these errors were encountered: