-
-
Notifications
You must be signed in to change notification settings - Fork 18.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: Utility function to swap all dtypes numpy
↔ pyarrow
#53648
Comments
5 tasks
HyukjinKwon
pushed a commit
to apache/spark
that referenced
this issue
Jul 10, 2023
### What changes were proposed in this pull request? The pr aims to upgrade `pandas` from 2.0.2 to 2.0.3. ### Why are the changes needed? 1.The new version brings some bug fixed, eg: - Bug in DataFrame.convert_dtype() and Series.convert_dtype() when trying to convert [ArrowDtype](https://pandas.pydata.org/docs/reference/api/pandas.ArrowDtype.html#pandas.ArrowDtype) with dtype_backend="nullable_numpy" ([GH53648](pandas-dev/pandas#53648)) - Bug in [read_csv()](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html#pandas.read_csv) when defining dtype with bool[pyarrow] for the "c" and "python" engines ([GH53390](pandas-dev/pandas#53390)) 2.Release notes: https://pandas.pydata.org/docs/whatsnew/v2.0.3.html ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes #41812 from panbingkun/SPARK-44267. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
ragnarok56
pushed a commit
to ragnarok56/spark
that referenced
this issue
Mar 2, 2024
### What changes were proposed in this pull request? The pr aims to upgrade `pandas` from 2.0.2 to 2.0.3. ### Why are the changes needed? 1.The new version brings some bug fixed, eg: - Bug in DataFrame.convert_dtype() and Series.convert_dtype() when trying to convert [ArrowDtype](https://pandas.pydata.org/docs/reference/api/pandas.ArrowDtype.html#pandas.ArrowDtype) with dtype_backend="nullable_numpy" ([GH53648](pandas-dev/pandas#53648)) - Bug in [read_csv()](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html#pandas.read_csv) when defining dtype with bool[pyarrow] for the "c" and "python" engines ([GH53390](pandas-dev/pandas#53390)) 2.Release notes: https://pandas.pydata.org/docs/whatsnew/v2.0.3.html ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes apache#41812 from panbingkun/SPARK-44267. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature Type
Adding new functionality to pandas
Changing existing functionality in pandas
Removing existing functionality in pandas
Problem Description
It would be nice to have a utility function that swaps the backend of all dtypes of an existing
DataFrame
/Series
/Index
.Feature Description
One could consider repurposing the
DataFrame.convert_dtypes
for this functionality.Currently, it does nothing:
Alternative Solutions
Potentially, one could also allow passing a transformation schema
dict[input_type, output_type]
to theastype
function.Additional Context
No response
The text was updated successfully, but these errors were encountered: