ARROW-9096: [Python] Pandas roundtrip with dtype="object" underlying numeric column index #7822
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arw2019
changed the title
arw2019
changed the title
arw2019
changed the title
emkornfield
reviewed
Jul 24, 2020
python/pyarrow/pandas_compat.py
Outdated
| @@ -1009,12 +1009,15 @@ def _is_generated_index_name(name): | |||
| return re.match(pattern, name) is not None | |||
|
|
|||
|
|
|||
| # ARROW-9096: added integer and floating | |||
There was a problem hiding this comment.
nit: generally only have comment like this for TODOs. git blame/git log can track when things were added.
emkornfield
reviewed
Jul 24, 2020
python/pyarrow/pandas_compat.py
Outdated
| @@ -1080,8 +1083,10 @@ def _reconstruct_columns_from_metadata(columns, column_indexes): | |||
| ] | |||
|
|
|||
| # Convert each level to the dtype provided in the metadata | |||
| # ARROW-9096: need numpy_type to match cast against original DataFrame | |||
There was a problem hiding this comment.
nit: drop JIRA reference. maybe also the whole comment since this is used directly below?
There was a problem hiding this comment.
makes sense - it's easy to see what's going on w/o this line so dropped it as suggested
|
thanks @wesm @emkornfield for reviewing!! |
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
…numeric column index - [x] closes ARROW-9096 - [x] tests added & passed This PR fixes the roundtrip conversion for Pandas DataFrames whose column index is numeric but has `dtype=object`, such as ``` df = pd.DataFrame([1], columns=pd.Index([1], dtype=object)) # underlying int df = pd.DataFrame([1], columns=pd.Index([1.1], dtype=object)) # underlying float df = pd.DataFrame([1], columns=pd.Index([datetime(2018, 1, 1)], dtype='object')) # underlying datetime ``` https://issues.apache.org/jira/browse/ARROW-3651 largely solved the datetime variant of this problem (such that the conversion ran correctly excepting that the dtype after roundtrip did not match). With the current fix a roundtrip of the problematic DataFrames from ARROW-3651 returns the exact original frame. Closes apache#7822 from arw2019/ARROW-9096 Authored-by: arw2019 <andrew.r.wieteska@gmail.com> Signed-off-by: Wes McKinney <wesm@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the roundtrip conversion for Pandas DataFrames whose column index is numeric but has
dtype=object, such ashttps://issues.apache.org/jira/browse/ARROW-3651 largely solved the datetime variant of this problem (such that the conversion ran correctly excepting that the dtype after roundtrip did not match). With the current fix a roundtrip of the problematic DataFrames from ARROW-3651 returns the exact original frame.