ARROW-7168: [Python] Respect the specified dictionary type for pd.Categorical conversion #5866
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rting pd.Categorical
pitrou
reviewed
Nov 19, 2019
| # (the type of the 'codes' in pandas is not part of the data type) | ||
| typ = pa.dictionary(index_type=pa.int16(), value_type=pa.string()) | ||
| result = pa.array(cat, type=typ) | ||
| assert result.type.equals(typ) |
There was a problem hiding this comment.
Shouldn't you check also the array value? (for example using result.to_pylist())
| typ = pa.dictionary(index_type=pa.int16(), value_type=pa.string()) | ||
| result = pa.array(cat, type=typ) | ||
| assert result.type.equals(typ) | ||
|
|
There was a problem hiding this comment.
Also, should you perhaps check passing a mask argument to pa.array()?
|
@pitrou thanks! Added some additional tests |
|
Also added some additional code to do it with a warning for now, falling back to the previous behaviour if the new behaviour fails. |
pitrou
approved these changes
Nov 20, 2019
|
We had some regressions with the changes in 0.15 related to following the types in the schema and the pandas index etc, so thought to be more cautious this time. But certainly not attached to it (I can't really judge if it will be something easy to run into or not). I think the extra maintenance work is limited, we will only need to remove the warnings and replace with an error at some point. |
|
Ok, merged as is :-) |
pitrou
pushed a commit
that referenced
this pull request
Dec 4, 2019
I guess this should be in #5866 Closes #5952 from mrkn/ARROW-7306 and squashes the following commits: e0793a3 <Kenta Murata> Add Result-returning version of FileSystemFromUri Authored-by: Kenta Murata <mrkn@mrkn.jp> Signed-off-by: Antoine Pitrou <antoine@python.org>
jorisvandenbossche
pushed a commit
that referenced
this pull request
Sep 15, 2020
…pe (int/string) Pandas dataframe to pyarrow Table
This PR homogenizes error messages for mixed-type `Pandas` inputs to `pa.Table`.
The message for `Pandas` column with `int` followed by `string` is now
```
In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')
```
the same as for `double` followed by `string`:
```
In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')
```
As a side effect, this snippet [xref #5866, ARROW-7168] now throws an `ArrowInvalid` (has been `FutureWarning` since 0.16):
```
In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object))
...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64())
...: result = pa.array(cat, type=typ)
(... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int
```
Finally, this *does* break a test [xref #4484, ARROW-4036] - see code comment
Closes #8044 from arw2019/ARROW-7663
Authored-by: arw2019 <andrew.r.wieteska@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
emkornfield
pushed a commit
to emkornfield/arrow
that referenced
this pull request
Oct 16, 2020
…pe (int/string) Pandas dataframe to pyarrow Table
This PR homogenizes error messages for mixed-type `Pandas` inputs to `pa.Table`.
The message for `Pandas` column with `int` followed by `string` is now
```
In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')
```
the same as for `double` followed by `string`:
```
In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')
```
As a side effect, this snippet [xref apache#5866, ARROW-7168] now throws an `ArrowInvalid` (has been `FutureWarning` since 0.16):
```
In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object))
...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64())
...: result = pa.array(cat, type=typ)
(... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int
```
Finally, this *does* break a test [xref apache#4484, ARROW-4036] - see code comment
Closes apache#8044 from arw2019/ARROW-7663
Authored-by: arw2019 <andrew.r.wieteska@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
…pe (int/string) Pandas dataframe to pyarrow Table
This PR homogenizes error messages for mixed-type `Pandas` inputs to `pa.Table`.
The message for `Pandas` column with `int` followed by `string` is now
```
In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')
```
the same as for `double` followed by `string`:
```
In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')
```
As a side effect, this snippet [xref apache#5866, ARROW-7168] now throws an `ArrowInvalid` (has been `FutureWarning` since 0.16):
```
In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object))
...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64())
...: result = pa.array(cat, type=typ)
(... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int
```
Finally, this *does* break a test [xref apache#4484, ARROW-4036] - see code comment
Closes apache#8044 from arw2019/ARROW-7663
Authored-by: arw2019 <andrew.r.wieteska@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/ARROW-7168
This change ensures that if you specify a
typeinpa.array, we ensure the output actually has this type when converting to dictionary array (as we also do for other types).The PR now implements this change, but we might want to do this with a deprecation first, as this can break people's code.