-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: interchange bitmasks not supported in interchange/from_dataframe.py #52824
Conversation
return np.ctypeslib.as_array( | ||
data_pointer, shape=(buffer.bufsize // (bit_width // 8),) | ||
) | ||
|
||
|
||
def bitmask_to_bool_ndarray( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks like this entire function was "dead code" on arrival, https://app.codecov.io/gh/pandas-dev/pandas/blob/main/pandas/core/interchange/from_dataframe.py shows the whole thing as uncovered by tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, so we actually did have a pure python version of this conversion .. (just not used)
pa = pytest.importorskip("pyarrow", "11.0.0") | ||
|
||
arr = [3.3, None, 2.1] | ||
table = pa.table({"arr": arr}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe also test with table.slice(1)
, to ensure a sliced object works as well (that might cover the usage of offset
in the new code)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good one, thanks! that one didn't really work to begin with, so I had to make some slight changes to get it to work
data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=col.size()) | ||
data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=data_buff.bufsize) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
had to change this one for the large-string
test, where these are different:
(Pdb) data_buff.bufsize
6
(Pdb) col.size()
2
If I don't change it, then we get:
In [5]: arr = ["Mon", "Tue"]
...: table = pa.table({"weekday": pa.array(arr, "large_string")})
...: exchange_df = table.__dataframe__()
...: result = from_dataframe(exchange_df)
In [6]: result
Out[6]:
weekday
0 Mo
1
this is the only test where data_buff.bufsize != col.size()
doc/source/whatsnew/v2.0.1.rst
Outdated
@@ -28,6 +28,8 @@ Bug fixes | |||
- Bug in :attr:`Series.dt.days` that would overflow ``int32`` number of days (:issue:`52391`) | |||
- Bug in :class:`arrays.DatetimeArray` constructor returning an incorrect unit when passed a non-nanosecond numpy datetime array (:issue:`52555`) | |||
- Bug in :func:`Series.median` with :class:`ArrowDtype` returning an approximate median (:issue:`52679`) | |||
- Bug in :func:`api.interchange.from_dataframe` was returning :class:`DataFrame`'s of incorrect sizes when called on slices (:issue:`52824`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Guess this needs to be moved to 2.0.2 now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One comment otherwise LGTM
Thanks @MarcoGorelli |
… in interchange/from_dataframe.py
….py (pandas-dev#52824) * support bitmasks in interchange * remove dead code * fixup for slice, add tests * tighten typing * reduce diff * post-merge fixup * add new whatsnew note * move to 2.0.2 * revert --------- Co-authored-by: MarcoGorelli <>
….py (pandas-dev#52824) * support bitmasks in interchange * remove dead code * fixup for slice, add tests * tighten typing * reduce diff * post-merge fixup * add new whatsnew note * move to 2.0.2 * revert --------- Co-authored-by: MarcoGorelli <>
doc/source/whatsnew/vX.X.X.rst
file if fixing a bug or adding a new feature.This would solve all the "interchange to pandas" issues I'm aware of, so if we could get it in we could really take the interchange protocol for a ride