BUG: interchange bitmasks not supported in interchange/from_dataframe.py #52824
Conversation
| return np.ctypeslib.as_array( | ||
| data_pointer, shape=(buffer.bufsize // (bit_width // 8),) | ||
| ) | ||
|
|
||
|
|
||
| def bitmask_to_bool_ndarray( |
There was a problem hiding this comment.
looks like this entire function was "dead code" on arrival, https://app.codecov.io/gh/pandas-dev/pandas/blob/main/pandas/core/interchange/from_dataframe.py shows the whole thing as uncovered by tests
There was a problem hiding this comment.
Ah, so we actually did have a pure python version of this conversion .. (just not used)
| pa = pytest.importorskip("pyarrow", "11.0.0") | ||
|
|
||
| arr = [3.3, None, 2.1] | ||
| table = pa.table({"arr": arr}) |
There was a problem hiding this comment.
Maybe also test with table.slice(1), to ensure a sliced object works as well (that might cover the usage of offset in the new code)
There was a problem hiding this comment.
good one, thanks! that one didn't really work to begin with, so I had to make some slight changes to get it to work
| data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=col.size()) | ||
| data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=data_buff.bufsize) |
There was a problem hiding this comment.
had to change this one for the large-string test, where these are different:
(Pdb) data_buff.bufsize
6
(Pdb) col.size()
2If I don't change it, then we get:
In [5]: arr = ["Mon", "Tue"]
...: table = pa.table({"weekday": pa.array(arr, "large_string")})
...: exchange_df = table.__dataframe__()
...: result = from_dataframe(exchange_df)
In [6]: result
Out[6]:
weekday
0 Mo
1 this is the only test where data_buff.bufsize != col.size()
doc/source/whatsnew/v2.0.1.rst
Outdated
| @@ -28,6 +28,8 @@ Bug fixes | |||
| - Bug in :attr:`Series.dt.days` that would overflow ``int32`` number of days (:issue:`52391`) | |||
| - Bug in :class:`arrays.DatetimeArray` constructor returning an incorrect unit when passed a non-nanosecond numpy datetime array (:issue:`52555`) | |||
| - Bug in :func:`Series.median` with :class:`ArrowDtype` returning an approximate median (:issue:`52679`) | |||
| - Bug in :func:`api.interchange.from_dataframe` was returning :class:`DataFrame`'s of incorrect sizes when called on slices (:issue:`52824`) | |||
There was a problem hiding this comment.
Guess this needs to be moved to 2.0.2 now
|
Thanks @MarcoGorelli |
… in interchange/from_dataframe.py
….py (pandas-dev#52824) * support bitmasks in interchange * remove dead code * fixup for slice, add tests * tighten typing * reduce diff * post-merge fixup * add new whatsnew note * move to 2.0.2 * revert --------- Co-authored-by: MarcoGorelli <>
….py (pandas-dev#52824) * support bitmasks in interchange * remove dead code * fixup for slice, add tests * tighten typing * reduce diff * post-merge fixup * add new whatsnew note * move to 2.0.2 * revert --------- Co-authored-by: MarcoGorelli <>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This would solve all the "interchange to pandas" issues I'm aware of, so if we could get it in we could really take the interchange protocol for a ride