BUG: Interchange object data buffer has the wrong dtype / from_dataframe incorrect

[stinodego]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from datetime import datetime
import pandas as pd

df = pd.DataFrame(
    {
        "string": ["a", "bc", None],
        "datetime": [datetime(2022, 1, 1), datetime(2022, 1, 2), datetime(2022, 1, 3)],
        "categorical": pd.Series(["a", "b", "a"], dtype="category"),
    }
)

dfi = df.__dataframe__()

col = dfi.get_column_by_name("string")
print(col.dtype)
# (<DtypeKind.STRING: 21>, 8, 'u', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.STRING: 21>, 8, 'u', '=') -> SHOULD BE: (<DtypeKind.UINT: 1>, 8, 'C', '=')

col = dfi.get_column_by_name("datetime")
print(col.dtype)
# (<DtypeKind.DATETIME: 22>, 64, 'tsn:', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.DATETIME: 22>, 64, 'tsn:', '=') -> SHOULD BE: (<DtypeKind.INT: 0>, 64, 'l', '=')

col = dfi.get_column_by_name("categorical")
print(col.dtype)
# (<DtypeKind.CATEGORICAL: 23>, 8, 'c', '=')
print(col.get_buffers()["data"][1])
# (<DtypeKind.INT: 0>, 8, 'c', '|') -> CORRECT!

Issue Description

As you can see, the dtype of the data buffer is the same as the dtype of the column. This is only correct for integers and floats. Categoricals, strings, and datetime types have an integer type as their physical representation. The data buffer should have this physical data type associated with it.

The dtype of the Column object should provide information on how to interpret the various buffers. The dtype associated with each buffer should be the dtype of the actual data in that buffer. This is the second part of the issue: the implementation of from_dataframe is incorrect - it should use the column dtype rather than the data buffer dtype.

Expected Behavior

Fixing the get_buffers implementation should be relatively simple (expected dtypes are listed as comment in the example above). However, this will break any from_dataframe implementation (also from other libraries) that rely on the data buffer having the column dtype.

So fixing this should ideally go in three steps:

1. Fix the from_dataframe implementation to use the column dtype rather than the data buffer dtype to interpret the buffers.
2. Make sure other libraries have also updated their from_dataframe implementation. See [Python] Interchange object data buffer has the wrong dtype / from_dataframe incorrect apache/arrow#37598 for the pyarrow issue.
3. Fix the data buffer dtypes.

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.11.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.90.1-microsoft-standard-WSL2
Version : #1 SMP Fri Jan 27 02:56:13 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.3
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.2.1
Cython : None
pytest : 7.4.0
hypothesis : 6.82.0
sphinx : 7.1.1
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.6.0
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : 2.0.20
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[MarcoGorelli]

thanks @stinodego for the report

could you clarify how to reproduce the assertion error please? I've tried building from source with the latest commits from both pandas and polars, but didn't get the assertionerror

[stinodego]

Hey @MarcoGorelli ,

I've opened a PR in the Polars repo, you can use that branch to reproduce this:
pola-rs/polars#10787

You can also see from the changes in that branch what the issue is here. The issue extends to temporal types too.

[MarcoGorelli]

thanks

This cannot be merged until implementations for other libraries are fixed too. See pandas issue #54781

2.1 is going out today, so at this point it's going to have to be in for version 2.2 I'm afraid, which will be at least 4 months' away. But, will get to it

[stinodego]

I updated the issue with a more extensive description and better reproducible example.

Tagging @MarcoGorelli and @rgommers for visibility 😸

[MarcoGorelli]

Right, I think you're correct here

It is rather odd that for datetime, pandas does some "gymnastics" around the buffer dtype:

pandas/pandas/core/interchange/from_dataframe.py

Lines 381 to 393 in 329fe8f

	dbuf, dtype = buffers["data"]
	# Consider dtype being `uint` to get number of units passed since the 01.01.1970
	data = buffer_to_ndarray(
	dbuf,
	(
	DtypeKind.UINT,
	dtype[1],
	getattr(ArrowCTypes, f"UINT{dtype[1]}"),
	Endianness.NATIVE,
	),
	offset=col.offset,
	length=col.size(),
	)

I think it should be like it is for categorical dtype:

pandas/pandas/core/interchange/from_dataframe.py

Lines 219 to 222 in 329fe8f

	codes_buff, codes_dtype = buffers["data"]
	codes = buffer_to_ndarray(
	codes_buff, codes_dtype, offset=col.offset, length=col.size()
	)

However, I don't think we can just change the datetime one to be

    data_buff, dtype = buffers["data"]
    data = buffer_to_ndarray(data_buff, dtype, offset=0, length=data_buff.bufsize)

immediately, as then that would break because other libraries' implementations still return datetime as the buffer dtype

So, we may need to proceed in two stages:

1. from_dataframe should use the buffer dtype, with a temporary workaround to convert the (currently wrong) buffer dtype to the correct one
2. update implementations' buffer dtype to be correct
3. remove the temporary workaround from from_dataframe

The above process may have to be really slow here in order to avoid breakages. But, I think it should be done

[stinodego]

The way I see it: Implementations of from_dataframe should just disregard the data buffer dtype entirely. column.dtype already tells you what to expect in the data buffer (e.g. column dtype STRING will mean an 8bit UINT data buffer).

If it's implemented like this (everywhere), the data buffer dtype can be changed without breaking from_dataframe.

[stinodego]

Thanks for picking this up @MarcoGorelli ! Is it perhaps worth to keep this open for "part 3" (updating the data buffer dtypes)?

[MarcoGorelli]

yup, good one

[MarcoGorelli]

brought this up recently, and people were ok with just breaking behaviour for 3.0, which is a few months away