BUG: visual bug: pd.NA not printed properly in REPL when DataFrame gets collapsed

[dominiquegarmier]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> df = pd.DataFrame({'foo': [pd.NA for _ in range(100)]})
>>> df  # prints NaN instead of <NA>
    foo
0   NaN
1   NaN
2   NaN
3   NaN
4   NaN
..  ...
95  NaN
96  NaN
97  NaN
98  NaN
99  NaN

[100 rows x 1 columns]
>>> df.iloc[:5]  # prints <NA> as expected
    foo
0  <NA>
1  <NA>
2  <NA>
3  <NA>
4  <NA>

Issue Description

it seems that pd.NA does not get printed properly whenever the DataFrame is too large to display all rows (NaN is printed instead of <NA>). The whole thing works if you slice the DataFrame to reduce the number of printed rows.

PS: I have previously seen the "top part" being printed properly and only the "bottom part" being printed incorrectly (top and bottom refers to above and below the "...". I could not reproduce this behaviour however.

Expected Behavior

(I think?) It should print <NA> instead of NaN

Installed Versions

/.../.venv/lib/python3.10/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.11.final.0
python-bits : 64
OS : Darwin
OS-release : 23.0.0
Version : Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.24.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, confirmed on main. Further investigations and PRs to fix are welcome!

[ziadk]

Hello @rhshadrach,

I have investigated and found that the problem comes from the casting done in the following function located in pandas/core/internals/concat.py:

def _dtype_to_na_value(dtype: DtypeObj, has_none_blocks: bool):
    """
    Find the NA value to go with this dtype.
    """

    if isinstance(dtype, ExtensionDtype):
        return dtype.na_value
    elif dtype.kind in "mM":
        return dtype.type("NaT")
    elif dtype.kind in "fc":
        return dtype.type("NaN")
    elif dtype.kind == "b":
        # different from missing.na_value_for_dtype
        return None
    elif dtype.kind in "iu":
        if not has_none_blocks:
            # different from missing.na_value_for_dtype
            return None
        return np.nan
    elif dtype.kind == "0":
        return np.nan
    raise NotImplementedError

The <NA> values produce dtype.kind == 0 and thus are casted into np.nan.

I would love to contribute a PR for this issue but prefer to have some guidance before doing so.

Thanks a lot.

[ziadk]

@rhshadrach ,

To add , i have tested with replacing

elif dtype.kind == "0":
        return np.nan

with

elif dtype.kind == "0":
        return None

And the problem is resolved.

[rhshadrach]

Thanks for the investigation @ziadk - I do not think you can touch the concat code in a way that won't break other things, and that this issue with concat itself needs to be solved as part of #28095.

I took a quick look and don't see a way to modify the caller for this issue. One idea was to produce the repr of each piece without concat and then combine them as strings, but that looks to require a major refactoring of the code.

[yuanx749]

The thing is during concat, pd.NA is cast to nan for object dtype.

A workaround is to specify the dtype explicitly, e.g.
df = pd.DataFrame({'foo': [pd.NA for _ in range(100)]}, dtype="Int64")

[rhshadrach]

A workaround is to specify the dtype explicitly, e.g.

Unfortunately I don't think this can be used because the conversion to string needs to handle all possible inputs.

[dominiquegarmier]

I made a quick PR, replacing pd.concat with .drop fixes the issue since presumable .drop copies the dataframe more faithfully.

pandas/pandas/io/formats/format.py

Line 701 in a39f783

self.tr_frame = concat((head, tail))

(I hope it's reliable to slice DataFrame.index, else this whole approach is flawed)