struct field access returns incorrect values

[cpcloud]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import pandas as pd
import polars as pl

df = pl.DataFrame(pd.DataFrame({"abc": [{"a": 1.0, "b": pd.NA}, pd.NA]}))

res = df.select(a=pl.col("abc").struct.field("a"))

raw_py = res.to_dicts()
assert raw_py == [{"a": 1.0}, {"a": None}]

Log output

No response

Issue description

struct field access now returns incorrect values for rows where the entire struct field is null, but seemingly only when the input data is a pandas DataFrame.

Expected behavior

I would expect entire-NULL rows to return null for any struct operation, including field access.

Installed versions

--------Version info---------
Polars:               1.2.1
Index type:           UInt32
Platform:             Linux-6.6.41-x86_64-with-glibc2.39
Python:               3.10.14 (main, Mar 19 2024, 21:46:16) [GCC 13.3.0]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          3.0.0
connectorx:           <not installed>
deltalake:            0.18.2
fastexcel:            <not installed>
fsspec:               2024.6.1
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           3.9.1
nest_asyncio:         1.6.0
numpy:                2.0.0
openpyxl:             <not installed>
pandas:               2.2.2
pyarrow:              17.0.0
pydantic:             2.8.2
pyiceberg:            <not installed>
sqlalchemy:           2.0.31
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[MarcoGorelli]

yup, thanks @cpcloud for the report

[cpcloud]

FWIW this was introduced between 1.1.0 and 1.2.1, if that helps narrow it down. I caught this when upgrading Ibis dependencies.

[ritchie46]

Thanks. We completely refactored the struct dtype, so I have a pretty good indication when it happened. ^^