infer_schema_length=None fails with unexpected type for nested data

[theelderbeever]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

example_data = [
    {
        "customer": "customer_1",
        "summaries": [
            {
                "id": "summary_1",
                "object": "object_1",
                "aggregated_value": 1000.0,
                "end_time": 1625155200,
                "livemode": True,
                "meter": "meter_1",
                "start_time": 1625078800,
            },
            {
                "id": "summary_2",
                "object": "object_2",
                "aggregated_value": 2000,
                "end_time": 1625241600,
                "livemode": False,
                "meter": "meter_2",
                "start_time": 1625165200,
            }
        ]
    },
    {
        "customer": "customer_2",
        "summaries": [
            {
                "id": "summary_3",
                "object": "object_3",
                "aggregated_value": 3000,
                "end_time": 1625328000,
                "livemode": True,
                "meter": "meter_3",
                "start_time": 1625251600,
            }
        ]
    }
]

pl.DataFrame(example_data, infer_schema_length=None)

Log output

❯ POLARS_VERBOSE=1 python notebooks/test.py
Traceback (most recent call last):
  File "/Users/taylorbeever/git/quiknode-labs/billing/billing-platform-pipelines/notebooks/test.py", line 43, in <module>
    pl.DataFrame(example_data, infer_schema_length=None)
  File "/Users/taylorbeever/.pyenv/versions/billing-platform-pipelines/lib/python3.11/site-packages/polars/dataframe/frame.py", line 366, in __init__
    self._df = sequence_to_pydf(
               ^^^^^^^^^^^^^^^^^
  File "/Users/taylorbeever/.pyenv/versions/billing-platform-pipelines/lib/python3.11/site-packages/polars/_utils/construction/dataframe.py", line 437, in sequence_to_pydf
    return _sequence_to_pydf_dispatcher(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taylorbeever/.pyenv/versions/3.11.8/lib/python3.11/functools.py", line 909, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taylorbeever/.pyenv/versions/billing-platform-pipelines/lib/python3.11/site-packages/polars/_utils/construction/dataframe.py", line 678, in _sequence_of_dict_to_pydf
    pydf = PyDataFrame.from_dicts(
           ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unexpected value while building Series of type Float64; found value of type Int64: 2000

Hint: Try setting `strict=False` to allow passing data with mixed types.

Issue description

Polars fails to correctly infer the datatype of a nested struct even with infer_schema_length=None. The column in the example that is failing is the aggregated_value field in the List(Struct( ... )).

Expected behavior

infer_schema_length should apply to nested types as well.

Installed versions

--------Version info---------
Polars:               0.20.30
Index type:           UInt32
Platform:             macOS-14.4.1-arm64-arm-64bit
Python:               3.11.8 (main, Apr 27 2024, 07:50:56) [Clang 15.0.0 (clang-1500.3.9.4)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          2.2.1
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2023.12.2
gevent:               <not installed>
hvplot:               0.9.2
matplotlib:           3.8.4
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               2.2.2
pyarrow:              <not installed>
pydantic:             2.5.3
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.29
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[cmdlineluser]

Can reproduce.

pl.DataFrame({"A": [[[1.0], [2]]]})
# shape: (1, 1)
# ┌─────────────────┐
# │ A               │
# │ ---             │
# │ list[list[f64]] │
# ╞═════════════════╡
# │ [[1.0], [2.0]]  │
# └─────────────────┘

pl.DataFrame({"A": [[{"B":1.0}, {"B":2}]]})
TypeError: unexpected value while building Series of type Float64; found value of type Int64: 2

INFER_SCHEMA_LENGTH is hardcoded to 25 here, but it doesn't seem to come into play:

• polars/py-polars/src/conversion/any_value.rs

  Lines 289 to 290 in bfcc1ee

  	} else if ob.is_instance_of::<PyList>() | ob.is_instance_of::<PyTuple>() {
  	const INFER_SCHEMA_LENGTH: usize = 25;

The issue seems to be that structs are treated differently to other types.

• polars/crates/polars-core/src/series/any_value.rs

  Line 485 in bfcc1ee

  fn any_values_to_list(

• polars/crates/polars-core/src/series/any_value.rs

  Line 622 in bfcc1ee

  fn any_values_to_struct(

e.g. inside to_list there is an explicit cast:

• polars/crates/polars-core/src/series/any_value.rs

  Lines 514 to 515 in bfcc1ee

  	} else {
  	match b.cast(inner_type) {

But to_struct ends up calling from_any_values_and_dtype again on the inner values:

• polars/crates/polars-core/src/series/any_value.rs

  Line 688 in bfcc1ee

  Series::from_any_values_and_dtype(field.name(), &field_avs, &field.dtype, strict)?

So in this case, we end up with a strict call on the inner values that fails.

Series::from_any_values_and_dtype("name", [1.0, 2], Float64, true)