LazyFrame select in 0.20.31 includes hive partition column even when not in specified columns

[theogaraj]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

Create a hive-partitioned parquet file:

>>> import polars as pl
>>> data = {
...     "a": [1, 2, 3, 4, 5],
...     "b": [5, 4, 3, 2, 1],
...     "x": [1,1,1,2,2]
... }
>>> df = pl.DataFrame(data)
>>> print(df)
shape: (5, 3)
┌─────┬─────┬─────┐
│ a   ┆ b   ┆ x   │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1   ┆ 5   ┆ 1   │
│ 2   ┆ 4   ┆ 1   │
│ 3   ┆ 3   ┆ 1   │
│ 4   ┆ 2   ┆ 2   │
│ 5   ┆ 1   ┆ 2   │
└─────┴─────┴─────┘
>>> df.write_parquet('mydata/Dataframe', use_pyarrow=True, pyarrow_options={'partition_cols': ['x']})

Now, scan the parquet file and select just columns a and b:

>>> import polars as pl
>>> df2 = pl.scan_parquet('mydata/Dataframe/x=*/*.parquet')
>>> print(df2.select('a', 'b').collect())
shape: (5, 3)
┌─────┬─────┬─────┐
│ a   ┆ b   ┆ x   │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1   ┆ 5   ┆ 1   │
│ 2   ┆ 4   ┆ 1   │
│ 3   ┆ 3   ┆ 1   │
│ 4   ┆ 2   ┆ 2   │
│ 5   ┆ 1   ┆ 2   │
└─────┴─────┴─────┘

Log output

No response

Issue description

I would expect that column x should not be selected since I didn't specify it in the select.

Expected behavior

If I downgrade to version 0.20.15 it behaves as expected, code below is with v0.20.15:

>>> import polars as pl
>>> df2 = pl.scan_parquet('mydata/Dataframe/x=*/*.parquet')
>>> print(df2.select('a', 'b').collect())
shape: (5, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 5   │
│ 2   ┆ 4   │
│ 3   ┆ 3   │
│ 4   ┆ 2   │
│ 5   ┆ 1   │
└─────┴─────┘

Installed versions

--------Version info---------
Polars:               0.20.31
Index type:           UInt32
Platform:             Windows-10-10.0.22631-SP0
Python:               3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         <not installed>
numpy:                2.0.0
openpyxl:             <not installed>
pandas:               <not installed>
pyarrow:              16.1.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[ritchie46]

This is already fixed on main.

[theogaraj]

@ritchie46 thanks for the prompt response!
Can you please clarify what "fixed on main" means? If I simply do pip install polars I get v0.20.31, which exhibits the behavior described above. Do I need to install differently to get the fix you mentioned?