[Python] Write_dataset() run time does not scale linearly with dataset size

[lmocsi]

Describe the bug, including details regarding any error messages, version, and platform.

I'd like to create a sample hive-partitioned dataset in parquet format.
Parameter "b" controls the amount of data.
b = 1010000 -> 10000 customer ids / 7440000 records -> Runs in 2 seconds
b = 1050000 -> 50000 customer ids / 37200000 records -> Runs in 8 minutes
b = 1100000 -> 100000 customer ids / 74400000 records -> Runs in 63 minutes
Shouldn't it scale linearly?

# Windows 11
#!pip install --upgrade polars==0.20.5
#!pip install --upgrade pyarrow==15.0.0
 
import polars as pl
import pyarrow.dataset as ds
 
from dateutil import rrule
from datetime import datetime

def ido():
    return datetime.now().strftime('%Y.%m.%d. %H:%M:%S')

print(ido(),'started')

a = 1000000
b = 1010000 # Runs in 2 seconds
#b = 1050000 # Runs in 8 minutes
#b = 1100000 # Runs in 63 minutes
df1 = pl.DataFrame({'PARTY_ID': [i for i in range(a, b)]})
df2 = pl.DataFrame({'CALENDAR_DATE': [datetime.strftime(i,'%Y-%m-%d %H:%M:%S') for i in list(rrule.rrule(rrule.DAILY, count=186, dtstart=datetime(2023, 7, 21)))]})
df3 = pl.DataFrame({'CREDIT_FL': ['Y','N','Y', 'Y'],
                    'AMOUNT': [123, 789, 22, 44]})
 
df4 = (df1.join(df2,
                how='cross'
                )
          .join(df3,
                how='cross'
                )
       )
print(ido(),'data created')
print(ido(),df4.shape)

dft = df4.to_arrow()
print(ido(),'data converted to arrow')

ds.write_dataset(
        dft,
        'my_table',
        format="parquet",
        partitioning=["CALENDAR_DATE"],
        partitioning_flavor="hive",
        existing_data_behavior="delete_matching",
    )

print(ido(),'finished')

Component(s)

Python