Create streaming lazy dataframe from a generator

[AroneyS]

Problem description

I have a very long generator function that I want to process as a column using Polars. Due to its size, I want to run it in lazy streaming mode using the generator as a source, but I have been unable to work out how to do it (if it is possible).

Creating a normal dataframe and then converting to lazy obviously doesn't work since the generator is exhausted before the lazy plan is run with collect(). This also happens with the LazyFrame initialiser, which is just a shortcut to above.

Are there any other options that don't involve writing then scanning a csv?

Example code:

import polars as pl

def Generator():
    yield 1
    yield 2
    yield 3

generator = Generator()
df = pl.DataFrame({"a": generator}).lazy()

print(df)
# naive plan...

print([i for i in generator])
# []

generator2 = Generator()
df = pl.LazyFrame({"a": generator2})

print(df)
# naive plan...

print([i for i in generator2])
# []

Also posted https://stackoverflow.com/questions/75680581/how-to-stream-from-a-generator-to-a-polars-dataframe-and-subsequent-lazy-plan