add Iterators.partition

[bkamins]

Having this function would be convenient in cases like https://discourse.julialang.org/t/arrow-stream-usage-clarification/89508.

With this PR adding new functionalities for 1.5 release starts. I will manage patches to 1.4.x releases in a separate branch if they are needed.

[bkamins]

CI failure is unrelated.

[bkamins]

Thank you!

I will wait with merging this. First I will finalize #3213 and make a patch release (the problem #3213 fixes is pretty common in combination with CSV.jl).

[quinnj]

This is a great addition!

[bkamins]

@quinnj - as you probably know I want to add it to allow easy splitting of a data frame into record batches for Arrow.jl.

[quinnj]

Yeah, I saw the discourse post. I wonder if we should add a note about Tables.partitions as well. I.e. doing Arrow.write(filename, Tables.partitions(Iterators.partition(df, n))) will work; is that the recommended invocation you have in mind?

[bkamins]

Ah - you are right. I need to change the implementation so that Iterations.PartitionIterator is returned, as I see that Tables.partition does not recognize any Generator, but requires a specific one. Thank you!

[bkamins]

After this change it will be enough to write Arrow.write(filename, Iterators.partition(df, n)), as Arrow.write calls Tables.partitions internally anyway.

[quinnj]

After this change it will be enough to write Arrow.write(filename, Iterators.partition(df, n)), as Arrow.write calls Tables.partitions internally anyway.

Ah yes, you're right. Nice; that's very simple/clean.

[bkamins]

OK - I have made the changes and tested that Arrow.write works as expected. Thank you!

@nalimilan - as usual (no rush) - it would be great if you had a look at the PR again.

[bkamins]

Thank you! I will merge this after we make DataFrames.jl 1.4.4 release.

[bkamins]

Thank you!