PgBulkInsert 9.0.0: missing Stream-based saveAll overload breaks streaming ingestion use-cases

[TheNullablePrototype]

Description

After upgrading to pgbulkinsert:9.0.0, the previous ability to use saveAll with a Stream has been removed.

This makes it impossible to use the library in true streaming scenarios (e.g. reading large files row-by-row from Excel/CSV without collecting everything into memory).

In previous versions, it was possible to pipe a lazy stream directly into saveAll. In 9.0.0 only Iterable is accepted, which forces full materialization of the dataset.

This effectively breaks streaming ingestion pipelines and increases memory usage for large datasets.

Problem/Example

implementation("org.dhatim:fastexcel-reader:0.20.0")
implementation("de.bytefish:pgbulkinsert:9.0.0")

    fun example() {

        val mapper = PgMapper.forClass(Entity::class.java) // skipped map fields to columns for example
        val writer = PgBulkInsert.PgBulkWriter(mapper)

        ReadableWorkbook(Files.newInputStream(Path.of("entities.xlsx"))).use { wb ->
            wb.firstSheet.openStream().use { rows -> // <- Stream<Row>
                dataSource.connection.use {
                    // Argument type mismatch: actual type is 'Stream<T?>!', but '(Mutable)Iterable<Entity!>!' was expected.
                    writer.saveAll(it, "schema.table", rows.map { mapRowToEntity(it) }) 
                }
            }
        }
    }

Impact

• Breaks streaming ingestion workflows
• Forces collect() / materialization into memory
• Makes it unsafe for large file imports (Excel/CSV, ETL pipelines)
• Regression compared to previous API flexibility

Expected behavior
saveAll should support streaming input, e.g.:

• Stream
• or at least provide an overload:

saveAll(Connection conn, String table, Stream<T> stream)

Workaround
Currently the only option is:

writer.saveAll(conn, "schema.table", Iterable { stream.iterator() })

[bytefish]

I have never written Kotlin, but is it possible to call asIterable instead of doing a full materialization with toList, like this:

writer.saveAll(conn, "schema.table", rows.map { mapRowToEntity(it) }.asIterable()) 

Maybe I am wrong here? I was always under the impression that an Iterable does not require a full materialization.

Nevertheless, I'll add an overload for a Stream! 👍

[bytefish]

OK. I was wrong. You shouldn’t cast a Stream to an Iterable. I get it now, it breaks the promise of an Iterable:

• https://stackoverflow.com/a/59873914/513875.

This changes my perspective. I’ll add a better implementation. Bear with me, I do a lot of C# and things work differently there.

Thanks for the issue!

[bytefish]

@TheNullablePrototype I have added an overload to fix the issue! The version 9.0.1 is being published to Central Maven at the moment. Let me know if it fixes the problem, or if something is missing.

[bytefish]

It should be available:

[TheNullablePrototype]

Thanks for the quick fix and for adding Stream support.

I can confirm the updated version is now available.
I haven't had time to fully test it in our ingestion pipeline yet, but I'll do that a bit later and report back if I find any issues.

[bytefish]

Great! Just reopen if it doesn’t fix the issue. Have a great day! 👍