Recommended path for streaming ingestion into DuckLake?

[spcvalente]

Hey team 👋

First off — DuckLake is genuinely exciting. Massive congrats to the team.

I've been trying to figure out the recommended way to stream data into DuckLake and I'm hitting a wall. Here's what I found so far:

• inyo-global/ducklake-kafka-connect — looks promising but the last commit was March 2026, the S3 integration test is disabled, and from what I can tell PostHog (the main adopter?) ended up replacing it with their own thing. Doesn't feel production-ready.

• PostHog/millpond — cleaner, actively maintained, but it's one-table-per-deployment and JSON-only. Good for their use case, not general-purpose.

• ducklake_add_data_files — cool escape hatch, but basically "write Parquet externally and register it," which pushes the problem upstream.

• pg_duckpipe — neat Postgres WAL approach, but Postgres-specific obviously.

So my questions:

1. Is there an official or recommended streaming ingestion path? Something the DuckLake team would point to as "this is how you do it"?
2. Is a Flink connector on the roadmap? The write protocol spec gap (discussed in Ducklake spec clarifications needed #934) seems like the main blocker for third-party implementers — is there a plan to close that?
3. Is a Kafka Connect sink something you're planning to own/support, or is that left to the community?
4. For high-frequency inserts today (e.g. Kafka topics with thousands of msgs/sec), what's the intended architecture? Write via DuckDB JDBC from a consumer? Use data inlining + a custom flusher?

Ideally for us the answer would be Kafka — but open to whatever the team thinks is the right primitive here.

Thanks!

[staticlibs]

Hi! Don't have a real answer to the question, just a comment on this bit:

Write via DuckDB JDBC from a consumer

Using JDBC Appender to write to a in-memory table and then insert the whole table at once (tuning batch size as needed) should be a performant option.

[Alex-Monahan]

Would sql-flow fit your use case? It is a Kafka consumer!
https://sql-flow.com/docs/tutorials/ducklake-sink