"Streaming vs. Batch" Is a Wrong Dichotomy, and I Think It's Confusing - Gunnar Morling

[giscus[bot]]

"Streaming vs. Batch" Is a Wrong Dichotomy, and I Think It's Confusing - Gunnar Morling

Often times, "Stream vs. Batch" is discussed as if it’s one or the other, but to me this does not make that much sense really.

https://www.morling.dev/blog/streaming-vs-batch-wrong-dichotomy/

[himankjog]

Can we think of streaming as processing variable sized batches?

[gunnarmorling]

Yes, I think that's a nice way to look at it.

[soaxelbrooke]

An interesting dimension that is extremely domain dependent is data availability. When you control the whole data supply chain from data source (eg app, device, etc) to ingest/aggregation/value production, streaming makes a lot more sense.

I find, though, that some domains involve many many heterogenous data sources, publishing in batches of different granularity, and with uncorrelated failure/liveness/SLAs. At any time, there are many partitions of data that we would expect to be live, but aren't, and need some grammar/organizing principal for avoiding processing that can't succeed or would provide incorrect results without that missing/not yet correct data.

How would a streaming system handle this situation? For example, "for customer X we have only received browsing data but not purchase data", and avoiding presenting incorrect metrics or producing ill informed ml models.

With batch, you can imagine using a data catalog/etc to signal data presence and calculate sufficiency for downstream data materialization, and being able to pull threads to answer why expected data is absent, and it's not clear that we can scale that concept fully to the streaming domain.

Curious to hear your thoughts 🙂

[gitnicos]

Well, all this falls under queuing. I would refrain from calling upon push/pull as delineator. And it is rather domain specific. In LLM inferencing the notion of streaming often involves micro-batching which is quite different from offline inferencing (a conventional way of batching concept).