Building a Durable Execution Engine With SQLite - Gunnar Morling

[giscus[bot]]

Building a Durable Execution Engine With SQLite - Gunnar Morling

Lately, there has been a lot of excitement around Durable Execution (DE) engines.
The basic idea of DE is to take (potentially long-running) multi-step workflows,
such as processing a purchase order or a user sign-up,
and make their individual steps persistent.
If a flow gets interrupted while running, for instance due to a machine failure,
the DE engine can resume it from the last successfully executed step and drive it to completion.

https://www.morling.dev/blog/building-durable-execution-engine-with-sqlite/

[murphye]

Great article! I have been playing with Temporal and reading up on the DE subject.

Theoretically, just for fun, would it be possible to memoize to Kafka to implement a distributed DE engine and sagas? This would replace the database for persistence. It could handle the orchestration across services somehow; I suppose there would be some tagging for the current and next step in the flow. The consumer flows need to know who responsible for the next step in execution.

[gunnarmorling]

Hey Eric, long time, no talk! So great to hear from you, I hope all is well.

Would it be possible to memoize to Kafka to implement a distributed DE engine and sagas?

In theory, yes. But as is, I don't think it would be very practical: when resuming a flow, you need to retrieve its previously executed steps efficiently from the log, using its id. Kafka doesn't support key-based look-up though, so you'd have to scan an entire partition. Or add some derived store, like RocksDB, into the mix.

[sebholstein]

Thanks for this great article!

Kafka doesn't support key-based look-up though, so you'd have to scan an entire partition.

@gunnarmorling @murphye
Nats.io would be perfect would for that. With streams and a consumer with a filtered subject pattern, this could be achieved easily (as the filtering happens on the nats server, so only relevant messages are delivered to the consumer client)

[phdbrianlee826]

thanks for this great article! is there product-ready python implementation of durable execution engine?

[gunnarmorling]

Thank you! Some production-grade DE solutions are linked within the post.

[crazysoftwarecoder]

Wonderfully written article. Reminds me of some earlier projects where I had developed frameworks and had to do a lot of meta programming to generate bytecode. Came here from linkedin - will read through your other articles.

Cheers

[gunnarmorling]

Thank you so much!