How to add madsim support for opendal?

[Xuanwo]

Hello, I am the maintainer of OpenDAL. This project is interesting and I would like to extend it with OpenDAL madsim support. Is there any way to do this without copying all opendal code to this repo?

[wangrunji0408]

Hi! Thank you for your interest in this project. 😄

To build a simulator of your service on madsim, you don't need to copy all existing code. Instead, you will be going to create a mock of all public APIs on top of the madsim infra. Take OpenDAL as an example. I just had a quick look of the opendal docs. It says that it is an abstraction layer over all storage services. So you only need to build a simulator that matchs the semantic of the abstraction layer (Operator and Object), and don't have to build for each service implementations. The simulator will consist of a client side and a server side. The server should emulate your service in a single-thread manner (etcd example), and it should be exposed using madsim's network API (etcd example), so that we can access it later from the client side (etcd example). All of them could be put in your crate with the #[cfg(madsim)] condition, so that when the cfg is not enabled, this crate still behaves the same as original.

For more details, you can refer to our simulation code of etcd-client, rdkafka, and aws-sdk-s3. You are also welcome to direct message me later. Looking forward to seeing madsim-opendal soon. 😜

[Xuanwo]

This issue, along with apache/opendal#1871, provides a great example of how open source works.

• madsim builds a deterministic simulator for distributed systems
• opendal will add madsim support so that all applications based on opendal can benefit from the deterministic simulator for storage.
• Existing application already uses madsim and opendal can extend their simulator to all storage without mocking all SDKs

Win, win, win!

[Xuanwo]

This task has been started in apache/opendal#2006