RFC-4638: Executor

[Xuanwo]

Add executor in opendal to allow running tasks concurrently in background.

+ let exec = Executor::new();
  let w = op
      .writer_with(path)
      .chunk(8 * 1024 * 1024) // 8 MiB per chunk
      .concurrent(16) // 16 concurrent tasks
+     .executor(exec) // Use specified executor
      .await?;

[hzxa21]

Thanks for proposing the RFC. This is very useful to our use case.

I have two questions:

1. When the task fails during execution, will the opendal retry layer (if specified) take effect?
2. When the task takes too long, will the opendal timeout layer (if specified) take effect?

[Xuanwo]

1. When the task fails during execution, will the opendal retry layer (if specified) take effect?

Yes. Failed task will be re-created and re-executed.

2. When the task takes too long, will the opendal timeout layer (if specified) take effect?

Sadly, timeout layer can't control the task managed by executor. One possible workaround is users wrap the future with their own timeout. I'm still working on better solutions.

[Xuanwo]

Sadly, timeout layer can't control the task managed by executor. One possible workaround is users wrap the future with their own timeout. I'm still working on better solutions.

Aha, I got an idea of how to fix it. TimeoutLayer will inject the timeout directly into the OpWrite. So the answer is YES. All layers will work.

[Xuanwo]

cc @tisonkun, would you like to take a review of this RFC?

[tisonkun]

@Xuanwo OK. I'll review this today.

[tisonkun]

LGTM.

I suggest you to take a look at async-executor's design and see if we can reuse the abstraction at some level.

There is also executor-trait that can be helpful.

[tisonkun]

LGTM.