Owned Buffer Design for oio::Read and oio::Write

[Xuanwo]

I've made numerous updates to oio::Read and oio::Write, learning a great deal with each refactor. Each one has been unique and incremental because we can't overhaul them all at once, and I've had to familiarize myself with the design's limitations and advantages. I believe this will be the final iteration. If it performs well, we'll aim to move this design towards release.

---

Current Design

use async fn for more readable:

pub trait Read {
    async fn read_at(&self, offset: u64, limit: usize) -> Result<oio::Buffer>;
}

pub trait ReadDyn: Unpin + Send + Sync {
    fn read_at_dyn(&self, offset: u64, limit: usize) -> BoxedFuture<Result<oio::Buffer>> 
        where Self: Sized;
}

pub trait Write: Unpin + Send + Sync {
    async fn write(&mut self, bs: Bytes) -> Result<usize>;
    async fn close(&mut self) -> Result<()>;
    async fn abort(&mut self) -> Result<()>;
}

pub trait WriteDyn: Unpin + Send + Sync {
    fn write_dyn(&mut self, bs: Bytes) -> BoxedFuture<Result<usize>>;
    fn close_dyn(&mut self) -> BoxedFuture<Result<()>>;
    fn abort_dyn(&mut self) -> BoxedFuture<Result<()>>;
}

Proposed Design

pub trait Read {
    async fn read_at<B: BufMut>(&self, buf: B, offset: u64) -> (B, Result<usize>);
}

pub trait ReadDyn: Unpin + Send + Sync {
    fn read_at_dyn<B: BufMut>(&self, buf: B, offset: u64) -> BoxedFuture<(B, Result<usize>)>;    
}

pub trait Write: Unpin + Send + Sync {
    async fn write<B: Buf>(&mut self, buf: B) -> (B, Result<usize>);
    async fn close(&mut self) -> Result<()>;
    async fn abort(&mut self) -> Result<()>;
}

pub trait WriteDyn: Unpin + Send + Sync {
    fn write_dyn<B: Buf>(&mut self, bs: B) -> BoxedFuture<(B, Result<usize>)>
        where Self: Sized;
    fn close_dyn(&mut self) -> BoxedFuture<Result<()>>;
    fn abort_dyn(&mut self) -> BoxedFuture<Result<()>>;
}

Motivation

• Remove the allocation of oio::Buffer since we can write into BufMut directly.
• Make it possible to users pass a buffer pool by accepting impl Buf and impl BufMut.
• Allowing vectored IO by default by Buf::chunks_vectored.
• Remove the extra copy while users calling opendal via futures::AsyncRead or futures::AsyncWrite.

Status

I'm working on the PoC and to test if proposed design works.

[Xuanwo]

Ok, this design doesn't work for many limiations from rust.