refactor(raw): Return written bytes in oio::Write #3005
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will add written bytes into
oio::Write
.Background
The
oio::Write
function was originally designed to ensure that both thewrite
andsink
operations write all content at once, in order to make it easier to use. However, this behavior creates a significant amount of additional work for services that need to ensure reentrancy in their writes.In some cases, such as with HDFS, implementing this behavior becomes impossible due to limitations like lack of support for seek write.
Additionally, it places a heavy burden on correctly implementing buffer logic because all operations must be reentrant. As a result, more content needs to be buffered and returned when required.
Current
This PR just adds a
written
bytes intooio::Write
, public API likeWriter::write
is not touched.Writer::sink
andWriter::copy
changed to return written bytes which should be treated as a breaking change.No services side behavior change yet. In the future, we will change the
fs
andhdfs
's implement so that we don't need to callwrite_all
and maintain internalpos
.