Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(raw): Return written bytes in oio::Write #3005

Merged
merged 1 commit into from
Sep 4, 2023
Merged

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Sep 4, 2023

This PR will add written bytes into oio::Write.

Background

The oio::Write function was originally designed to ensure that both the write and sink operations write all content at once, in order to make it easier to use. However, this behavior creates a significant amount of additional work for services that need to ensure reentrancy in their writes.

In some cases, such as with HDFS, implementing this behavior becomes impossible due to limitations like lack of support for seek write.

Additionally, it places a heavy burden on correctly implementing buffer logic because all operations must be reentrant. As a result, more content needs to be buffered and returned when required.

Current

This PR just adds a written bytes into oio::Write, public API like Writer::write is not touched. Writer::sink and Writer::copy changed to return written bytes which should be treated as a breaking change.

No services side behavior change yet. In the future, we will change the fs and hdfs's implement so that we don't need to call write_all and maintain internal pos.

Signed-off-by: Xuanwo <github@xuanwo.io>
Copy link
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@suyanhanx
Copy link
Member

Do you have any subsequent changes planned? Or open an issue if you can?

@Xuanwo
Copy link
Member Author

Xuanwo commented Sep 4, 2023

Do you have any subsequent changes planned? Or open an issue if you can?

I'm still exploring.

@Xuanwo Xuanwo merged commit 2c94a21 into main Sep 4, 2023
91 checks passed
@Xuanwo Xuanwo deleted the refactor-write branch September 4, 2023 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants