Tracking Issue for RFC 2930 (read-buf)

[nikomatsakis]

This is a tracking issue for the RFC "2930" (rust-lang/rfcs#2930).
The feature gate for the issue is #![feature(read_buf)].

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC
  • Initial implementation in Implement most of RFC 2930, providing the ReadBuf abstraction #81156
  • Vectorized APIs (read_buf_vectored, ReadBufs).
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization PR (see instructions on rustc-dev-guide)
  • Note that this is a use of rustc_must_implement_one_of, which is a language-observable thing, and thus needs some amount of oversight/documentation about that as a prerequisite of stabilization.

Unresolved Questions

• Should read_buf return the number of bytes read like read does or should the ReadBuf track it instead? Some operations, like checking for EOF, are a bit simpler if read_buf returns the value, but the confusion around what is and is not trustworthy is worrysome for unsafe code working with Read implementations.
• What should assume_init be named?
• Should the API use a wrapper around &mut ReadBuf to prevent unexpected swapping of the caller-provided ReadBuf?
• Resolve soundness issue: Unsound BufWriter copy_to specialization with the unstable read_buf feature #93305

Implementation history

• Implement most of RFC 2930, providing the ReadBuf abstraction #81156

[nikomatsakis]

( cc @rust-lang/libs )

[beepster4096]

I'm interested in working on this.

@rustbot claim

[beepster4096]

I am slightly confused by the API of ReadBufs. What return type should methods like initialized have: &[u8] or &[IoSlice]? If its the former, how do you know which slices are initialized/filled? If it's the latter, what happens if a slice is partially initialized/filled?

edit: probably I should ask this on zulip instead of github

[sfackler]

It would return &[IoSliceMut], and only include the slices that are fully initialized.

[programmerjake]

It would return &[IoSliceMut], and only include the slices that are fully initialized.

I would have expected it to return (&[IoSliceMut], &[u8]) where it is some number of fully initialized buffers and the initialized portion of the partially initialized buffer.

[sfackler]

Sure, that makes sense.

[sfackler]

@drmeepster are you still working on this?

[beepster4096]

Yeah, I am. Although I'm currently working on #79607 because I needed it for this.

[beepster4096]

Okay, I can continue working on this now that we have MaybeUninit::write_slice

[sunshowers]

I have a PR to add an inner_mut method to Tokio's implementation of ReadBuf: tokio-rs/tokio#3443. As far as I can tell it's a valid use case that there's no other way to do short of pointer arithmetic, so it may make sense to have this be in upstream Rust's ReadBuf as well.

[erickt]

Have we considered extending ReadBuf to be generic on the type, rather than be constrained to u8? I'm guessing much of ReadBuf is generic over the type.

This came up because I'm looking into fixing some UB in rayon, which is passing around uninitialized &mut [T] in its collect iterator. I think the main thing making rayon use this over a &mut Vec<T> is that it wants to split the output buffer across threads, but there's no safe way to do this without initializing the slice. I'd like to replace this with a safe abstraction that's probably quite similar to the design proposed for ReadBuf (plus a split_at method), so I thought maybe there are other people potentially interested in this functionality.

Going further, it would also be interesting to see if we could rewrite Vec to sit upon ReadBuf.

[djc]

I made very similar points here:

https://internals.rust-lang.org/t/readbuf-as-part-of-rust-edition-2021/14256/9?u=djc

[Amanieu]

Note that we now have a spare_capacity_mut method on Vec which gives you a &mut [MaybeUninit<T>] for the uninitialized part of a vector.

[ChayimFriedman2]

Currently, if you Write::write() into BorrowedCursor more space than it can hold, it panics like if you call append(). I'd expect that it write only part of the data for write() and err with ErrorKind::WriteZero for write_all(), like impl Write for &mut [u8] does.

[Pr0methean]

Could we please have a shared interface trait for Seek::stream_position and BorrowedBuf::len? That would help in adapting library crates such as zip_next to use a BorrowedBuf (which would make it possible to read back one compressed file while writing another without loading the whole file into memory).

[CAD97]

I just want to note that while rustc_must_implement_one_of is a prerequisite to stabilizing Read::read_buf, BorrowedBuf/BorrowedCursor are separately useful even without the existence of Read::read_buf, so could be stabilized separately. I happen to be writing some unsafe APIs which could be made meaningfully safer with the use of BorrowedCursor.

[jmillikin]

I'm interested in BorrowedBuf and BorrowedCursor for use in no_std environments. Would it be possible to move them into core and stabilize them separately from the new std::io::{Read,Write} functionality?

The current implementation of those types has no dependency on std, and I'd be happy to send out the PRs if the Rust folks are willing to review them.

[the8472]

The BorrowedCursor documentation could use some polish. It says some slightly contradictory or misleading things. The docs start with

A writeable view of the unfilled portion of a BorrowedBuf.

but the very next paragraph:

Provides access to the initialized and uninitialized parts of the underlying BorrowedBuf.

Looking at the actual implementations makes it obvious that they mostly pass-through and grab slices from the underlying BorrowedBuf and totally ignore the write position (start). The write position is only relevant for written().

[tgross35]

Does core_io_borrowed_buf really need to be a separate feature gate, or could it be rolled into read_buf? Bit confusing that the BorrowedBuf/BorrowedCursor docs all point to #117693 rather than this issue, assuming the same types are designed to work in core.

Also related to @the8472's comment, docs need examples.

[a1phyr]

There is something "fun" with read_buf as it is defined today, but I am not sure it was done on purpose: unlike read, it is possible to return both data and an error.
This is possible because the cursor keeps tracks of the read bytes itself, so returning an error is not incompatible with reading bytes.

I don't know if this is expected and it is probably useful, but it might be surprising if some implementations start doing that.

In fact, read_buf_exact documents that it has such a behavior:

If this function returns an error, all bytes read will be appended to cursor.

Something more problematic is that it is impossible to write a correct read implementation on top of read_buf (either the written bytes or the error would have to be discarded).

[ChrisDenton]

Tbh, I think read is in the wrong here. For example, I/O reads very much depends on the behaviour of the underlying OS which we have no control over. Currently we are indeed forced to discard any OS errors if any bytes are read.