Tracking issue for UTF-16 decoding iterators

[SimonSapin]

#27808 proposes exposing in std::char two iterator adaptors Utf16Decoder and Utf16LossyDecoder. This functionality was previously only available with an API that require allocation (String::from_utf16{,_lossy}) or using the unstable rustc_unicode crate directly.

They are exposed unstable with a new utf16_decoder feature name. I’d like to stabilize them when we’re confident with the naming and API.

[alexcrichton]

🔔 This issue is now entering its cycle-long final comment period for stabilization in 1.9 🔔

[alexcrichton]

My personal opinion:

Ok, so the APIs that this is talking about are:

• std::char::decode_utf16
• std::char::DecodeUtf16
• impl Iterator<Item=Result<char, u16>> for DecodeUtf16
• std::char::REPLACEMENT_CHARACTER

I would not personally be a huge fan of stabilizing this cycle, some concerns being:

• If we have decode_utf16, shouldn't we have decode_utf8? We have String::from_utf16{,_lossy} to mirror str::from_utf8{,_lossy}.
• Do we want to support decode_utfXX_lossy? Are they easy enough to do with a .map(|e| e.unwrap_or(REPLACEMENT_CHARACTER))?
• I don't think we have many other top-level functions in the standard library that take iterators and return iterators?
• Having only one char constant in the module seems kinda odd, surely there are more constant-like things (like newline?) which we may wish to have?
• The error type of the iterator may be a little unidiomatic, we probably want to wrap it up in something to provide extra contextual information in the future.

I think I'd be more comfortable with stabilizing given some utf-8 decoding functions as well, but it's probably worth also looking at the matrix of conversions we have:

A	B	A -> B	B -> A
Iterator<u8>	Iterator<char>	not implemented	`flat_map(
Iterator<u16>	Iterator<char>	char::decode_utf16	`flat_map(
&[u8]	&str	str::from_utf8	str::as_bytes
&[u16]	String	String::from_utf16	impossible
Iterator<u16>	str	impossible	str::encode_utf16

I guess in that sense we'd be "complete" with decode_utf8? Is that all we need to put this issue to rest?

[aturon]

@alexcrichton

I don't think we have many other top-level functions in the standard library that take iterators and return iterators?

I don't see this as a concern, really. The more we work with iterators, the more natural things like "iterator transformers" such as this are. It's a fine ball to get rolling in my opinion.

The error type of the iterator may be a little unidiomatic, we probably want to wrap it up in something to provide extra contextual information in the future.

Agreed.

Having only one char constant in the module seems kinda odd, surely there are more constant-like things (like newline?) which we may wish to have?

Perhaps so -- but IMO this should influence organization more than anything. That is, we might want to think about a submodule for constants, if we do anticipate adding more over time.

Most of the other points have the flavor of: why stabilize just this one piece? I agree that I'd really like to have an overall vision here; I feel like every cycle we stabilize a couple of related methods. That said, your matrix is pretty useful, and does indeed suggest we should land both this and an analogous utf8 decoder.

[BurntSushi]

I would be in favor of postponing until we have a more complete vision here as well (I'm a consumer of UTF-8 functions like this, which I typically just implement in-crate). There may be other routines we want to consider as well, for example, decoding a UTF-8 sequence in reverse can often be useful.

[SimonSapin]

• Do we want to support decode_utfXX_lossy? Are they easy enough to do with a .map(|e| e.unwrap_or(REPLACEMENT_CHARACTER))?

Since decode_utf16() has a dedicated return type this could also be decode_utf16().lossy(). IMO this is nicer than a function that would need to be imported separately. FWIW I had this in the PR originally and removed it at your suggestion :) #27808 (comment)

• Having only one char constant in the module seems kinda odd, surely there are more constant-like things (like newline?) which we may wish to have?

We can remove the constant if lossy decoding is built-in.

I guess in that sense we'd be "complete" with decode_utf8? Is that all we need to put this issue to rest?

Yes, I do think we’re missing something lower-level than we currently have in std for decoding UTF-8. Something that does not allocate (could potentially be in libcore) and is more flexible than str::from_utf8 (which is strict (not lossy) and requires the entire input to be in a single slice).

But since &str and String use UTF-8 internally I don’t think decoding to Iterator<char> is ideal. We’d like to use slices of the input when possible.

I have some experiments at https://github.com/SimonSapin/rust-utf8. (In you’re interested, the commit history shows a number of different APIs I tried.) It supports "incremental" lossy decoding: input is a number of &[u8] chunks and a single code point may span multiple chunks. Output borrows input when possible and never allocates.

But this is significantly more API surface than, say, an iterator adaptor. And there’s probably a wide variety of use cases with slightly different constraints (@BurntSushi mentioned decoding in reverse), so I don’t know if it makes sense to try and support all of them in std.

Still, it’d be nice to have a single UTF-8 decoding primitive std that everything else could be based on. That way future performance improvements like #30740 (more may be possible with SIMD) would apply automatically, ideally. Though I don’t know yet what that primitive should be exactly, or if it’s even possible.

[BurntSushi]

@SimonSapin Would fn decode_utf8(src: &[u8]) -> Result<(char, usize), ::std::str::Utf8Error> be enough? See, e.g., https://github.com/rust-lang-nursery/regex/blob/master/src/utf8.rs#L53-L64 and https://github.com/rust-lang-nursery/regex/blob/master/src/utf8.rs#L115-L117 for the reverse case. (They are both unoptimized and return an Option instead of a Result.)

[SimonSapin]

@BurntSushi Building something that yields &str slices borrowing the input on top of this API is a fair amount of extra logic, even more so to support code points spanning chunks. And it would do more work than necessary (compute actual 32-bit values rather than just figure out which byte sequences are well-formed), I don’t know if the compile could optimize that away. And this API does not enable optimizations like #30740 that decode more than one code point at once.

[alexcrichton]

The libs team discussed this during triage yesterday and the conclusion was to stabilize essentially everything as-is modulo changing the error returned by the iterator.

We felt that there's room for decoding an iterator of u8 to char, but we can always add that later.

[lfairy]

Is there any reason that these items are defined in std_unicode and not core?

I was investigating #49319 and ended up here.

[SimonSapin]

I’ll respond in #49319 since this thread has been closed for two years :)