Optimize std::str::Chars::next and std::str::Chars::next_back

[Kmeakin]

Before/after for next: https://godbolt.org/z/9Te84YzhK
Before/after for next_back: https://godbolt.org/z/fTx1a7oz1

std::sys_common::wtf8::Wtf8CodePoints will also benefit from this, since it uses the same next_code_point and next_code_point_reverse functions internally.

I also added tests for all codepoints in the range 0..=char::MAX (including surrogats that can only appear in WTF-8), so the new implementations have been exhaustively tested

[rustbot]

r? @scottmcm

rustbot has assigned @scottmcm.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[Kmeakin]

ping @scottmcm ?

[scottmcm]

So you know, you can make diff views in godbolt: https://godbolt.org/z/Thn1bf9qG

[scottmcm]

The general structure here does make sense to me, but overall I feel like it removed a bunch of helpers and constants unnecessarily. Not having utf8_first_byte, sure, but this ends up repeating the X << 6 | (Y & 0x3F) in a bunch of places, so keeping the utf8_acc_cont_byte to do that would make sense to me. The standard library is always compiled with optimizations, and the MIR inliner will inline it, so there's no reason to avoid the function call. Having the u32::from in there would also make the two functions more similar, since now the forward one is using as u32 in a different line instead with no obvious reason whey they should differ.

[Kmeakin]

The general structure here does make sense to me, but overall I feel like it removed a bunch of helpers and constants unnecessarily. Not having utf8_first_byte, sure, but this ends up repeating the X << 6 | (Y & 0x3F) in a bunch of places, so keeping the utf8_acc_cont_byte to do that would make sense to me. The standard library is always compiled with optimizations, and the MIR inliner will inline it, so there's no reason to avoid the function call. Having the u32::from in there would also make the two functions more similar, since now the forward one is using as u32 in a different line instead with no obvious reason whey they should differ.

I could not get LLVM to produce the movzx even with various combinations of assume and disjoint_bitor. I'll file an issue against LLVM instead of putting micro-optimizations like that in Rust

[Kmeakin]

r? @scottmcm

[rustbot]

Requested reviewer is already assigned to this pull request.

Please choose another assignee.