Add explicit `Iterator::count` impl for `str::EncodeUtf16` by SimonSapin · Pull Request #159467 · rust-lang/rust

SimonSapin · 2026-07-17T21:21:10Z

In the early days of Unicode, many systems adopted it by storing in-memory strings in (roughly) Vec<u16>, encoded as UCS-2 / UTF-16 / WTF-16. The “natural” way to index such strings or measure their length is counting UTF-16 code units. In some cases, such lengths or offsets found their way in protocols or APIs, such that even a Rust implementation that uses &str internally needs to count 16-bit code units for compatibility. For example this comes up in Servo’s implementation of some DOM APIs.

One way to convert from an &str index (counted 8-bit code units) is:

let index_16bit = string[..i].encode_utf16().count();

Before this PR, the default Iterator::count relies on Iterator::next which decodes UTF-8 and encodes to UTF-16, doing more work than needed. Instead, this PR matches on bit patterns in the underlying UTF-8 string.

For comparison, str::Chars already has a sophisticated impl for Iterator::count in library/core/src/str/count.rs.

rustbot · 2026-07-17T21:21:15Z

r? @Darksonn

rustbot has assigned @Darksonn.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: libs
libs expanded to 12 candidates
Random selection from 6 candidates

SimonSapin · 2026-07-17T21:29:40Z

+    #[inline]
+    fn count(self) -> usize {


I don’t know if this should be #[inline], for now it follows the example of other methods in this impl block

View changes since the review

In the early days of Unicode, many systems adopted it by storing in-memory strings in (roughly) `Vec<u16>`, encoded as UCS-2 / UTF-16 / WTF-16. The “natural” way to index such strings or measure their length is counting UTF-16 code units. In some cases, such lengths or offsets found their way in protocols or APIs, such that even a Rust implementation that uses `&str` internally needs to count 16-bit code units for compatibility. For example this comes up in Servo’s implementation of some DOM APIs. One way to convert from an `&str` index (counted 8-bit code units) is: ```rust let index_16bit = string[..i].encode_utf16().count(); ``` Before this PR, the default `Iterator::count` relies on `Iterator::next` which decodes UTF-8 and encodes to UTF-16, doing more work than needed. Instead, this PR matches on bit patterns in the underlying UTF-8 string. For comparison, `str::Chars` already has a sophisticated impl for `Iterator::count` in `library/core/src/str/count.rs`.

joboet · 2026-07-18T09:33:41Z

Fantastic, thank you!
@bors r+
r? joboet

rust-bors · 2026-07-18T09:33:44Z

📌 Commit 630793c has been approved by joboet

It is now in the queue for this repository.

Darksonn · 2026-07-18T09:47:14Z

+        self.chars.as_str().as_bytes().iter().fold(extra, |acc, &byte| {
+            acc.wrapping_add({


This fold() looks like a manual reimplementation of sum().

View changes since the review

It is indeed. I accidentally removed it at some point but I had a comment saying this manually inlines <usize as Sum>::sum in order to use extra as the initial accumulator value, instead of using zero then adding extra

I also changed + with #[rustc_inherit_overflow_checks] to wrapping_add since this can never overflow

If you think that's better I can change it back to .map(...).sum() + extra

I suppose it's fine as-is.

Darksonn · 2026-07-18T09:52:09Z

 }

+#[test]
+fn test_utf16_count() {


How about iterating through all unicode codepoints and verifying that count() gives the same answer as manually counting with a for loop? There are only one million codepoints, so it shouldn't take long.

View changes since the review

That feels a bit excessive? There's only four kinds of UTF-8 byte sequences / relevant ranges of code points, and the tests cover each of them

rust-bors · 2026-07-18T11:02:55Z

⌛ Testing commit 630793c with merge ecb69f1...

Workflow: https://github.com/rust-lang/rust/actions/runs/29641888078

Add explicit `Iterator::count` impl for `str::EncodeUtf16` In the early days of Unicode, many systems adopted it by storing in-memory strings in (roughly) `Vec<u16>`, encoded as UCS-2 / UTF-16 / WTF-16. The “natural” way to index such strings or measure their length is counting UTF-16 code units. In some cases, such lengths or offsets found their way in protocols or APIs, such that even a Rust implementation that uses `&str` internally needs to count 16-bit code units for compatibility. For example this comes up in Servo’s implementation of some DOM APIs. One way to convert from an `&str` index (counted 8-bit code units) is: ```rust let index_16bit = string[..i].encode_utf16().count(); ``` Before this PR, the default `Iterator::count` relies on `Iterator::next` which decodes UTF-8 and encodes to UTF-16, doing more work than needed. Instead, this PR matches on bit patterns in the underlying UTF-8 string. For comparison, `str::Chars` already has a sophisticated impl for `Iterator::count` in `library/core/src/str/count.rs`.

…oboet Add explicit `Iterator::count` impl for `str::EncodeUtf16` In the early days of Unicode, many systems adopted it by storing in-memory strings in (roughly) `Vec<u16>`, encoded as UCS-2 / UTF-16 / WTF-16. The “natural” way to index such strings or measure their length is counting UTF-16 code units. In some cases, such lengths or offsets found their way in protocols or APIs, such that even a Rust implementation that uses `&str` internally needs to count 16-bit code units for compatibility. For example this comes up in Servo’s implementation of some DOM APIs. One way to convert from an `&str` index (counted 8-bit code units) is: ```rust let index_16bit = string[..i].encode_utf16().count(); ``` Before this PR, the default `Iterator::count` relies on `Iterator::next` which decodes UTF-8 and encodes to UTF-16, doing more work than needed. Instead, this PR matches on bit patterns in the underlying UTF-8 string. For comparison, `str::Chars` already has a sophisticated impl for `Iterator::count` in `library/core/src/str/count.rs`.

JonathanBrouwer · 2026-07-18T11:18:23Z

@bors yield
Yielding to enclosing rollup

rust-bors · 2026-07-18T11:18:26Z

Auto build was cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/29641888078

The next pull request likely to be tested is #159510.

…uwer Rollup of 5 pull requests Successful merges: - #159467 (Add explicit `Iterator::count` impl for `str::EncodeUtf16`) - #158545 (Move `std::io::read_to_string` to `alloc::io`) - #159328 (tests/assembly-llvm: pin frame pointer in issue-141649 aarch64 test) - #159470 (Fix string indexing in diagnostic format strings) - #159500 (Move compiletest CLI parsing to `cli.rs`)

…uwer Rollup of 8 pull requests Successful merges: - #159425 (Windows: add context when opening NUL for child stdio fails) - #159467 (Add explicit `Iterator::count` impl for `str::EncodeUtf16`) - #157860 (Move rustdoc run-make tests into new `tests/run-make/rustdoc/`) - #158545 (Move `std::io::read_to_string` to `alloc::io`) - #159328 (tests/assembly-llvm: pin frame pointer in issue-141649 aarch64 test) - #159459 (Detect when trait bound requires closure to return itself) - #159470 (Fix string indexing in diagnostic format strings) - #159500 (Move compiletest CLI parsing to `cli.rs`)

Rollup merge of #159467 - SimonSapin:encode_utf16-count, r=joboet Add explicit `Iterator::count` impl for `str::EncodeUtf16` In the early days of Unicode, many systems adopted it by storing in-memory strings in (roughly) `Vec<u16>`, encoded as UCS-2 / UTF-16 / WTF-16. The “natural” way to index such strings or measure their length is counting UTF-16 code units. In some cases, such lengths or offsets found their way in protocols or APIs, such that even a Rust implementation that uses `&str` internally needs to count 16-bit code units for compatibility. For example this comes up in Servo’s implementation of some DOM APIs. One way to convert from an `&str` index (counted 8-bit code units) is: ```rust let index_16bit = string[..i].encode_utf16().count(); ``` Before this PR, the default `Iterator::count` relies on `Iterator::next` which decodes UTF-8 and encodes to UTF-16, doing more work than needed. Instead, this PR matches on bit patterns in the underlying UTF-8 string. For comparison, `str::Chars` already has a sophisticated impl for `Iterator::count` in `library/core/src/str/count.rs`.

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 17, 2026

rustbot assigned Darksonn Jul 17, 2026

rustbot added the T-libs Relevant to the library team, which will review and decide on the PR/issue. label Jul 17, 2026

SimonSapin commented Jul 17, 2026

View reviewed changes

SimonSapin force-pushed the encode_utf16-count branch from 981fca9 to 630793c Compare July 18, 2026 07:56

rustbot assigned joboet Jul 18, 2026

rust-bors Bot added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jul 18, 2026

rustbot unassigned Darksonn Jul 18, 2026

rust-bors Bot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 18, 2026

Darksonn reviewed Jul 18, 2026

View reviewed changes

JonathanBrouwer mentioned this pull request Jul 18, 2026

Rollup of 5 pull requests #159510

Closed

JonathanBrouwer mentioned this pull request Jul 18, 2026

Rollup of 8 pull requests #159516

Merged

rust-bors Bot merged commit ae70887 into rust-lang:main Jul 18, 2026
13 of 14 checks passed

rustbot added this to the 1.99.0 milestone Jul 18, 2026

		self.chars.as_str().as_bytes().iter().fold(extra, \|acc, &byte\| {
		acc.wrapping_add({

Uh oh!

Uh oh!

Conversation

SimonSapin commented Jul 17, 2026

Uh oh!

rustbot commented Jul 17, 2026

Uh oh!

SimonSapin Jul 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joboet commented Jul 18, 2026

Uh oh!

rust-bors Bot commented Jul 18, 2026

Uh oh!

Darksonn Jul 18, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SimonSapin Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Darksonn Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Darksonn Jul 18, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SimonSapin Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

rust-bors Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonathanBrouwer commented Jul 18, 2026

Uh oh!

rust-bors Bot commented Jul 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SimonSapin Jul 17, 2026 •

edited by rustbot

Loading

Darksonn Jul 18, 2026 •

edited by rustbot

Loading

SimonSapin Jul 18, 2026 •

edited

Loading

Darksonn Jul 18, 2026 •

edited by rustbot

Loading

rust-bors Bot commented Jul 18, 2026 •

edited

Loading