Skip to content

Conversation

@dmsnell
Copy link
Member

@dmsnell dmsnell commented Sep 10, 2025

Trac ticket: Core-63863
See: #9825, #9830, #9498, (#9826), #9827, #9798, #9828, #9829

Introduces a new method, wp_utf8_chunks(), for iterating through a string which might contain invalid spans of UTF-8 bytes.

This function is largely a low-level function for working with text data. With it, it’s possible to operate on the valid UTF-8 portions of a string while preserving the invalid parts. For instance, this might be used to operate on improperly-constructed JSON data.

@github-actions
Copy link

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props dmsnell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@github-actions
Copy link

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

  • The Plugin and Theme Directories cannot be accessed within Playground.
  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
    it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

@dmsnell dmsnell marked this pull request as draft September 11, 2025 03:33
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch from f7e2115 to 0991d08 Compare September 11, 2025 06:37
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch from 0991d08 to 85fb018 Compare September 23, 2025 04:01
@dmsnell dmsnell marked this pull request as ready for review September 23, 2025 04:02
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch 5 times, most recently from b52768a to 9e57e41 Compare September 25, 2025 18:47
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch 5 times, most recently from 048705c to a1c7281 Compare October 6, 2025 21:18
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch 3 times, most recently from c24e21d to 424b53a Compare October 9, 2025 20:53
This new generator function iterates through valid and invalid spans
of bytes in a UTF-8 string. It is a convenience wrapper around the
new `_wp_scan_utf8()` which aids a number of operations which should
work on strings containing invalid spans of UTF-8, such as:

 - Identifying where the invalid spans are.
 - Operating on the valid portions while preserving the invalid spans.
 - Displaying broken strings.

These operations can be useful during validation, sanitization,
debugging, and processing uncontrolled inputs which may contain
malformed sequences.
@dmsnell dmsnell force-pushed the utf8/add-utf8-chunks branch from 424b53a to 77d8e4c Compare October 9, 2025 23:39
@dmsnell
Copy link
Member Author

dmsnell commented Oct 19, 2025

Without a clear and compelling need to abstract this in Core I’m closing for now. It’s easy enough to create in an application-specific way. If such a need emerges we can re-open this for later consideration.

@dmsnell dmsnell closed this Oct 19, 2025
@dmsnell dmsnell deleted the utf8/add-utf8-chunks branch October 19, 2025 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant