Use wp_is_valid_utf8() and wp_scrub_utf8() from the new utf8.php decoder #200
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er wont autoload them
dmsnell
reviewed
Nov 1, 2025
| } else { | ||
| $is_valid_utf8 = ! _wp_has_noncharacters_fallback( $blueprint_string ); | ||
| } | ||
| $is_valid_utf8 = ! wp_has_noncharacters( $blueprint_string ); |
Member
There was a problem hiding this comment.
I left a note in the other PR making this change, but it’s possible there is a misunderstanding here between valid UTF-8 and noncharacters, as noncharacters are valid UTF-8, and invalid UTF-8 cannot form noncharacters.
there is, however, wp_is_valid_utf8()
Collaborator
Author
There was a problem hiding this comment.
Ah good point, I mixed it all up. Thank you for catching this.
Collaborator
Author
|
|
adamziel
changed the title
This was referenced Nov 1, 2025
adamziel
added a commit
that referenced
this pull request
Nov 2, 2025
Speeds up XMLProcessor by consuming any ASCII bytes with `strspn` and avoiding calls to the utf8 decoder for most tags out there. The PHPUnit test suite for WXR files It speeds up parsing the 10MB WXR file in the test set from ~1.7s on average to ~0.6s on average. This PR also moves from `utf8_codepoint_at` to `_wp_scan_utf8` for UTF-8 decoding without any speed penalty – see #200 for prior context. cc @dmsnell
adamziel
added a commit
that referenced
this pull request
Nov 2, 2025
This was referenced Nov 3, 2025
Merged
adamziel
added a commit
to WordPress/wordpress-importer
that referenced
this pull request
Nov 4, 2025
Adds support for rewriting URLs inside CSS syntax, e.g. here: ```html <div style="background-image:url(/wp-content/uploads/2025/09/image-2-766x1024.jpeg)"> ``` Before this PR, the `style` attributes in, e.g., the cover block were skipped by the URL rewriter and continued pointing to the old site. Fixes #223 ## Implementation details This PR backports `CSSProcessor`, `CSSURLProcessor`, and a few related PRs around Unicode handling from the WordPress/php-toolkit repo: * WordPress/php-toolkit#197 * WordPress/php-toolkit#195 * WordPress/php-toolkit#199 * WordPress/php-toolkit#200 * WordPress/php-toolkit#201 * WordPress/php-toolkit#202 Note the CSSProcessor and CSSURLProcessor are tested against 300 test cases containing various tricky inputs, quoted and unquoted URLs, strings, comments, unicode escape sequences, and more. ## Testing instructions This PR comes with a new test case specifically for various tricky CSS inputs. You're also welcome to try and import a WXR file that contains an inline background-image reference and confirm the URL is correctly rewritten.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces two instances of the old UTF-8 decoding utilities with the new utf-8.php toolkit by @dmsnell:
This PR only touches two tactical usages of the old tools:
wp_is_valid_utf8wp_scrub_utf8instead of_wp_scrub_utf8_fallbackMore refactoring is coming once there's a faster alternative to
_wp_scan_utf8, see https://core.trac.wordpress.org/ticket/63863#comment:51Related to #196.
Follows up on #199 and #197.
Testing instructions
If the CI passes, we're good. Unicode-related scenarios are covered by tests.