Make document's character encoding reflect byte-order marks #5359
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change fixes a bug where document's character encoding was set to the return value of the encoding sniffing algorithm rather than to the actual encoding used, which differed when the stream started with a byte-order mark. This change incorporates BOM sniffing into the encoding sniffing algorithm, ensuring both encodings are identical. Closes whatwg#1077.
domenic
added
the
do not merge yet
domenic
reviewed
Mar 16, 2020
domenic
approved these changes
Mar 16, 2020
annevk
pushed a commit
to web-platform-tests/wpt
that referenced
this pull request
Mar 19, 2020
moz-v2v-gh
pushed a commit
to mozilla/gecko-dev
that referenced
this pull request
Mar 21, 2020
…estonly Automatic update from web-platform-tests Encoding: BOM detection For whatwg/html#5359. -- wpt-commits: 7d9b5a5facaf0b9d986e63e0a031983899d30c79 wpt-pr: 22276
xeonchen
pushed a commit
to xeonchen/gecko
that referenced
this pull request
Mar 21, 2020
…estonly Automatic update from web-platform-tests Encoding: BOM detection For whatwg/html#5359. -- wpt-commits: 7d9b5a5facaf0b9d986e63e0a031983899d30c79 wpt-pr: 22276
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-wordified
that referenced
this pull request
Mar 23, 2020
…estonly Automatic update from web-platform-tests Encoding: BOM detection For whatwg/html#5359. -- wpt-commits: 7d9b5a5facaf0b9d986e63e0a031983899d30c79 wpt-pr: 22276 UltraBlame original commit: ed0a5ea70a1f04bdad4e54f7dfc89ed9e067fe7a
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-comments-removed
that referenced
this pull request
Mar 23, 2020
…estonly Automatic update from web-platform-tests Encoding: BOM detection For whatwg/html#5359. -- wpt-commits: 7d9b5a5facaf0b9d986e63e0a031983899d30c79 wpt-pr: 22276 UltraBlame original commit: ed0a5ea70a1f04bdad4e54f7dfc89ed9e067fe7a
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-wordified-and-comments-removed
that referenced
this pull request
Mar 23, 2020
…estonly Automatic update from web-platform-tests Encoding: BOM detection For whatwg/html#5359. -- wpt-commits: 7d9b5a5facaf0b9d986e63e0a031983899d30c79 wpt-pr: 22276 UltraBlame original commit: ed0a5ea70a1f04bdad4e54f7dfc89ed9e067fe7a
annevk
reviewed
Mar 24, 2020
annevk
removed
the
do not merge yet
andreubotella
pushed a commit
to andreubotella/servo
that referenced
this pull request
Dec 30, 2020
The process of decoding the network byte stream to Unicode is backed by an instance of `encoding_rs::Decoder`, which will switch the encoding it uses if it finds a BOM in the byte stream. However, this change in encoding is not communicated back to the caller and so `document.characterSet` gives the wrong result. This change fixes that. See whatwg/html#5359 and whatwg/encoding#203 for the spec-level backing for this change.
andreubotella
pushed a commit
to andreubotella/servo
that referenced
this pull request
Dec 30, 2020
The process of decoding the network byte stream to Unicode is backed by an instance of `encoding_rs::Decoder`, which will switch the encoding it uses if it finds a BOM in the byte stream. However, this change in encoding is not communicated back to the caller and so `document.characterSet` gives the wrong result. This change fixes that. See whatwg/html#5359 and whatwg/encoding#203 for the spec-level backing for this change. Signed-off-by: Andreu Botella <abb@randomunok.com>
5 tasks
andreubotella
pushed a commit
to andreubotella/servo
that referenced
this pull request
Dec 31, 2020
The process of decoding the network byte stream to Unicode is backed by an instance of `encoding_rs::Decoder`, which will switch the encoding it uses if it finds a BOM in the byte stream. However, this change in encoding is not communicated back to the caller and so `document.characterSet` gives the wrong result. This change fixes that. See whatwg/html#5359 and whatwg/encoding#203 for the spec-level backing for this change. Signed-off-by: Andreu Botella <abb@randomunok.com>
bors-servo
added a commit
to servo/servo
that referenced
this pull request
Jan 1, 2021
Fix `document.characterSet` not reflecting byte order marks. The process of decoding the network byte stream to Unicode is backed by an instance of `encoding_rs::Decoder`, which will switch the encoding it uses if it finds a BOM in the byte stream. However, this change in encoding is not communicated back to the caller and so `document.characterSet` gives the wrong result. This change fixes that. See whatwg/html#5359 and whatwg/encoding#203 for the spec-level backing for this change. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #28005 (GitHub issue number if applicable) <!-- Either: --> - [X] There are tests for these changes OR - [ ] These changes do not require tests because ___ <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
bors-servo
added a commit
to servo/servo
that referenced
this pull request
Jan 1, 2021
Fix `document.characterSet` not reflecting byte order marks. The process of decoding the network byte stream to Unicode is backed by an instance of `encoding_rs::Decoder`, which will switch the encoding it uses if it finds a BOM in the byte stream. However, this change in encoding is not communicated back to the caller and so `document.characterSet` gives the wrong result. This change fixes that. See whatwg/html#5359 and whatwg/encoding#203 for the spec-level backing for this change. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #28005 (GitHub issue number if applicable) <!-- Either: --> - [X] There are tests for these changes OR - [ ] These changes do not require tests because ___ <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change fixes a bug where document's character encoding was set to the return value of the encoding sniffing algorithm rather than to the actual encoding used, which differed when the stream started with a byte-order mark. This change incorporates BOM sniffing into the encoding sniffing algorithm, ensuring both encodings are identical.
Closes #1077.
This PR must be merged after whatwg/encoding#203.
(See WHATWG Working Mode: Changes for more details.)
/infrastructure.html ( diff )
/parsing.html ( diff )