Do not use percent decode on strings #3111

annevk · 2017-10-10T12:54:38Z

Also stop using "UTF-8 decode without BOM or fail" and instead use "UTF-8 decode without BOM" for fragment identifiers.

Tests: ...

/browsing-the-web.html ( diff )
/infrastructure.html ( diff )

annevk · 2017-10-10T12:56:49Z

Firefox and Safari use "UTF-8 decode without BOM" here. It seems that Edge and Chrome do not. I tested with <div id=� and a fragment of #%C2 (which is illegal UTF-8). I think it's better to align with Firefox and Safari as that behavior is more consistent and potentially allows for the removal of "UTF-8 decode without BOM or fail" down the line.

cc @tkent-google

annevk · 2017-10-10T13:03:20Z

Note that we regressed defining what happens upon failure in c230f55. We could revert to that behavior instead, terminating the algorithm when decode returns failure, but I would prefer not using the "or fail" variant at all per above.

annevk · 2017-10-10T13:03:49Z

cc @alphan102 @zcorpan

annevk · 2017-10-10T13:15:49Z

While creating tests I discovered this only happens if the document encoding is UTF-8. It seems like fragments might be tied to the document encoding too somehow?

annevk · 2017-10-10T13:24:29Z

It also seems that none of the hooks defined in https://encoding.spec.whatwg.org/#specification-hooks work here, as even #%EF%BB%BF%C2 does not force UTF-8.

@hsivonen thoughts on what our options are here and what we should do?

annevk · 2017-10-10T13:40:41Z

Note that I'm not sure what browsers do here makes a whole lot of sense, given that when the URL is parsed they'll translate an input in a fragment such as U+00FF to its UTF-8 encoded equivalent. Meaning that if you have #ÿ in a URL and id=ÿ elsewhere, it'll break if the document is not UTF-8 encoded.

domenic · 2017-10-10T20:00:24Z

Sounds like there's still some work to do here, so tagging accordingly. Also happy if you want to split up the quick fixes (e.g. using the correct string-accepting algorithms) from the normative fixes.

annevk · 2017-10-11T07:13:20Z

Per my analysis in my last comment I think it would be best if we could merge this as-is and get implementations to align. I think not using UTF-8 decode here would more likely result in breakage of legacy pages. Granted, they might have been broken for quite a while now.

It would be nice to hear at least one implementation agree though before filing bugs.

domenic · 2017-10-11T15:57:59Z

OK, well, at least there will be tests so we can see how bad the mismatch with implementations is, right?

I'm a little confused how changing things to differ from how implementations work now could cause breakage of legacy pages, and it seems like the kind of thing we might need compat analysis for.

See whatwg/html#3111 for context.

hsivonen · 2017-12-18T12:36:25Z

I don't know what the appropriately reasonable and Web-compatible definition is.

As for what Gecko does, the name attribute value undergoes special treatment before ending up in the DOM, which violates the usual mapping of HTML source to the DOM. (Gecko bug.)

annevk · 2017-12-18T15:05:41Z

After further discussion with @hsivonen on IRC it turns out Firefox also exposes a "decode without BOM handling" hook (not present in the Encoding Standard) used exclusively for this (as far as web-exposed exposure goes).

It seems however that other browsers are somewhat inconsistent here. I cannot browse to an U+00FF ID using %FF on a windows-1252 page in Edge for instance. I think we could attempt to restrict this to UTF-8.

(The reason why I think that will make more legacy pages work is that the URL parser now UTF-8 encodes. It used to use the local encoding. However, decoding still uses the local encoding, which likely broke legacy pages. If they haven't been updated, switching to decoding with UTF-8 will make them work again.)

annevk · 2018-01-09T12:02:57Z

@tkent-google any thoughts on this? You didn't cover this #2902 so it'd be good to know what you think of exclusively using UTF-8.

tkent-google · 2018-01-11T07:25:00Z

It seems Google Chrome doesn't support non-UTF8 percent decoding for scroll-to-the-fragment, and I couldn't find any bug reports about non-UTF8 encoding support in our bug database. So I don't oppose exclusively using UTF-8.

See whatwg/html#3111 for context.

annevk · 2018-01-12T10:35:15Z

@tkent-google it doesn't seem that Chrome just runs UTF-8 decode though and calls it a day, per my tests in web-platform-tests/wpt#8723. Is there anything else going on?

tkent-google · 2018-01-15T00:50:14Z

I think test results indicates Chrome runs UTF-8 decode for percent encoded fragments.

fragment-and-encoding.html
location.hash = "%EF%BB%BF"; works because Chrome decodes it as UTF-8 even with charset=windows-1252
location.hash = "%FF"; doesn't work because it's not valid UTF-8.
framgent-and-encoding-2.html
location.hash = "%C2"; doesn't work because it's not valid UTF-8.

annevk · 2018-01-15T08:00:46Z

@tkent-google ah, so if you cannot decode you try to do a literal match? E.g., it would end up matching id=%FF? I was thinking you would look for U+FFFD (which is the result of decoding that byte with error handling).

tkent-google · 2018-01-16T08:49:45Z

if you cannot decode you try to do a literal match?

No. Literal matching is tried before percent decoding. It matched to the current algorithm of "determine what the indicated part of the document"

Correction of the previous comment:
I'm sorry that my previous comment wasn't correct.
I found location.hash = "%FF"; worked with Chrome. It matches to id=&#xFF. I reviewed the implementation, and the current algorithm of percent-decoding is:

UTF-8 decode without BOM or fail
If it failed, decode it as ISO-8859-1.

https://cs.chromium.org/chromium/src/url/url_util.cc?type=cs&q=DecodeURLEscapeSequences&sq=package:chromium&l=787

annevk · 2018-01-16T12:31:36Z

The way that seems to work though is that anything that was already UTF-8 decoded is kept. I don't think we want to standardize on that. That is, for %EF%BB%BF%FF you end up with U+FEFF U+00FF in Chrome, which is really weird.

Falling back to https://infra.spec.whatwg.org/#isomorphic-decode when UTF-8 decode fails seems reasonable, but we should do it across the entire input string.

annevk · 2018-01-17T10:05:45Z

@tkent-google would such a change be acceptable?

tkent-google · 2018-01-18T02:27:34Z

Falling back to https://infra.spec.whatwg.org/#isomorphic-decode when UTF-8 decode fails seems reasonable, but we should do it across the entire input string.

It's acceptable. Other browser vendors might have different opinions.

I just added counters to Chrome for mixed decoding case, isomorphic decoding case, and so on. http://crbug.com/802988

- url_util.* url::DecodeURLEscapeSequences() returns what encodings are applied on decoding. - KURL.* Add optional DecodeURLResult argument to blink:: DecodeURLEscapeSequences(). - web_feature.mojom, LocalFrameView.cpp, and enums.xml Add UseCounters. These UseCounters are for the discussion in whatwg/html#3111 . Bug: 802988 Change-Id: Ie171212a2ca97ee5dc0e5bb4eeb98463865e5ab3 Reviewed-on: https://chromium-review.googlesource.com/869696 Reviewed-by: Mike West <mkwst@chromium.org> Commit-Queue: Kent Tamura <tkent@chromium.org> Cr-Commit-Position: refs/heads/master@{#530004}

annevk · 2018-01-23T12:46:23Z

Thanks for adding the ScrollToFragment* counters to https://www.chromestatus.com/metrics/feature/timeline/popularity! I guess we'll revisit this issue in several months to see what options we have.

tkent-google · 2018-05-16T02:47:27Z

I just added counters to Chrome for mixed decoding case, isomorphic decoding case, and so on. http://crbug.com/802988

Currently data/graphs in chromestatus.com are broken. Data from the internal source:

ScrollToFragmentRequested: 30.78%
ScrollToFragmentSucceedWithRaw: 6.490%
ScrollToFragmentSucceedWithASCII: 0.009423%
ScrollToFragmentSucceedWithUTF8: 0.01023%
ScrollToFragmentSucceedWithIsomorphic: 0.001333%
ScrollToFragmentSucceedWithMixed: 0.001326%
ScrollToFragmentFailWithASCII: 27.95%
ScrollToFragmentFailWithUTF8: 1.425%
ScrollToFragmentFailWithIsomorphic: 0.2417%
ScrollToFragmentFailWithMixed: 0.2415%

Note:

These numbers are ratio of page views.
Our current code tries to find a raw fragment string without percent-decoding first.
It succeeds in 6.490% page views.
ScrollToFragment{Succeed,Fail}With{ASCII,UTF8,Isomorphic,Mixed} mean how we treated percent-decode result. ASCII means percent-decode result doesn't contain non-ASCII code. UTF8 means the result contain non-ASCII code and decode as UTF-8 successfully.

Summary:

We don't need to support isomorphic decoding, and of course chrome-specific mixed decoding.
We don't know how many pages need scroll-to-fragment with non-UTF8 page encodings. It should be less than 1.425% + 0.2417% + 0.2415%. However the failure rate of scroll-to-fragment is very high even with ASCII, and we can't guess it.

Also stop using "UTF-8 decode without BOM or fail" and instead use "UTF-8 decode without BOM" for fragment identifiers. Tests: web-platform-tests/wpt#8723.

annevk · 2018-05-22T10:31:36Z

Okay, I think that means we should proceed with this PR as-is. I rebased it so OP now includes links to readable previews/diffs and the commit message links to the tests.

See whatwg/html#3111 for context.

annevk · 2018-05-22T10:43:44Z

@domenic do you want to review this?

domenic

Editorially LGTM. I've gotten a little lost on what the tests/browser bugs situation is here, but I trust you'll take care of it.

annevk · 2018-05-23T07:17:55Z

Thanks. @tkent-google, could you review web-platform-tests/wpt#8723?

This will require subtle changes to all browsers. I file bugs once all the bits are reviewed.

* Navigation fragment decode and encodings See whatwg/html#3111 for context. * make tests more usable * address review feedback

annevk · 2018-05-23T09:30:18Z

Bugs:

… a=testonly Automatic update from web-platform-testsNavigation fragment decode and encodings (#8723) * Navigation fragment decode and encodings See whatwg/html#3111 for context. * make tests more usable * address review feedback -- wpt-commits: 5b878a1e5de29aa4e68c48e0122878f983f036ff wpt-pr: 8723

- url_util.* url::DecodeURLEscapeSequences() returns what encodings are applied on decoding. - KURL.* Add optional DecodeURLResult argument to blink:: DecodeURLEscapeSequences(). - web_feature.mojom, LocalFrameView.cpp, and enums.xml Add UseCounters. These UseCounters are for the discussion in whatwg/html#3111 . Bug: 802988 Change-Id: Ie171212a2ca97ee5dc0e5bb4eeb98463865e5ab3 Reviewed-on: https://chromium-review.googlesource.com/869696 Reviewed-by: Mike West <mkwst@chromium.org> Commit-Queue: Kent Tamura <tkent@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#530004} Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src Cr-Mirrored-Commit: dadd77a1df99121e213ab3f418fe0dfcbea6bf91

… a=testonly Automatic update from web-platform-testsNavigation fragment decode and encodings (#8723) * Navigation fragment decode and encodings See whatwg/html#3111 for context. * make tests more usable * address review feedback -- wpt-commits: 5b878a1e5de29aa4e68c48e0122878f983f036ff wpt-pr: 8723 UltraBlame original commit: 455d53e15aff62c2234c19995a0e2c5f7b5127b6

… a=testonly Automatic update from web-platform-testsNavigation fragment decode and encodings (#8723) * Navigation fragment decode and encodings See whatwg/html#3111 for context. * make tests more usable * address review feedback -- wpt-commits: 5b878a1e5de29aa4e68c48e0122878f983f036ff wpt-pr: 8723

annevk mentioned this pull request Oct 10, 2017

Consider percent-encoding more characters in "fragment state" whatwg/url#344

Closed

domenic added needs tests Moving the issue forward requires someone to write tests do not merge yet Pull request must not be merged per rationale in comment labels Oct 10, 2017

annevk added a commit to web-platform-tests/wpt that referenced this pull request Dec 18, 2017

Navigation fragment decode and encodings

fb0cf58

See whatwg/html#3111 for context.

annevk mentioned this pull request Dec 18, 2017

Navigation fragment decode and encodings web-platform-tests/wpt#8723

Merged

annevk added a commit to web-platform-tests/wpt that referenced this pull request Jan 12, 2018

Navigation fragment decode and encodings

c5e5eaa

See whatwg/html#3111 for context.

whatwg deleted a comment from bonebizz21 Jan 15, 2018

Do not use percent decode on strings

a53dfde

Also stop using "UTF-8 decode without BOM or fail" and instead use "UTF-8 decode without BOM" for fragment identifiers. Tests: web-platform-tests/wpt#8723.

annevk force-pushed the annevk/percent-decode branch from 298a597 to a53dfde Compare May 22, 2018 10:25

annevk added a commit to web-platform-tests/wpt that referenced this pull request May 22, 2018

Navigation fragment decode and encodings

12d1939

See whatwg/html#3111 for context.

annevk removed needs tests Moving the issue forward requires someone to write tests do not merge yet Pull request must not be merged per rationale in comment labels May 22, 2018

domenic approved these changes May 22, 2018

View reviewed changes

tkent-google pushed a commit to web-platform-tests/wpt that referenced this pull request May 23, 2018

Navigation fragment decode and encodings (#8723)

5b878a1

* Navigation fragment decode and encodings See whatwg/html#3111 for context. * make tests more usable * address review feedback

annevk merged commit ce8404f into master May 23, 2018

annevk deleted the annevk/percent-decode branch May 23, 2018 09:27

Do not use percent decode on strings #3111

Do not use percent decode on strings #3111

Uh oh!

Conversation

annevk commented Oct 10, 2017 • edited by pr-preview bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

annevk commented Oct 10, 2017

Uh oh!

annevk commented Oct 10, 2017

Uh oh!

annevk commented Oct 10, 2017

Uh oh!

annevk commented Oct 10, 2017

Uh oh!

annevk commented Oct 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

annevk commented Oct 10, 2017

Uh oh!

domenic commented Oct 10, 2017

Uh oh!

annevk commented Oct 11, 2017

Uh oh!

domenic commented Oct 11, 2017

Uh oh!

hsivonen commented Dec 18, 2017

Uh oh!

annevk commented Dec 18, 2017

Uh oh!

annevk commented Jan 9, 2018

Uh oh!

tkent-google commented Jan 11, 2018

Uh oh!

annevk commented Jan 12, 2018

Uh oh!

tkent-google commented Jan 15, 2018

Uh oh!

annevk commented Jan 15, 2018

Uh oh!

tkent-google commented Jan 16, 2018

Uh oh!

annevk commented Jan 16, 2018

Uh oh!

annevk commented Jan 17, 2018

Uh oh!

tkent-google commented Jan 18, 2018

Uh oh!

annevk commented Jan 23, 2018

Uh oh!

tkent-google commented May 16, 2018

Uh oh!

annevk commented May 22, 2018

Uh oh!

annevk commented May 22, 2018

Uh oh!

domenic left a comment

Choose a reason for hiding this comment

Uh oh!

annevk commented May 23, 2018

Uh oh!

annevk commented May 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

annevk commented Oct 10, 2017 •

edited by pr-preview bot

Loading

annevk commented Oct 10, 2017 •

edited

Loading