-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Properly handle non-UTF8 percent encoded characters in query part #19
Comments
Fixed in yarl 0.6.0 |
Just FYI: in URLs path and fragment are always encoded to UTF8 and percent-escaped, while query parameters are encoded to page encoding (i.e. to encoding of a page URL was extracted from) and percent-escaped. So you can't extract an url like 'http://example.com?имя=вася' from a web page as an unicode string and then percent-escape it (e.g. to send an HTTP request) without specifiying page encoding. Likewise, if the URL is already percent-escaped in page body then you need to know original page encoding in order to unescape it, even if you already have page body as unicode. |
Sorry, I don't follow you. On decoding already encoded symbols in non-UTF8 coding just are leaved as is. |
My point is that this works if you're using yarl to prepare URL for sending to a server which uses UTF-8, but it doesn't work if server uses another encoding to serve its HTML pages. It means that currently yarl can't be used in the following case:
Maybe it is not related directly to this issue, sorry if I misread it. |
Even if decoding is used only for displaying it is good to have it correct :) |
It's not related directly I believe but you've raised interesting question.
Could you point on RFC for this rule? Or is it only de-facto behavior? |
Yep, I think that's the way to go. It should also be made clear that this encoding is an encoding of a page URL was extracted from. Path and fragment should still use UTF8, not this encoding, and hostname should still use IDNA.
It is a de-facto behavior of all browsers; as all other such gotchas, this is written down in https://url.spec.whatwg.org/ (see https://url.spec.whatwg.org/#query-encoding-example). |
See for the reference aio-libs/aiohttp#1364
The text was updated successfully, but these errors were encountered: