You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
URLs on some sites erroneously contain valid "safe" characters in an invalid way and the standard Python library is unable to deal with this; therefore it might be nice if w3lib could. For example the hash character # normally marks the beginning of the fragment; but, it is possible that the url contains Numeric Character References (NCRs) like ® for example.
w3lib.url.safe_url_string() uses urllib.quote() with the following safe chars:
Since safe_url_string() is used in SgmlLinkExtractor, for example with canonicalization turned on, we get fragment misinterpretation as the first hash triggers the slice:
/Pioneer_Speakers_with_iPod®~iPhone&
Using urllib.quote() directly does not work since it encodes all hashes, including the fragment hash:
URLs on some sites erroneously contain valid "safe" characters in an invalid way and the standard Python library is unable to deal with this; therefore it might be nice if w3lib could. For example the hash character
#
normally marks the beginning of the fragment; but, it is possible that the url contains Numeric Character References (NCRs) like®
for example.w3lib.url.safe_url_string()
usesurllib.quote()
with the following safe chars:and the following (invalid) url does not get altered:
urlparse.urldefrag()
is confused:Since
safe_url_string()
is used inSgmlLinkExtractor
, for example with canonicalization turned on, we get fragment misinterpretation as the first hash triggers the slice:Using
urllib.quote()
directly does not work since it encodes all hashes, including the fragment hash:What is needed is perhaps a Entity/NCR regex that first converts the references and then does the safe encoding. So that in the end we get:
The text was updated successfully, but these errors were encountered: