[MRG+1] Use w3lib.url.canonicalize_url() from w3lib 1.15.0

[ashkulz]

fixes #2157

[kmike]

Thanks!

It seems a few other places needs to be changed:

• setup.py file needs a version bump for w3lib;
• it looks like some code in scrapy.utils.url was only required for canonicalize_url - I wonder if we can drop it now.

[codecov-io]

Current coverage is 83.40% (diff: 100%)

Merging #2168 into master will decrease coverage by 0.05%

Powered by Codecov. Last update 9a734e6...bb3b806

[ashkulz]

Thanks, incorporated your feedback. Did a rebase, as not sure whether an additional commit would be acceptable (might be a good idea to incorporate what is recommended as a part of docs/contributing.rst).

[kmike]

@ashkulz: it is good to have a readable commits history, thanks for taking time to do the rebase!

Currently we are not too strict on rebase vs extra commits - sometimes we ask for a rebase when a commit history gets messy, but not always. Probably that's why it is not in contributing.rst - we haven't agreed on a policy, it is just a common sense now.

[kmike]

I believe this import is needed for backwards compatibility

[kmike]

hmm, this function was here for a long time; I wonder if we should import it from w3lib.url to avoid breaking user code

[kmike]

other removed functions were here for much less time; I think it is fine to remove them as it is unlikely removal will break anything

[ashkulz]

@kmike: sorry for the delay. Is there a necessity to keep _safe_chars and _unquotepath, as those are supposed to be internal interfaces for which backwards compatibility may not apply?

[redapple]

@ashkulz , we sometimes break this guideline to be on the safe side with what users import.
We may have to live with this for some time.
Another thing to do (in another PR) could be to add a warning in the release notes that a later version may not expose _safe_chars and _unquotepath.

[ashkulz]

@redapple: any other feedback? I've already pushed a rebased commit which includes this change, unless you want me to include an entry in docs/news.rst in this PR itself.

[redapple]

@ashkulz , no other feedback. Looks good to me!
Thanks again.

[kmike]

Thanks @ashkulz!