[networking] Remove dot segments during URL normalization

[coletdjnz]

This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process, particularly for the urllib handler.

Closes #3355, #6526

This is adapted from the remove_dot_segments pseudo-code in the RFC and some inspiration from urllib3/rfc396 libraries (though it came out very close to them).

I have also renamed escape_url to normalize_url to better represent what it is doing, and moved these functions to utils.networking.

Template

Before submitting a pull request make sure you have:

• At least skimmed through contributing guidelines including yt-dlp coding conventions
• Searched the bugtracker for similar pull requests
• Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Fix or improvement to an extractor (Make sure to add/update tests)
• New extractor (Piracy websites will not be accepted)
• Core bug fix/improvement
• New feature (It is strongly recommended to open an issue first)

Copilot Summary

🤖 Generated by Copilot at 258810c

Summary

🚚🔧🧪

This pull request refactors and improves the URL handling functions in yt-dlp. It introduces a new function normalize_url that removes dot segments and escapes non-ASCII characters in a URL, and uses it in various modules. It also adds a new test case for the dot segment removal algorithm.

To handle URLs with more care
They moved and renamed escape_url
Now normalize_url is the tool
That removes dot segments and escapes
And makes the networking module cool

Walkthrough

• Refactor and move the functions for escaping and normalizing URLs to yt_dlp/utils/networking.py and implement the RFC 3986 5.2.4 algorithm for removing dot segments from a path (link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link)
• Add a new handler for the path /redirect_dotsegments in the HTTPTestRequestHandler class, which sends a 301 redirect response with a location header that contains dot segments (link)
• Add a new test case for the remove_dot_segments and normalize_url functions, which uses the TestHTTPRequestHandler class and the validate_and_send function to send two requests: one to the path /a/b/./../../headers and one to the path /redirect_dotsegments. The test case asserts that both requests result in a 200 status and a final URL of /headers (link)

[dirkf]

Does this handle the problem URLhttps://reoa92d.com/../uploaded/1649416469.mp4#t=0.1 from the original #3355 report? I thought this was against RFCs but maybe there's new logic (ISTR checking the WHAT spec for this when I made up the patch in that issue).

[coletdjnz]

Does this handle the problem URLhttps://reoa92d.com/../uploaded/1649416469.mp4#t=0.1 from the original #3355 report? I thought this was against RFCs but maybe there's new logic (ISTR checking the WHAT spec for this when I made up the patch in that issue).

Yes it works (though I think the domain is dead), and pretty sure it is not against standard (at least RFC3986 which browsers follow for this I believe):

RFC 3986 5.2.4 Step A:

A. If the input buffer begins with a prefix of "../" or "./",
then remove that prefix from the input buffer; otherwise,

Step C is similar for /../