[core] Rework HTTP networking #2861

coletdjnz · 2022-02-23T02:02:12Z

Merge before #3668

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through contributing guidelines including yt-dlp coding conventions
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

This overhaul changes the underlying network request handling in yt-dlp.

The main goal of this was to abstract away all the network code to no longer directly interact with urllib for HTTP requests. This was so we could easily add support for more modern, updated HTTP libraries such as urllib3 to bring persistent connections, while also keeping our no-hard-dependency requirement.

TODO: https://github.com/coletdjnz/yt-dlp-dev/projects/2

Changes

The new HTTP networking interface:

Our own Request/Response/Exception classes that we have full control over.
Introduces RequestHandlers - these send supported HTTP requests and return HTTP responses
Introduces the RequestDirector - this allows us to use multiple RequestHandlers
This allows us to support multiple HTTP client libraries (each contained within a RequestHandler)

Other Improvements:

Fix issues with per request proxies on redirects for urllib
Cleaned up various legacy code
Support for ALL_PROXY env proxy setting (and all:// proxy setting to cover all protocols)
Support for socks5h (with socks5 being treated as socks5h in the ydl layer)
- Closes Please add socks5h proxy support #6325, Support for socks5h:// proxies ytdl-org/youtube-dl#22618, Accept socks5h:// proxy protocol name ytdl-org/youtube-dl#28093

Known issues / regressions

Expected: urllib (and requests) handlers check the proxy scheme to see check if it is supported.
- urllib does not support https:// proxies, so it will not be able to handle a request with one.
- Previously, urllib would erroneously treat https:// proxies as http:// proxies. It is possible some users have wrongly supplied an https:// proxy, so their configurations will break. We never supported https:// proxies and the behavior was undefined, so we should not have to support compat for this.
Expected: if a user has ALL_PROXY set for any other program, yt-dlp will now pick it up.
External extractors/downloaders etc. using urlopen() catching urllib exceptions other than HTTPError. We only provide compat for HTTPError, the others not. We could add compat but I imagine it is unlikely there would be an issue.

Future

Updating external downloaders to use ydl.proxies, clean_proxies and clean_headers
Requests support ([rh:requests] Add support for requests #3668)
RH Preference System (will be needed when adding support for curl-impersonate)
curl-impersonate handler
Websockets handler
Disable redirects for a request
Expose more settings/configuration to user
[Feature Request] Different proxy per website #1191
[AbemaTV] post network stack overhaul #3615
[feature request] QUIC (http/3) and Brotli #1817
--

This branch has been force updated. Old commit history can be found in this branch: https://github.com/coletdjnz/yt-dlp-dev/tree/network-overhaul-v2

yt_dlp/networking/utils.py

yt_dlp/utils.py

pukkandan

My thoughts after skimming through

yt_dlp/extractor/common.py

yt_dlp/utils.py

yt_dlp/YoutubeDL.py

yt_dlp/networking/_urllib.py

yt_dlp/YoutubeDL.py

test/test_InfoExtractor.py

No actual changes - code is only moved around

New networking interface consists of a `RequestDirector` that directs each `Request` to appropriate `RequestHandler` and returns the `Response` or raises `RequestError`. The handlers define adapters to transform its internal Request/Response/Errors to our interfaces. User-facing changes: - Fix issues with per request proxies on redirects for urllib - Support for `ALL_PROXY` environment variable for proxy setting - Support for `socks5h` proxy - Closes yt-dlp#6325, ytdl-org/youtube-dl#22618, ytdl-org/youtube-dl#28093 - Raise error when using `https` proxy instead of silently converting it to `http` Authored by: coletdjnz

Authored by: coletdjnz, pukkandan

pukkandan · 2023-07-15T10:55:06Z

Merged in c365dba 227bf1a 3d2623a

No actual changes - code is only moved around

New networking interface consists of a `RequestDirector` that directs each `Request` to appropriate `RequestHandler` and returns the `Response` or raises `RequestError`. The handlers define adapters to transform its internal Request/Response/Errors to our interfaces. User-facing changes: - Fix issues with per request proxies on redirects for urllib - Support for `ALL_PROXY` environment variable for proxy setting - Support for `socks5h` proxy - Closes yt-dlp#6325, ytdl-org/youtube-dl#22618, ytdl-org/youtube-dl#28093 - Raise error when using `https` proxy instead of silently converting it to `http` Authored by: coletdjnz

Authored by: coletdjnz, pukkandan

coletdjnz added enhancement New feature or request help-wanted Extra attention is needed needs-testing Patch needs testing labels Feb 23, 2022

coletdjnz mentioned this pull request Feb 23, 2022

[core/networking:urllib3] Add urllib3 backend handler [PR 2/2] #2862

Closed

12 tasks

fstirlitz mentioned this pull request Mar 13, 2022

Assorted cleanups II #2173

Closed

coletdjnz mentioned this pull request Mar 14, 2022

[downloader/http] Make retrying more lenient #3065

Merged

9 tasks

This was referenced Apr 22, 2022

Coalesce adjacent fragments from the same file in the fragment downloader #3521

Open

Fix redirect HTTP method handling #3577

Merged

FreeTV (freetv.com) #3486

Closed

Lesmiscore mentioned this pull request May 2, 2022

[AbemaTV] post network stack overhaul #3615

Closed

9 tasks

coletdjnz mentioned this pull request May 7, 2022

[rh:requests] Add support for requests #3668

Merged

9 tasks

pukkandan mentioned this pull request May 7, 2022

[ceskatelevize.cz] Cannot download manifest - SSLV3_ALERT_HANDSHAKE_FAILURE #2043

Closed

coletdjnz mentioned this pull request May 8, 2022

[youporn] Fix extraction of metadata #2768

Merged

9 tasks

coletdjnz commented May 14, 2022

View reviewed changes

yt_dlp/networking/utils.py Outdated Show resolved Hide resolved

pukkandan force-pushed the master branch 2 times, most recently from a63ff77 to b14d523 Compare May 18, 2022 03:35

coletdjnz mentioned this pull request May 27, 2022

[twitter:shortener] extractor gives 404 error #3878

Closed

7 tasks

coletdjnz removed help-wanted Extra attention is needed needs-testing Patch needs testing labels Jun 8, 2022

coletdjnz force-pushed the network-overhaul branch from c3dd16b to f221720 Compare June 11, 2022 08:25

coletdjnz commented Jun 11, 2022

View reviewed changes

yt_dlp/utils.py Outdated Show resolved Hide resolved

This was referenced Jun 22, 2022

[Feature Request] Different proxy per website #1191

Open

[compat] Backport urllib windows registry proxy parsing #4136

Closed

pukkandan reviewed Jun 23, 2022

View reviewed changes

pukkandan reviewed Jun 24, 2022

View reviewed changes

yt_dlp/networking/_urllib.py Outdated Show resolved Hide resolved

coletdjnz mentioned this pull request Jun 24, 2022

Fix bug when handling HTTP requests #4158

Merged

9 tasks

pukkandan reviewed Jun 24, 2022

View reviewed changes

yt_dlp/YoutubeDL.py Outdated Show resolved Hide resolved

pukkandan reviewed Jun 24, 2022

View reviewed changes

test/test_InfoExtractor.py Outdated Show resolved Hide resolved

pukkandan force-pushed the network-overhaul branch 2 times, most recently from 15115a9 to c21bc7d Compare June 24, 2022 20:30

pukkandan added 4 commits July 15, 2023 10:49

Move HTTPHeaderDict

c896026

Move RequestDirector

4e8aeea

Expose objects from common

7b5d119

Cleanup

6d7d8af

pukkandan force-pushed the network-overhaul branch from 1ebb0a6 to 6d7d8af Compare July 15, 2023 06:06

pukkandan added 2 commits July 15, 2023 11:46

Change temp logger to be a subset of planned API

5028fe5

Merge remote-tracking branch 'origin/master' into pr/2861

8e3bc89

pukkandan force-pushed the network-overhaul branch from a015b66 to 8e3bc89 Compare July 15, 2023 06:16

pukkandan added 5 commits July 15, 2023 13:42

Split utils

e17fe9d

Fix deprecated typing

7ce8271

Bring back test_client_certificate

a7a4b4d

isort

2478dd0

fix

627df34

pukkandan force-pushed the network-overhaul branch from 1165025 to 393627c Compare July 15, 2023 09:57

Revert unrelated changes

591f915

pukkandan force-pushed the network-overhaul branch from 393627c to 591f915 Compare July 15, 2023 10:03

pukkandan added a commit to pukkandan/yt-dlp-dev that referenced this pull request Jul 15, 2023

[networking] Add module (yt-dlp#2861)

c365dba

No actual changes - code is only moved around

pukkandan pushed a commit to pukkandan/yt-dlp-dev that referenced this pull request Jul 15, 2023

[compat, networking] Deprecate old functions (yt-dlp#2861)

3d2623a

Authored by: coletdjnz, pukkandan

pukkandan closed this Jul 15, 2023

aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024

[networking] Add module (yt-dlp#2861)

417c4d9

No actual changes - code is only moved around

aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024

[compat, networking] Deprecate old functions (yt-dlp#2861)

02fc0dc

Authored by: coletdjnz, pukkandan

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] Rework HTTP networking #2861

[core] Rework HTTP networking #2861

coletdjnz commented Feb 23, 2022 •

edited

pukkandan left a comment

pukkandan commented Jul 15, 2023

[core] Rework HTTP networking #2861

[core] Rework HTTP networking #2861

Conversation

coletdjnz commented Feb 23, 2022 • edited

Merge before #3668

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

TODO: https://github.com/coletdjnz/yt-dlp-dev/projects/2

Changes

Known issues / regressions

Future

pukkandan left a comment

Choose a reason for hiding this comment

pukkandan commented Jul 15, 2023

coletdjnz commented Feb 23, 2022 •

edited