Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to exclude specific host(s) from --proxy switch #16809

Open
Vangelis66 opened this issue Jun 25, 2018 · 0 comments
Open

Option to exclude specific host(s) from --proxy switch #16809

Vangelis66 opened this issue Jun 25, 2018 · 0 comments

Comments

@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Jun 25, 2018

  • I've verified and I assure that I'm running youtube-dl 2018.06.25

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

Description of your issue, suggested solution and other information

Hello and thanks to the dev team 😄
My OS is Windows Vista SP2 32-bit, latest yt-dl windows executable (as stated).

I want to use yt-dl to dump a certain private HLS master playlist, let's call it
https://host1/blahblah/master.m3u8?token.
The master playlist URI is geo-fenced from my physical location,
so are its individual variants and segment URIs; to connect to host1 I have thus
to use a special (commercial) HTTPS proxy, so with yt-dl I would type:

youtube-dl --proxy="https://proxyhost:proxyport" --hls-prefer-native "https://host1/blahblah/master.m3u8?token"

As is not uncommon nowadays, the HLS segments are AES-128 encrypted;
the key delivery URI is of the format:
https://host2/HLS/GetKey.aspx?parameters

My specific issue:

host2 is NOT GEO-FENCED from my physical location, however proxyhost is configured by its maintainers in a special manner so as to refuse connections to host2; IOW, the proxy is only being used
for segment downloading, while segment decryption key acquisition is done DIRECT...

In Firefox, I can write a local PAC file (to proxy only host1 | exclude from proxying host2)
or use the browser's GUI settings to implement the proxy and exclude from it host2; that way,
the HLS stream can be viewed in the browser... But in yt-dl, with only using the --proxy option,
I end up in failure; the download is aborted after the 1st segment has been downloaded but not decrypted:

[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 470
(removed)
[download]   0.2% of ~559.70MiB at Unknown speed ETA Unknown ETAERROR: unable to
 download video data: <urlopen error Tunnel connection failed: 403 Forbidden>
Traceback (most recent call last):
  File "C:\Python\Python34\lib\urllib\request.py", line 1183, in do_open
  File "C:\Python\Python34\lib\http\client.py", line 1137, in request
  File "C:\Python\Python34\lib\http\client.py", line 1182, in _send_request
  File "C:\Python\Python34\lib\http\client.py", line 1133, in endheaders
  File "C:\Python\Python34\lib\http\client.py", line 963, in _send_output
  File "C:\Python\Python34\lib\http\client.py", line 898, in send
  File "C:\Python\Python34\lib\http\client.py", line 1279, in connect
  File "C:\Python\Python34\lib\http\client.py", line 874, in connect
  File "C:\Python\Python34\lib\http\client.py", line 857, in _tunnel
OSError: Tunnel connection failed: 403 Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\YoutubeDL.py", line 1908, in process_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\YoutubeDL.py", line 1847, in dl
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\downloader\common.py", line 364, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\downloader\hls.py", line 168, in real_download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\YoutubeDL.py", line 2211, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 464, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 482, in _open
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\bu
ild\youtube_dl\utils.py", line 1101, in https_open
  File "C:\Python\Python34\lib\urllib\request.py", line 1185, in do_open
urllib.error.URLError: <urlopen error Tunnel connection failed: 403 Forbidden>

Solution:

Sadly, I am not a coder; it took me quite a while to identify the root cause of my issue,
then spent another 2 hours of searching on-line + trial & error for a solution; it turns
out I can set in the Windows command line the following environment variable:
SET NO_PROXY=host2 (followed by ENTER) and then, in the same CLI window, issue my yt-dl cmd:
youtube-dl --proxy="https://proxyhost:proxyport" --hls-prefer-native "https://host1/blahblah/master.m3u8?token"

To hopefully save the hassle from other yt-dl users, could we please have a new --no-proxy-for-host
(or similar) documented option in yt-dl? Value could be a comma separated list of hosts not to be proxied via the --proxy supplied proxy...

I understand this should be a low-priority feature request, however thanks in advance for possibly considering it... 😄

Best regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.