Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate extractor proxy and download proxy #3391

Closed
yan12125 opened this issue Jul 28, 2014 · 3 comments
Closed

Separate extractor proxy and download proxy #3391

yan12125 opened this issue Jul 28, 2014 · 3 comments

Comments

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Jul 28, 2014

Some china video sharing sites such as Youku and Sohu limit some of their vidoes available only in Mainland China. In such cases, I need a proxy in China to get the video. For instance, proxy.uku.im:8888, the proxy used by a popular browser plugin Unblock Youku. However, this proxy allow only addresses for checking China IP. Take Sohu as example, hot.vrs.letv.com is allowed while tv.sohu.com not. Is it possible to use two different proxies, one for extracting video info and the other for real downloading? A similar project named you-get [1] has this ability.
[1] https://github.com/soimort/you-get
@yan12125 yan12125 mentioned this issue Feb 27, 2015
Closed
@jaimeMF
Copy link
Collaborator

@jaimeMF jaimeMF commented Feb 27, 2015

Why can't you use the same proxy for extraction and download?
@yan12125
Copy link
Collaborator Author

@yan12125 yan12125 commented Feb 27, 2015

I have some extra modifications based on #5077 at yan12125/youtube-dl@a6d3627. Below are my test results.
(202.108.50.75:80 is the fastest proxy listed in http://cn-proxy.com/)

$ youtube-dl --verbose --proxy "202.108.50.75:80" "http://tv.sohu.com/20150226/n409160515.shtml"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--proxy', '202.108.50.75:80', 'http://tv.sohu.com/20150226/n409160515.shtml']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.26.1
[debug] Git HEAD: f7b8b0c
[debug] Python version 3.4.3 - Linux-3.19.0-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 2.5.4, ffprobe 2.5.4, rtmpdump 2.4
[debug] Proxy map: {'http': '202.108.50.75:80', 'https': '202.108.50.75:80'}
[debug] Alternative proxy map: {'http': '202.108.50.75:80', 'https': '202.108.50.75:80'}
[Sohu] 409160515: Downloading webpage
[Sohu] 409160515: Downloading JSON data for 2258522
[Sohu] 409160515: Downloading JSON data for 2258523
[Sohu] 409160515: Downloading JSON data for 2258524
[Sohu] 409160515: Downloading nor video URL part 1 of 8
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/extractor/common.py", line 314, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1666, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

$ youtube-dl --verbose --alternative-proxy "202.108.50.75:80" "http://tv.sohu.com/20150226/n409160515.shtml"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--alternative-proxy', '202.108.50.75:80', 'http://tv.sohu.com/20150226/n409160515.shtml']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.26.1
[debug] Git HEAD: f7b8b0c
[debug] Python version 3.4.3 - Linux-3.19.0-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 2.5.4, ffprobe 2.5.4, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Alternative proxy map: {'https': '202.108.50.75:80', 'http': '202.108.50.75:80'}
[Sohu] 409160515: Downloading webpage
[Sohu] 409160515: Downloading JSON data for 2258522
[Sohu] 409160515: Downloading JSON data for 2258523
[Sohu] 409160515: Downloading JSON data for 2258524
[Sohu] 409160515: Downloading nor video URL part 1 of 8
[Sohu] 409160515: Downloading super video URL part 1 of 8
[Sohu] 409160515: Downloading high video URL part 1 of 8
[Sohu] 409160515: Downloading nor video URL part 2 of 8
[Sohu] 409160515: Downloading super video URL part 2 of 8
[Sohu] 409160515: Downloading high video URL part 2 of 8
[Sohu] 409160515: Downloading nor video URL part 3 of 8
[Sohu] 409160515: Downloading super video URL part 3 of 8
[Sohu] 409160515: Downloading high video URL part 3 of 8
[Sohu] 409160515: Downloading nor video URL part 4 of 8
[Sohu] 409160515: Downloading super video URL part 4 of 8
[Sohu] 409160515: Downloading high video URL part 4 of 8
[Sohu] 409160515: Downloading nor video URL part 5 of 8
[Sohu] 409160515: Downloading super video URL part 5 of 8
[Sohu] 409160515: Downloading high video URL part 5 of 8
[Sohu] 409160515: Downloading nor video URL part 6 of 8
[Sohu] 409160515: Downloading super video URL part 6 of 8
[Sohu] 409160515: Downloading high video URL part 6 of 8
[Sohu] 409160515: Downloading nor video URL part 7 of 8
[Sohu] 409160515: Downloading super video URL part 7 of 8
[Sohu] 409160515: Downloading high video URL part 7 of 8
[Sohu] 409160515: Downloading nor video URL part 8 of 8
[Sohu] 409160515: Downloading super video URL part 8 of 8
[Sohu] 409160515: Downloading high video URL part 8 of 8
[download] Downloading playlist: 409160515
[Sohu] playlist 409160515: Collected 8 video ids (downloading 8 of them)
[download] Downloading video 1 of 8
[debug] Invoking downloader on 'http://36.48.159.234/sohu/s26h23eab6/4/124/118/WZo5FKPBTYCBfKJX6Spk2B.mp4?key=wlDPbg1G7VGIIXhIOhKk4sPMqDhgEGPoMsXGHLUsMUU.'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part1.mp4
[download] 100% of 35.30MiB in 03:27
[download] Downloading video 2 of 8
[debug] Invoking downloader on 'http://70.39.188.2/sohu/s26h23eab6/3/193/169/Bb0tNVgsT8xCpit3lOkh5E.mp4?key=iCBPmdHMncGW6PIIH7FtrOFuPtLv4NJqxL15EA..'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part2.mp4
[download] 100% of 35.23MiB in 00:13
[download] Downloading video 3 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/127/133/lL2w8XSrpH1PTuV5f9rhMB.mp4?key=pxzLwjJv_wnTQ5V0ThCca99j0RkxJqRy'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part3.mp4
[download] 100% of 35.24MiB in 00:03
[download] Downloading video 4 of 8
[debug] Invoking downloader on 'http://114.80.179.212/sohu/s26h23eab6/9/27/141/xKw6QZotSka9xTRCgSaPxG.mp4?key=wWLhlvGW3u2Nktt3lpaappCH3TL7L8hVyiUXTcNnpxM.'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part4.mp4
[download]   0.8% of 35.22MiB at 17.14KiB/s ETA 34:47^C
ERROR: Interrupted by user

$ youtube-dl --verbose --alternative-proxy "proxy.uku.im:8888" "http://tv.sohu.com/20150226/n409160515.shtml"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--alternative-proxy', 'proxy.uku.im:8888', 'http://tv.sohu.com/20150226/n409160515.shtml']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.26.1
[debug] Git HEAD: f7b8b0c
[debug] Python version 3.4.3 - Linux-3.19.0-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 2.5.4, ffprobe 2.5.4, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Alternative proxy map: {'http': 'proxy.uku.im:8888', 'https': 'proxy.uku.im:8888'}
[Sohu] 409160515: Downloading webpage
[Sohu] 409160515: Downloading JSON data for 2258522
[Sohu] 409160515: Downloading JSON data for 2258523
[Sohu] 409160515: Downloading JSON data for 2258524
[Sohu] 409160515: Downloading super video URL part 1 of 8
[Sohu] 409160515: Downloading nor video URL part 1 of 8
[Sohu] 409160515: Downloading high video URL part 1 of 8
[Sohu] 409160515: Downloading super video URL part 2 of 8
[Sohu] 409160515: Downloading nor video URL part 2 of 8
[Sohu] 409160515: Downloading high video URL part 2 of 8
[Sohu] 409160515: Downloading super video URL part 3 of 8
[Sohu] 409160515: Downloading nor video URL part 3 of 8
[Sohu] 409160515: Downloading high video URL part 3 of 8
[Sohu] 409160515: Downloading super video URL part 4 of 8
[Sohu] 409160515: Downloading nor video URL part 4 of 8
[Sohu] 409160515: Downloading high video URL part 4 of 8
[Sohu] 409160515: Downloading super video URL part 5 of 8
[Sohu] 409160515: Downloading nor video URL part 5 of 8
[Sohu] 409160515: Downloading high video URL part 5 of 8
[Sohu] 409160515: Downloading super video URL part 6 of 8
[Sohu] 409160515: Downloading nor video URL part 6 of 8
[Sohu] 409160515: Downloading high video URL part 6 of 8
[Sohu] 409160515: Downloading super video URL part 7 of 8
[Sohu] 409160515: Downloading nor video URL part 7 of 8
[Sohu] 409160515: Downloading high video URL part 7 of 8
[Sohu] 409160515: Downloading super video URL part 8 of 8
[Sohu] 409160515: Downloading nor video URL part 8 of 8
[Sohu] 409160515: Downloading high video URL part 8 of 8
[download] Downloading playlist: 409160515
[Sohu] playlist 409160515: Collected 8 video ids (downloading 8 of them)
[download] Downloading video 1 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/124/118/WZo5FKPBTYCBfKJX6Spk2B.mp4?key=BWZ-NKS2U7IB6IbTT-ql4zDsciO-0LlU'
[download] 《神雕侠侣（2014）》第46集-409160515_part1.mp4 has already been downloaded
[download] 100% of 35.30MiB
[download] Downloading video 2 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/193/169/Bb0tNVgsT8xCpit3lOkh5E.mp4?key=5SnjvVW9Gfm4WAV4fZJswMK0CvamUgYU'
[download] 《神雕侠侣（2014）》第46集-409160515_part2.mp4 has already been downloaded
[download] 100% of 35.23MiB
[download] Downloading video 3 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/127/133/lL2w8XSrpH1PTuV5f9rhMB.mp4?key=xUPccECv4bZEE8h2PQEzW4UbBrfdtCAf'
[download] 《神雕侠侣（2014）》第46集-409160515_part3.mp4 has already been downloaded
[download] 100% of 35.24MiB
[download] Downloading video 4 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/27/141/xKw6QZotSka9xTRCgSaPxG.mp4?key=TsnjLLtGpsJWdiHZsXQMPmyMFYt1tl06'
[download] Resuming download at byte 294046
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part4.mp4
[download] 100% of 35.22MiB in 03:23
[download] Downloading video 5 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/54/117/NoOhEDDCTdmME2N7Jt8m3H.mp4?key=K9uxlYhABfSvgQWZM35AO_Uk_VxSaVdF'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part5.mp4
[download] 100% of 35.23MiB in 00:09
[download] Downloading video 6 of 8
[debug] Invoking downloader on 'http://122.226.184.135/sohu/s26h23eab6/1/1/178/UZeixXkdSmaibDl7mlY5FB.mp4?key=m_0vUW4N4s4Ton_vjCMhshpDqlz3rhgkZTyJx4Xxr9c.'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part6.mp4
[download] 100% of 35.31MiB in 00:11
[download] Downloading video 7 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/240/191/xqQ5J93UIUrVLuOTXuYYaD.mp4?key=TGcfh4FiwgQajaCSAPS9BEb5mxtO0y7Y'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part7.mp4
[download] 100% of 35.26MiB in 00:13
[download] Downloading video 8 of 8
[debug] Invoking downloader on 'http://newflv.sohu.ccgslb.net/0/202/u0WzN9PWI1FM9auN93elOF.mp4?key=cg6WL0UsGO-RcI7FJS5eKjEUWF5peIDD'
[download] Destination: 《神雕侠侣（2014）》第46集-409160515_part8.mp4
[download] 100% of 18.48MiB in 00:35

$ youtube-dl --verbose --proxy "proxy.uku.im:8888" "http://tv.sohu.com/20150226/n409160515.shtml"            
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--proxy', 'proxy.uku.im:8888', 'http://tv.sohu.com/20150226/n409160515.shtml']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.26.1
[debug] Git HEAD: f7b8b0c
[debug] Python version 3.4.3 - Linux-3.19.0-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 2.5.4, ffprobe 2.5.4, rtmpdump 2.4
[debug] Proxy map: {'https': 'proxy.uku.im:8888', 'http': 'proxy.uku.im:8888'}
[debug] Alternative proxy map: {'https': 'proxy.uku.im:8888', 'http': 'proxy.uku.im:8888'}
[Sohu] 409160515: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/extractor/common.py", line 314, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1666, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

As shown above, some China proxies are somewhat unstable and/or slow. This approach can minimize the proxy usage. Proxies are used only when needed. Otherwise, use direct connection for faster and more stable downloading.
@yan12125
Copy link
Collaborator Author

@yan12125 yan12125 commented Mar 7, 2015

It's implemented in 91410c9. Closing.
@yan12125 yan12125 closed this Mar 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Multiple proxies
2 participants
@jaimeMF @yan12125
You can’t perform that action at this time.