to export proxy does not work to download youtube videos

[ghost]

what output?
$ youtube-dl http://www.youtube.com/watch?v=lm8N0eW_tZI
[youtube] Setting language
[youtube] lm8N0eW_tZI: Downloading video webpage
[youtube] lm8N0eW_tZI: Downloading video info webpage
ERROR: unable to download video info webpage: <urlopen error [Errno 110] Connection timed out>
what problem?
I'm behind the China GFW, need to connect to youtube completely via proxy. But even I 'export http_proxy=http://my_proxy_server:port', it turns out time out problem during the downloading video info webpage stage.
I'm sure my proxy server is working fine. And when I'm using VPN, it can download. So I doubt if it uses proxy in that stage.
My OS is Archlinux. It causes this problem after the last update. Before that, it works fine.

[strex]

same problem with the current version
With an older version from 2012.10.09 it works.

http_proxy=xx.xx.xx.xx:xxx youtube-dl http://www.youtube.com/watch?v=6UztEfwHt14
also does not work anymore.

I´m using Debian 6.0

[ghost]

To strex...
fallback to 2013.02.25. everything is ok.

[jaimeMF]

Is 2013.02.25 the last version that worked to you?
If not, could you find the last working version?

[ghost]

[2013-04-28 08:59] [PACMAN] downgraded youtube-dl (2013.02.25-1 -> 2012.12.31-1)
can download youtube videos via exported http_proxy, but can't playing when downloading

$ youtube-dl http://www.youtube.com/watch?v=DicLEWiFjtk -o - | mplayer -
MPlayer2 UNKNOWN (C) 2000-2012 MPlayer Team
[ass] FcFreeTypeQueryFace failed
Can't open joystick device /dev/input/js0: No such file or directory
Can't init input joystick
Playing -.
Reading from stdin...
[youtube] Setting language
[youtube] DicLEWiFjtk: Downloading video webpage
[youtube] DicLEWiFjtk: Downloading video info webpage
[youtube] DicLEWiFjtk: Extracting video information
[download] Destination: -
Traceback (most recent call last):
File "/usr/bin/youtube-dl", line 6, in
youtube_dl.main()
File "/usr/lib/python3.3/site-packages/youtube_dl/init.py", line 516, in main
_real_main()
File "/usr/lib/python3.3/site-packages/youtube_dl/init.py", line 500, in _real_main
retcode = fd.download(all_urls)
File "/usr/lib/python3.3/site-packages/youtube_dl/FileDownloader.py", line 525, in download
self.process_info(video)
File "/usr/lib/python3.3/site-packages/youtube_dl/FileDownloader.py", line 468, in process_info
success = self._do_download(filename, info_dict)
File "/usr/lib/python3.3/site-packages/youtube_dl/FileDownloader.py", line 752, in _do_download
stream.write(data_block)
TypeError: must be str, not bytes
Cannot seek backward in linear streams!
Seek failed
Failed to recognize file format.
Exiting... (End of file)

maybe due to my Archlinux + mplayer2.
at least last month, it worked everything fine.

[jaimeMF]

Could you try to run 2013.03.29 or the first releases of April (http://youtube-dl.org/downloads/) to discover the version that breaks proxy support.

[ghost]

Fallback one by one, finally localed it. since http://youtube-dl.org/downloads/2013.04.03, it had this problem.

[jaimeMF]

Thanks, that will make a bit easier to catch the error.

[ghost]

No No No sorry
http://youtube-dl.org/downloads/2013.04.03 is OK
since http://youtube-dl.org/downloads/2013.04.11

[jaimeMF]

Ok, could you post the output when running youtube-dl -v http://www.youtube.com/watch?v=lm8N0eW_tZI.

[ghost]

$ youtube-dl -v http://www.youtube.com/watch?v=lm8N0eW_tZI
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://www.youtube.com/watch?v=lm8N0eW_tZI']
[debug] youtube-dl version 2013.04.03
[debug] Python version 3.3.1 - Linux-3.8.8-2-ARCH-i686-with-arch
[debug] Proxy map: {'http': 'http://127.0.0.1:8118'}
[youtube] Setting language
[youtube] lm8N0eW_tZI: Downloading video webpage
[youtube] lm8N0eW_tZI: Downloading video info webpage
[youtube] lm8N0eW_tZI: Extracting video information
[download] Destination: lm8N0eW_tZI.mp4
[download] 0.0% of 493.06M at 51.22k/s ETA --:--^C
ERROR: Interrupted by user

$ youtube-dl -v http://www.youtube.com/watch?v=lm8N0eW_tZI
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://www.youtube.com/watch?v=lm8N0eW_tZI']
[debug] youtube-dl version 2013.04.11
[debug] Python version 3.3.1 - Linux-3.8.8-2-ARCH-i686-with-arch
[debug] Proxy map: {'http': 'http://127.0.0.1:8118'}
[youtube] Setting language
[youtube] lm8N0eW_tZI: Downloading video webpage
[youtube] lm8N0eW_tZI: Downloading video info webpage
***** stop here for a long time and never go on *****

[jaimeMF]

Probably the issue is in SHA: 927c8c4, it's the only commit that changes the download process. Could you try that commit and the previous?
@phihag in that commit an url is used instead of a compat_urllib_request.Request object, could that be the problem.

[ghost]

[youtube] lm8N0eW_tZI: Downloading video info webpage
ERROR: unable to download video info webpage: <urlopen error [Errno 110] Connection timed out>
File "/usr/lib/python3.3/site-packages/youtube_dl/InfoExtractors.py", line 121, in _request_webpage
return compat_urllib_request.urlopen(url_or_request)
File "/usr/lib/python3.3/urllib/request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.3/urllib/request.py", line 473, in open
response = self._open(req, data)
File "/usr/lib/python3.3/urllib/request.py", line 491, in _open
'_open', req)
File "/usr/lib/python3.3/urllib/request.py", line 451, in _call_chain
result = func(*args)
File "/usr/lib/python3.3/urllib/request.py", line 1287, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib/python3.3/urllib/request.py", line 1255, in do_open
raise URLError(err)

[jaimeMF]

Is that the continuation of the output?

[ghost]

Yes
and I install it by editting Archlinux PKGBUILD from
https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/youtube-dl
but I don't know how to compile it from github source in a specific commit.

[phihag]

@jaimeMF No, specifying an URL is perfectly fine. The problem is that the commit switches to HTTPS by default. I'm not sure whether that's the result of a broken proxy, but there are good reasons not to download via HTTP by default (namely, that HTTP is filtered by the Chinese Firewall).

[phihag]

@westmin You can simply checkout the hash you want and then run youtube-dl with

python -m youtube_dl -v ....

Do you know if your proxy supports HTTPS? I.e. does

export https_proxy=http://127.0.0.1:8118

fix the problem for you?

[jaimeMF]

@westmin Download the zipfile from :https://github.com/rg3/youtube-dl/archive/927c8c49246bb7ae98596b3cf518425d46f39100.zip, extract it and run pythyon3 -m youtube_dl -v the_url. Then try with the previous commit: https://github.com/rg3/youtube-dl/archive/0ba994e9e323f4162ed12542beb079df1f4b2825.zip

Use this if you don't have git, otherwise follow phihag method.

[ghost]

ok. let me try both of your way.........

[ghost]

@phihag
$ youtube-dl -v http://www.youtube.com/watch?v=lm8N0eW_tZI
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://www.youtube.com/watch?v=lm8N0eW_tZI']
[debug] youtube-dl version 2013.04.11
[debug] Python version 3.3.1 - Linux-3.8.8-2-ARCH-i686-with-arch
[debug] Proxy map: {'https': 'http://127.0.0.1:8118', 'http': 'http://127.0.0.1:8118'}
[youtube] Setting language
[youtube] lm8N0eW_tZI: Downloading video webpage
[youtube] lm8N0eW_tZI: Downloading video info webpage
[youtube] lm8N0eW_tZI: Extracting video information
[download] Destination: lm8N0eW_tZI.mp4
[download] 0.1% of 493.06M at 61.48k/s ETA --:--^C
ERROR: Interrupted by user

It works!

[ghost]

I check that commit. found:

• video_info_url = ('http://www.youtube.com/get_video_info?&video_id=%s%s&ps=default&eurl=&gl=US&hl=en'
• video_info_url = ('https://www.youtube.com/get_video_info?&video_id=%s%s&ps=default&eurl=&gl=US&hl=en'
  I think the problem is because I only export http_proxy, but the stage of 'Downloading video info webpage' is to use https connection. so it will connect to that https address directly. you know no matter http or https connection is block by China GFW.
  After export https_proxy, it will connect to that https address via proxy, so it works ok.
  So I don't need to check out @jaimeMF 's way.
  Thanks!

[phihag]

@westmin Thanks, we'll use the HTTPS proxy even if only a HTTP one is specified then.

[phihag]

@westmin If you're using your own proxy, the GFW can't decode HTTPS requests, but can decode HTTP ones. So unless your proxy is blocked itself, HTTPS should be sufficient to evade the GFW. For reference, see #739 and #691.