Sometimes I get 403 errors

[dev0k]

Hello guys,

i use youtube-dl for download my music. But sometimes I get a 403 error.

url=youtube-dl --user-agent 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' -g --skip-download --write-info-json --cookies cookie.y2m -o source $1 | tail -n 1 -

cookie=$( cat cookie.y2m | awk '{printf("%s=%s;",$6,$7)}' )

axel -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 ' -H "Cookie: $cookie" $url

(axel is a downloader (like wget))

Very funny! Sometimes it works, sometimes not! But why..?

Thanks :)

[dev0k]

What the fuck ...

root@mybase: curl 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' -L > video2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
19 12.1M 19 2429k 0 0 507k 0 0:00:24 0:00:04 0:00:20 581k^C

curl works, without cookies! (only with the parameter ('-L'))

and now axel... super fast but..

root@mybase:# axel 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' > video3
Start Request
HTTP/1.1 403 Forbidden

but why?!

[Tailszefox]

Hm, I'm not getting any issue, neither with curl nor axel, the video just gets downloaded correctly. Do you have any particular URLs for which it doesn't always work, or does it seem random?

As a side note, if you used the same URLs twice, that may be the reason your second download got refused. If you try to download a video and then try it again a bit later, it may not work since the link may have expired, or be deleted since it's already been used. Do you get a 403 error if you try with axel first?

[dev0k]

Thanks. I've found the problem. In my script the url has not escaped.

axel $url

Sometimes error. why?!

axel "$url"

No Exception.

But why has he given only SOMETIMES an error message? Thats funny..

[Tailszefox]

Ah, right, I thought you already did that. Not sure, might be because of the way YouTube reacts to malformed URLs. Regardless, as long as it works now, it's all good!

[FiloSottile]

Probably breaks only if shell meaningful chars get into the url, and this might be random. Nice to hear the issue is solved, closing.

[dev0k]

root@y:/home/product# axel 'YOUTUBE-URL' -o test
Start Request: YOUTUBE-URL
HTTP/1.1 403 Forbidden

root@y:/home/product# curl -L 'YOUTUBE-URL' > test
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 12.1M 4 579k 0 0 109k 0 0:01:54 0:00:05 0:01:49 114k^C

curl works.. axel dont work :-/ yesterday it still worked. Now comes the 403 error. What now? :-/

[FiloSottile]

Umh... the User-Agent? You should supply the youtube-dl one to axel.

[dev0k]

But why it works that way with wget, curl, etc? Sometimes it is also with axel, but only sometimes.

[FiloSottile]

Well, it might be the different User-Agent, but my best bet would be that the axel download gets killed as it opens too much concurrent connections.

[dev0k]

root@y:~# axel -n 1 -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' '$url'

Only 1 Connection and same useragent.

and.. 403 error :-/

Can you maybe try with Axel? The YT Url is: http://www.youtube.com/watch?v=4wOoLLDXbDY.

[Tailszefox]

% axel `youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY"`
Initializing download: http://o-o---preferred---sn-5hn7sned---v14---lscache6.c.youtube.com/videoplayback?upn=2w9wpU4czQ8&sparams=cp%2Cgcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=919318%2C904461%2C916617%2C922401%2C920704%2C912806%2C927201%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&ms=au&expire=1352863289&itag=22&ipbits=48&gcr=fr&sver=3&ratebypass=yes&mt=1352839511&ip=2a02%3A8420%3A5722%3A6100%3Ad0b9%3Afc58%3A6278%3A66f2&mv=m&source=youtube&key=yt1&cp=U0hUR1hRUV9NU0NONF9QS1VDOnN0eTRXZ1NpSU0x&id=e303a82cb0d76c36&signature=9A3B839DFC5C00F81F65A77D2800CD002E111AE4.C2957062DF9B83E4D70A13629A260E9576A6F9C7
File size: 48520653 bytes
Opening output file videoplayback
Starting download

[  0%]  .......... .......... .......... .......... ..........  [  14,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  21,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  21,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  25,7KB/s]
[...]

Download then continues and finishes normally. I tried it at least five times, each time getting a different URL from YouTube since I'm making youtube-dl make a new request each time, and it seems to work with or without a "proper" user agent, at least for me.

[dev0k]

root@y:# axel ./youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY"
(469, 591)
(498, 591)
(498, 1182)
(498, 1773)
(498, 2364)
(498, 2955)
(498, 3546)
(498, 4137)
(498, 4728)
(498, 5319)
(498, 5910)
(498, 6501)
Starte Abruf: http://o-o---preferred---sn-25g7rn7k---v14---lscache6.c.youtube.com/videoplayback?upn=ioWaURvK7Mc&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=900080%2C922401%2C920704%2C912806%2C927201%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&key=yt1&expire=1352863289&itag=22&ipbits=8&sver=3&ratebypass=yes&mt=1352840172&ip=94.23.233.25&mv=m&source=youtube&ms=au&cp=U0hUR1hRUV9NU0NONF9QS1VDOi1PaVMzeXVSYjdG&id=e303a82cb0d76c36&signature=2134E4C08FE1EF2EE94E0619DEC5FBED3CADC53D.1D8AEFC904E0F87358D337CDC6CD167B82D4957B
HTTP/1.1 403 Forbidden

Can you please paste this:
axel --version
youtube-dl --version

[Tailszefox]

Seems you're using an old version of youtube-dl, given that debug output you get. You should try compiling the latest version, I don't know if it's related to your issue, I doubt it, but it's worth trying. Anyway:

% axel --version
Axel version 2.4 (Linux)

Copyright 2001-2002 Wilmer van der Gaast.

% youtube-dl --version
2012.10.09

The youtube-dl version I use is the latest as I regularly update my local repository to match the current version.

[dev0k]

root@y:# axel youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY "
(469, 591)
(498, 591)
(498, 1182)
(498, 1773)
(498, 2364)
(498, 2955)
(498, 3546)
(498, 4137)
(498, 4728)
(498, 5319)
(498, 5910)
(498, 6501)
Starte Abruf: http://o-o---preferred---sn-25g7rn7k---v14---lscache6.c.youtube.co m/videoplayback?upn=Frb8DUNFPYg&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypas s%2Csource%2Cupn%2Cexpire&fexp=906437%2C910013%2C922401%2C920704%2C912806%2C9272 01%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&key=yt1&expire =1352866889&itag=22&ipbits=8&sver=3&ratebypass=yes&mt=1352841791&ip=94.23.233.25 &mv=m&source=youtube&ms=au&cp=U0hUR1hRVF9NU0NONF9QS1hJOlRSSjVRdTAyRERS&id=e303a8 2cb0d76c36&signature=C6D0D10A4FD37715B0801F7858CEC20C3D85F8E1.CCEC1976655171BDB1 F6324EB98C76BE0B8C0E90
HTTP/1.1 403 Forbidden
root@y:# youtube-dl --version
2012.10.09

What do the numbers at the beginning?

I've even tested on a different server, there it worked.

[Tailszefox]

This is some debug output left there by mistake. It's been fixed in commit 5d40a47, which means you're using an outdated version. As I said, I don't think the issues are related, but it's always good to have the latest version.

The only thing I can think of that would cause this problem is that axel downloads the video with a different IP that youtube-dl uses when requesting the URL of the video. Maybe one is using a proxy while the other isn't. If it works with curl or wget, maybe you could try something like this:

% axel -o ip.txt "http://ifconfig.me/ip"
% wget -O ip2.txt "http://ifconfig.me/ip"

Then, check the files ip.txt and ip2.txt. If they don't have the same IP, that's the issue. If they do, then...I'm afraid I'm out of ideas, sorry.

[dev0k]

Thanks for your help. The IP addresses are the same.
One thing I ask myself, I have youtube-dl updated to the latest version, but I still get the numbers at the beginning. (10/09/2012).
Then I have the same installed on another server. Then I tried it, and there it works. What could be wrong?

[Tailszefox]

Do you mean the debug output, or the version number from youtube-dl? The version number isn't updated automatically, and the youtube-dl binary provided here isn't always up to date either. The only way to ensure you actually do have the latest version is to clone the repository and use make youtube-dl to compile it.

As for why it works on a server and not on another, I can only guess it's a connectivity or proxy issue that only happens on one of the server, and not the other. That sounds inconvenient but I'm afraid there's nothing youtube-dl can do about that, as it seems to be more related to axel than anything.

[FiloSottile]

As a last chance, I would try (updating and) passing also the cookies to axel. We have an option for that.

[dev0k]

Okay.

root@y:/home# youtube-dl "http://www.youtube.com/watch?v=RQaz8VFTzlQ&feature=related"
[youtube] Setting language
[youtube] RQaz8VFTzlQ: Downloading video webpage
[youtube] RQaz8VFTzlQ: Downloading video info webpage
[youtube] RQaz8VFTzlQ: Extracting video information
(456, 591)
(485, 591)
(485, 1182)
(485, 1773)
(485, 2364)
(485, 2955)
(485, 3546)
(485, 4137)
(485, 4728)
(485, 5319)
(485, 5910)
(485, 6501)
[download] Destination: RQaz8VFTzlQ.flv
root@y:/home# youtube-dl --version
2012.10.09

These numbers are still there, although "# print >> sys.stderr, self.getpos ()" is commented out. Why?

[dev0k]

So, I have the solution!

root @ y :/ home # youtube-dl-g 'http://www.youtube.com/watch?v=4wOoLLDXbDY' | xargs curl-I
HTTP/1.1 302 Moved Temporarily
Last-Modified: Wed, 02 May 2007 10:26:10 GMT
Date: Wed, 14 November 2012 09:23:52 GMT
Expires: Wed, 14 November 2012 09:23:52 GMT
Cache-Control: private, max-age = 900
Location: NEW YOUTUBE URL
Connection: close
X-Content-Type-Options: nosniff
Content-Type: text / html
Server: gvs 1.0

And then: axel "$ new url" and WORKS!

Thanks to you :)