-
Notifications
You must be signed in to change notification settings - Fork 375
403s #19
Comments
Works fine here Can you please post the stack trace and the version you are using? |
I just got the latest you put up (it was 15 hrs old when I updated). It returned a 403 with an older version I was running. Now it throws an exception. I am going to check to make sure I properly integrated your latest but it still downloads other videos. I’m now calling DownloadUrlResolver with http://www.youtube.com/watch?v=k6EQAOmJrbw The exception thrown is ‘the given key was not present in the dictionary’ and it is throwing on the first iteration of foreach (Uri url in downloadUrls) I think because downloadUrls is null I’m attaching the pagesource in pagesource.html Here is the stack trace:
|
Are you sure you are running version 0.6.2? Because 0.6.1 and 0.6.0 had exactly the issue you encountered. |
Hello, I've been experiencing 403 issues as well for the last couple of days, first on 0.5.0, and later after integrating the changes I could from 0.6.2 (still running VS2010). Examples of videos that trigger 403s: Also, I believe that I've seen the exception that joedean3 is talking about. It is thrown by DownloadUrlResolver's ExtractDownloadUrls() method. Its caused by YouTube backend changes: In the videos that trigger 403s, the "sig" parameter has changed its name to "s", causing the following line to fail:
I've replaced it with the following:
However, while that gets rid of the exceptions, it doesn't solve the 403 issue. |
Hm, I still can't reproduce the 403 exception. Can you please post the whole exception including the exception name, message and stacktrace? |
I just ran one test and it’s not happening anymore… At least not on the video I was testing before. I will surely follow up with you should I get it or any other issues again. Thanks JOE From: Dennis Daume [mailto:notifications@github.com] Hm, I still can't reproduce the 403 exception. Can you please post the whole exception including the exception name, message and stacktrace? — |
On my machine, the 403s are continuing as usual. To make sure they weren't coming from my code, I downloaded the last version (0.6.2), and used the included ExampleApplication. Initially, I got parsing exceptions, but after I replaced the "sig" parameter with "s" (as specified in my previous message), the 403s came back. URL of the video I used: http://www.youtube.com/watch?v=rgqHT-iy8kA Exception details:
ResponseURI:
|
OK, here it is… I was getting 403s but with your latest, it throws an exception: Name:System.Collections.Generic.KeyNotFoundException Message: "The given key was not present in the dictionary." StackTrace: at System.Collections.Generic.Dictionary`2.get_Item(TKey key) at YoutubeExtractor.DownloadUrlResolver.d__0.MoveNext() in DownloadUrlResolver.cs:line 81 at YoutubeExtractor.DownloadUrlResolver.GetVideoInfos(IEnumerable`1 downloadUrls, String videoTitle) in DownloadUrlResolver.cs:line 94 at YoutubeExtractor.DownloadUrlResolver.GetDownloadUrls(String videoUrl) in DownloadUrlResolver.cs:line 44 From: Dennis Daume [mailto:notifications@github.com] Hm, I still can't reproduce the 403 exception. Can you please post the whole exception including the exception name, message and stacktrace? — |
What is the format code of the video you are downloading? |
18 |
Ha! Now I can reproduce the KeyNotFoundException, thanks! |
Ok, as it turns out this is because the uploader has not made the video available in my country...lets see if I can throw a more descriptive exception. Could you please give me another video URL with the format code that throws a 403 exception? |
And the format code? This is important as I believe that it throws only on certain video types |
The same one, 18. Actually, I tried a bunch of other codes from GetDownloadUrls(), and they all throw 403s: 44, 35, 43, 34, 5. |
I hate that I can't reproduce this. In which country do you live? Maybe I can pipe my connection through a proxy in your country and see if this "solves" it. |
I'm from Israel. |
Thank you for your efforts, I must go for now, but I'll be back in the evening. |
Well it seems I got it working most of the time. This problem mostly occurs with the VEVO videos. Take a look at this project Here is some code of mine, that seems to solve the problem with the most of those vids. Love your project. I hope it helps .....
|
smurz, where does the first parameter ( videoInfo ) in ExtractDownloadUrls() come from? |
michael79, it's a source of the get_video_info page string requestUrl = String.Format("http://www.youtube.com/get_video_info?&video_id={0}&el=detailpage&ps=default&eurl=&gl=US&hl=en", id); string videoInfo = HttpHelper.DownloadString(requestUrl); thou this method seems to not work if the signature is other than "43 chars point 43 chars" |
[Part I/II] With some changes, I managed to get smurz's solution to work: First of all, your ExtractDownloadUrls() has a bug, in about 50% of the cases playerStreamMap will have a leading space, which then causes the following line to fail:
as the key will then be " url" instead of "url" (note the space). I've fixed this by trimming the leading & trailing spaces: playerStreamMap = playerStreamMap.Trim(); The complete code for ExtractDownloadUrls() looks like this: private static IEnumerable<Uri> ExtractDownloadUrls(string videoInfo, string pageSource)
{
var playerStreamMap = string.Empty;
var playerConfig = GetInsideBraces(pageSource, ";ytplayer.config = ");
if (playerConfig.Length != 0)
{
playerConfig = Regex.Unescape(playerConfig);
var sMatch = Regex.Matches(playerConfig, "[&,]s=");
//check for 's' signature
if (sMatch.Count != 0)
{
var fmt = Regex.Match(playerConfig, "\"url_encoded_fmt_stream_map\": \".*?\"");
if (fmt.Groups.Count != 0)
{
playerStreamMap = fmt.Groups[0].Value.Split(':')[1].Replace("\"", "");
}
}
}
// Trim possible leading space
playerStreamMap = playerStreamMap.Trim();
var urlMap = string.IsNullOrEmpty(playerStreamMap) ? HttpHelper.ParseQueryString(videoInfo)["url_encoded_fmt_stream_map"] : playerStreamMap;
string[] splitByUrls = urlMap.Split(',');
var url = string.Empty;
foreach (string str in splitByUrls)
{
var queries = HttpHelper.ParseQueryString(str);
if (queries.ContainsKey("url"))
{
if (queries.Keys.Contains("sig"))
{
url = string.Format("{0}&fallback_host={1}&signature={2}", queries["url"], queries["fallback_host"], queries["sig"]);
}
else if (queries.Keys.Contains("s"))
{
string sParam = queries["s"];
string sDecoded = DecodeSignature(sParam);
url = string.Format("{0}&fallback_host={1}&signature={2}", queries["url"], queries["fallback_host"], sDecoded);
}
if (!url.Contains("ratebypass"))
url += "&ratebypass=yes";
}
yield return new Uri(url);
}
} |
[Part II/II] Second, I changed the DecodeSignature() method, so that it works with the signatures I'm getting - which are in the form of (42 chars).(42 chars). I've taken the algorithm from a link that smurz gave here: https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/youtube.py For everyone's convenience, here is the original code (in Python) def _decrypt_signature(self, s):
"""Decrypt the key"""
if len(s) == 88:
return s[48] + s[81:67:-1] + s[82] + s[66:62:-1] + s[85] + s[61:48:-1] + s[67] + s[47:12:-1] + s[3] + s[11:3:-1] + s[2] + s[12]
elif len(s) == 87:
return s[62] + s[82:62:-1] + s[83] + s[61:52:-1] + s[0] + s[51:2:-1]
elif len(s) == 86:
return s[2:63] + s[82] + s[64:82] + s[63]
elif len(s) == 85:
return s[76] + s[82:76:-1] + s[83] + s[75:60:-1] + s[0] + s[59:50:-1] + s[1] + s[49:2:-1]
elif len(s) == 84:
return s[83:36:-1] + s[2] + s[35:26:-1] + s[3] + s[25:3:-1] + s[26]
elif len(s) == 83:
return s[52] + s[81:55:-1] + s[2] + s[54:52:-1] + s[82] + s[51:36:-1] + s[55] + s[35:2:-1] + s[36]
elif len(s) == 82:
return s[36] + s[79:67:-1] + s[81] + s[66:40:-1] + s[33] + s[39:36:-1] + s[40] + s[35] + s[0] + s[67] + s[32:0:-1] + s[34]
else:
raise ExtractorError(u'Unable to decrypt signature, key length %d not supported; retrying might work' % (len(s))) Unfortunately, I've been unable to find videos with other signature forms (I suspect these are location-dependent), so my method won't work with signatures other than (42 chars).(42 chars). private static string DecodeSignature(string s)
{
string result = string.Empty;
switch (s.Length)
{
case 86: // (42 chars).(42 chars)
result = s.Substring(2, 61) + s[82] + s.Substring(64, 18) + s[63];
break;
/* Other signature lengths will go here */
default:
throw new ArgumentException("Can't decrypt signature: " + s);
}
return result;
} If any of you are getting different signatures, and want to convert the Python code to C#, you can use the online Python compiler at http://www.compileonline.com/execute_python_online.php, to make sure that your code is identical. |
Its mostly applied to vevo links, you dont have to worry about country specific content,as its widely applied to them in any region. |
I just published a NuGet pre-release package that hopefully fixes this issue. If everything works, I'll release an non-alpha package. |
Well, the signature decripting procedure is contained in the YouTube player itself. If you decompile it (I used ShowMyCode.com) you'll see the class called SignatureDecipher written in an Actionscript. The "decipher" function produces the correct signature. public class SignatureDecipher {
|
@smurz The latest commit does exactly this |
the last -pre didn't solve the problem of 403: |
Ok guys, I'm finally able to reproduce the 403 error without using a proxy, for this video: This makes everything a lot easier |
Hi.
I found it in a .swf into the YT web page, as suggested above. The (long) function ported before from gantt's script doesn't work anymore. This solution works also for all the other "vevo" videos. Hope this helps. bye. |
I'll tack this on here:
Also with length of 86 might have changed to ( javascript :P ):
|
Closing as duplicate of #16 |
http://www.youtube.com/watch?v=kDWgsQhbaqU
many others
The text was updated successfully, but these errors were encountered: