Response status code does not indicate success: 429 (Too Many Requests)

[haichay88]

it seem issue from youtube limit rate, i have same issue
"message":"Response status code does not indicate success: 429 (unknown)."
although one request only

[Tyrrrz]

How often are hitting that?

[haichay88]

sometime , i received,it seem issue from youtube limit rate,

[ComputerMaster1st]

Hi There,

I've managed to get a rough measurement on this issue as my discord bot get's flooded with 429s.

I've estimated we can have 21 downloads within the space of about a few minutes before being blocked for a certain time period. I think it resets at a specific time. Not sure. After further experimenting, it may be possible to get away with maybe 1-2 download requests per minute. This is not confirmed as it's just a possible estimate.

Only way we can test what the rate-limit is is to make a simple program that downloads a big playlist and adjust the rate-limit bit by bit until we get the sweet spot. Unless YT has rate-limits posted somewhere.

[Tyrrrz]

Hi @ComputerMaster1st ,
I'd say that it's weird that it starts throttling at ~20 downloads/minute, I'd expect YouTube to handle more. That said, I don't really have a solution to this given that it's server-side.

[ComputerMaster1st]

Hi There,

From what I've last heard, YT have changed stuff on their side to prevent anything & everything resembling a bot to download content from them. I know they've been fighting against bots for a long time, hence the whole signature nonsense.

[YoshiRulz]

Could it optionally send the Google login cookie with each request? Then individuals shouldn't trigger the bot detection.

[Tyrrrz]

@YoshiRulz you can do that now by injecting a custom HttpClient with your headers.

[Tyrrrz]

@ComputerMaster1st honestly, the way they're fighting bots with signatures looks more like it's a software specification coming from the stakeholders that the developers don't care too much about. YouTube is much better at deterring bots by doing their random changes and not so much with signatures.

[resulozlu]

hello I'm using, but can we overcome this problem with proxy server

[haichay88]

{"message":"Response status code does not indicate success: 429 (unknown).","data":{},"innerException":null,"stackTrace":" at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()\n at System.Net.Http.HttpClient.d__27.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at YoutubeExplode.YoutubeClient.d__35.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at YoutubeExplode.YoutubeClient.d__39.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at YoutubeExplode.YoutubeClient.d__43.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at

[Tyrrrz]

I recommend adding backoff-and-retry via Polly or similar

[resulozlu]

hello @Tyrrrz Can you teach us this method

[resulozlu]

@Tyrrrz gives an error when many downloads are made

[resulozlu]

Response status code does not indicate success: 429 (unknown).

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (unknown).

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

[HttpRequestException: Response status code does not indicate success: 429 (unknown).]
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
YoutubeExplode.d__35.MoveNext() +364
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
YoutubeExplode.d__41.MoveNext() +334
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
Facade.d__4.MoveNext() in D:\ASPNET\Youtube\Facade\YoutubeDownloader.cs:80
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
WebApplication.Controllers.d__0.MoveNext() in D:\ASPNET\Youtube\WebApplication\Controllers\DetailsController.cs:53
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +99
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +58
System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult) +97
System.Web.Mvc.Async.<>c__DisplayClass8_0.b__1(IAsyncResult asyncResult) +17
System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult) +32
System.Web.Mvc.Async.<>c__DisplayClass11_0.b__0() +58
System.Web.Mvc.Async.<>c__DisplayClass11_2.b__2() +228
System.Web.Mvc.Async.<>c__DisplayClass7_0.b__1(IAsyncResult asyncResult) +10
System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult) +34
System.Web.Mvc.Async.<>c__DisplayClass3_6.b__4() +35
System.Web.Mvc.Async.<>c__DisplayClass3_1.b__1(IAsyncResult asyncResult) +100
System.Web.Mvc.Async.WrappedAsyncResult1.CallEndDelegate(IAsyncResult asyncResult) +10 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult) +27
System.Web.Mvc.<>c.b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState) +11
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult) +47
System.Web.Mvc.<>c.b__151_2(IAsyncResult asyncResult, Controller controller) +13
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +22 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult) +28
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult) +10
System.Web.Mvc.<>c.b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState) +28
System.Web.Mvc.Async.WrappedAsyncVoid1.CallEndDelegate(IAsyncResult asyncResult) +29 System.Web.Mvc.Async.WrappedAsyncResultBase1.End() +49
System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +28
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result) +9
System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +152
System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +125

[Tyrrrz]

How many requests per minute are you making?

[Tyrrrz]

Looks like an entire test run crashed due to this error
https://ci.appveyor.com/project/Tyrrrz/youtubeexplode/builds/27671002

[Tyrrrz]

Is anyone still getting 429? Looks like CI stopped getting them for a while.

[Tyrrrz]

Looks like there's some discussion about this here: ytdl-org/youtube-dl#21729

[Tyrrrz]

I think ultimately the only solution here is to build a Polly policy to back off and retry on such failures. I can wrap this and similar exceptions in TransientException or something similar.

[Tyrrrz]

Decided to take a look at youtube-dl's issues board to see if they're dealing with something similar.
Needless to say I lol'ed.
ytdl-org/youtube-dl#23638

Luckily, I can reproduce it pretty consistently on the CI runner but not locally. That said, I'm not sure if this can be "fixed".

I've added tracing on every request/response to see what kind of information we can fish out.

[ComputerMaster1st]

Hi There,

Not sure if I should suggest this... Have you thought about fetching vids via YT Clones or do they end up going to the same endpoint aswell? (google in this case since vid urls point there)

Just a suggestion. Not sure if you've already considered it.

[Tyrrrz]

What are YT Clones?

[ComputerMaster1st]

YouTube Clones.

[Tyrrrz]

That didn't explain much

[ComputerMaster1st]

YouTube Clones are as the name implies. They're basically copies of youtube and one of them did have their own download links. Can't remember which one it was.

[Tyrrrz]

Hm, haven't used or seen any of them.

[ComputerMaster1st]

I don't know if using proxy servers would help either.

[Tyrrrz]

I'd say it should help but I'm wary to incorporate it in the library itself. You can use it on the consumer layer by supplying your own HttpClient to YoutubeClient.

[SlowLogicBoy]

It is basically not possible to fix this issue with this Library.

1. We don't know how many processes you are running to download/watch videos and how many requests you are doing to youtube/google video servers, none the less your whole local network.
2. We don't know the limit of requests per minute or second that youtube has put, because it differs from region to region and from network to network (i.e on CI servers it will fail more often because there might be other projects that also test some kind of connection to youtube)

How to prevent this issue:

1. As per @Tyrrrz recommendation use Polly to handle that exception and retry after some time
2. Put thresholds or timers on every how often you can make a request to youtube servers, and adjust them until it's stable (but keep in mind any TV's Phones PC's that are connecting to youtube to watch videos will also limit your request count as far as I've checked it is capped per public ip address)
3. God damn how many request you have to do to get this issue? Slow down people, I have done pretty heavy loads on youtube and still didn't get this issue.

[Tyrrrz]

I've read in some of the youtube-dl issues that the ban wears off after 12 hours or something like that, in which case Polly might not help, lol. Also, after adding diagnostics, I can see from the CI runs that the 429 errors are not accompanied by Retry-After headers or anything similar.

It is quite annoying that it fails so often on CI though. This is currently my biggest issue with this whole thing.

[YoshiRulz]

In my fork (different network stack) I've made a flag to toggle between real requests/responses and reading data from a file, then per-commit CI checks for regressions and I can manually check for YouTube changes (or set it up to run every month). Can you do something similar with HttpClient?

[Tyrrrz]

I can, but there's not much benefit in running CI on cached data. I'm not really worried about regressions because there's almost no business logic, but I do need to be sure that the YouTube integration is working.

Also, I think running it less often won't help because I was running it nightly before and it was still getting throttled once in a while. I'm guessing there are other projects that also send requests to YouTube from GitHub Actions and the external IP is shared between jobs.

[Jeager2]

I recommend adding Polly into the app to handle the 429 and rate limiting.

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.

https://github.com/App-vNext/Polly

Read over the github page above. A short ready through the page should give you a fairly good idea of what it can do with failures, specific exceptions, retry delays and complete failure scenarios.

The policies are quite well thought out imo.

Best wishes

[zjjohncq]

Today I got the 429 error. Is there any update or discovery?

[ComputerMaster1st]

Hi @zjjohncq,

I think they're going to implement "Polly" into it. I'm not sure at this time. The best thing to do is to slow down how often you hit YouTube at this point.

[zjjohncq]

Is there specific how long we need to wait beween 2 requests: 1 min? 10 min or longer?

[ComputerMaster1st]

No clue. Sorry.