Inconsistent get_transcript results

[jamirian071498]

When I repeatedly call "get_transcript" with the same video id, sometimes I get back the transcript, while other times I get back a "VideoUnavailable" error. As far as I can tell, this can happen with any youtube video. I am using version 0.3.1.

[jdepoix]

Are you doing those calls in a short amount of time, or over a bigger time span? And about how many calls are we approximately talking?
While I never was able to replicate this, multiple users have reported that they ran into YouTube rate limits, when doing many requests in a short amount of time. Unfortunately there's nothing this module can do about limitations set up by YouTube itself.
However, maybe I could probably add a custom Exception for this if I could find out what the actual response is which YouTube sends in this case, but as I said I just can't replicate this.

[joacosaralegui]

Hey guys. I see how this may be a follow up from this issue. As far as i was able to determine, YouTube tends to block this kind of continuous requests on STATIC IP servers.

In my case for example I had a Google Cloud VM running with a fixed external IP, and every "x" days of requests (around 2000 requests per day, for 3 or 4 days) it got banned. Then i would switch the IP of the server and it was good to go for another 4 days, and then it was blocked and had to come back to the original IP and so forth.

This is not the case with our local IPs as usually the ISP gives you a dynamic IP or hides it behind a double NAT, so maybe thats the reason you can't replicate the error.

What I did to fix it was to setup a Google Cloud VM whitout external IP, and make a Google Cloud NAT to manage that VM connection to internet, so now the requests go out with several different IPS, managed by Google. So far, it has worked ok.

So my recommendation would be to try to avoid static IP's and that kind of stuff.
Dont know if that's your case but hope it helps!

[jdepoix]

@joacosaralegui thanks for the insights!

[jamirian071498]

I do not believe my issue has to do with the YouTube API rate limits. The rate limit is apparently 10000 requests per day and I am using somewhere around 5 requests per hour.

One thing I have noticed is that the API works consistently when I call it locally, but not when I call it from an AWS lambda. If possible any help would be greatly appreciated!

[jdepoix]

So if you're only having this issue when using AWS, I could imagine that AWS is using an IP address for those requests which is also used by others. Therefore your rate limit is shared between multiple users. However, this could be difficult to verify, or work around. Is there a way to have lambdas use a dedicated IP for external calls? 🤔

[jdepoix]

@jamirian071498 I will close this now as I don't think there's a problem at hand which can be solved by this module, but feel free to share your learnings if you find out more about this problem.