[Youtube Issue] Error when loading from YouTube (429/captcha form)

[jagrosh]

Describe the bug
YouTube has recently starting enforcing ratelimits. This is likely because streaming the data directly harms their service (as they receive no ad revenue when videos are streamed in this way). When your IP is ratelimited, no data from YouTube can be loaded, and attempting to load anything from YouTube will result in an error.

Does this affect you?

• If you see Invalid status code for video page response: 429 in error messages, it means that your IP has been ratelimited.
• If you see Sorry for the interruption. We have been receiving a large volume of requests from your network. in error messages, it means that YouTube is is requesting a captcha for your requests.

Solution
There is no solid solution to this at the moment. Your best options are:

• Turn off the bot for a few days (and don't attempt to use anything that connects to YouTube from the same IP address)
• Host from a different IP address

Version Info
This affects all connections to YouTube, regardless of which version of JMusicBot is used.

Additional Info
Here's the information that YouTube/Google provides when a music bot (or any service really) gets ratelimited:

Our systems have detected unusual traffic from your computer network. Please try your request again later. Why did this happen?
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.
This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. Learn more
Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.

[TheKleopiNetwork]

Add the option for a local persistant music cache so music from youtube has to only be streamed (and saved) once.
Should reduce the load of the autoplaylist and frequently requested tracks.
Mind licenses.

[jagrosh]

A cache of songs is something I've thought about before (and will continue to think about), but there are several issues.

Lavaplayer (the internal library used for loading youtube and other sources) is strictly a streaming tool, and thus the songs that are streamed never have associated files. To cache music, all of the streamed packets would need to be collected, encoded, and saved to a valid format. This would technically be possible, but would break down when streaming live sources or when seeking within tracks. Therefore, the songs would need to be downloaded separate from the stream, which would mean potentially worsening the ratelimit issue (depending on when things were downloaded).

This would additionally require a lot of work to write such a downloading & encoding system, or adding new dependencies like python & youtube-dl to handle the downloading. This would also require a new searching mechanism to handle caching the results of searches (as even searching - not even playing - can affect ratelimiting).

Other minor issues include disk space and search results changing over time, both of which could potentially be solved by having a time- or disk-size-based expiry system for the cache. Licenses and copyrighted content would be another issue.

Overall, I think the easiest solution for the majority of people would be to just play more local files; use local files or mp3 urls in playlists, and keep other local files under simple and easy-to-access filenames. Adding play shortcuts (key-value pairs of keyword to song uri) might be a feature to assist with this as well.

[johncsuti]

This is true, I have been hit with a rate limit but my friend that works for google’s YouTube department said that if you fake a browser agent it will make it look like a normal person. This will not initiate a rate limit as soon or at all in some cases. Maybe something we should look into. I have connected my system to a squid proxy that fakes the browser agent and have not been hit since. The bot has been running on YouTube every day since May.

[johncsuti]

This is also promising
sedmelluq/lavaplayer#232

[jagrosh]

if you fake a browser agent it will make it look like a normal person

I believe lavaplayer already does this, but this is definitely not the only factor. Google has different limits for different IP ranges as well. For example, running the bot on a home network (consumer IP block) will be far less-likely to get locked out than running it from a known cloud network.

This is also promising
sedmelluq/lavaplayer#232

That doesn't look promising for small, self-hosted bots. It seems more aimed towards larger bots that likely have a lot of IP addresses at their disposal (or can afford to purchase blocks of IPs) that can be cycled through. I don't IP cycling will be useful for the average JMusicBot user, who likely hosts on their home network or a VPS with only one IP (or even a shared IP).

Also, it seems that another new issue is popping up: sedmelluq/lavaplayer#248

Overall, it seems like Google is very intent on stopping music bots & other similar applications, so my instinct would be to move away from using YouTube as the primary source for songs instead of constantly jumping through ever-changing hoops.

[streamkiller]

https://hastebin.com/oguyocumet.xml

As requested console ouput of the error

[streamkiller]

I've copied the complete folder and ran it on another pc and it worked like a charm, I am gonna look into it if I can find the issue that causes this.

I will do a system restore to 11 days ago when it was working good, the programs that were updated are these (top list is currently installed bottom how it was 11 days ago):
https://gyazo.com/33f94c18e1cc4876380c4280477d34d9

[jagrosh]

Based on the console output, this looks like that is a similar issue; same cause (a lot of traffic) but instead of a 429, YouTube is requesting you complete a captcha. You might be able to complete that in your browser, but I don't think there is any solid/long-term solution other than just keeping the bot offline for a while, and making sure that you're not opening too many streams at once.

<p class='largeText'>Sorry for the interruption. We have been receiving a large volume of requests from your network.</p>

<p>To continue with your YouTube experience, please fill out the form below.</p>

[johncsuti]

Yeah the captcha would only complete for one request to go through not anymore so even if he completed it it would not help