Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Video unavailable #811

Closed
simmstein opened this issue Oct 9, 2019 · 104 comments
Closed

Video unavailable #811

simmstein opened this issue Oct 9, 2019 · 104 comments
Labels
bug Something isn't working

Comments

@simmstein
Copy link

simmstein commented Oct 9, 2019 •
edited
Loading

Hello,

Versions

  • Invidious version: 0.19.1-e03b4b7 (sha256 of the compiled file 271291b36749ee7bae51c3834d08da8eb8bc7e4842525489c56a3399ef0b5d17)
  • OS: Debian 10 (amd64)
  • Crystal version: 0.31.1 [0e2e1d067] (2019-09-30)
  • Database: Postgresql 9.6+181+deb9u2
  • Logs: empty (no error)

What happened

I click on a video's thumbnail and the message "Video unavailable." is shown.

Expected

Play the video clicked.

Information

The video is played using invidio.us (0.19.1-e03b4b7 master).
@simmstein
Copy link
Author

From the server:

$ youtube-dl "https://www.youtube.com/watch?v=XXXXXXXXX"
[youtube] -OZ--RQcaY4: Downloading webpage
[youtube] -OZ--RQcaY4: Downloading video info webpage
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
Traceback (most recent call last):
  File "/usr/bin/youtube-dl", line 11, in <module>
    load_entry_point('youtube-dl==2019.1.17', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 477, in main
    _real_main(argv)
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 467, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 2002, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 793, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py", line 1672, in _real_extract
    token = video_info.get('token') or video_info.get('account_playback_token')
AttributeError: 'NoneType' object has no attribute 'get'

To be confirmed: this is not a issue from invidious.

@atahanacar
Copy link

12 hours ago, I had the same issue, but the issue is no longer there right now. Seems like Youtube changed some policies regarding connections.

@Perflyst
Copy link
Contributor

Perflyst commented Oct 9, 2019

Your IP got blocked by Google, other instances face the same issues.

An example to test if you are blocked:

root@invidious:~# curl 'https://www.youtube.com/watch?v=ukDht4jRcT0&gl=US&hl=en&disable_polymer=1&has_verified=1&bpctr=9999999999'
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com/sorry/index?continue=https://www.youtube.com/watch%3Fv%3DukDht4jRcT0%26gl%3DUS%26hl%3Den%26disable_polymer%3D1%26has_verified%3D1%26bpctr%3D9999999999&amp;hl=en&amp;q=EgRf2BjmGJjv9uwFIhkA8aeDS2A2A_pZaQC7pukXkNR0Cy6xim-gMgFy">here</A>.
</BODY></HTML>

@PureTryOut
Copy link

I have the same error since like a week, but I get a full result if I try that test of @Perflyst.

@simmstein
Copy link
Author

simmstein commented Oct 9, 2019 •
edited
Loading

By using my server as a proxy, I had to fill recaptcha to play a video on youtube.com (private tab).
By using curl, I had the full page and not a redirection.

@omarroth omarroth added the bug Something isn't working label Oct 9, 2019
@OdinGitDat
Copy link
Contributor

OdinGitDat commented Oct 9, 2019 •
edited
Loading

I have the same for every video on v0.19.1-d36c536 vanilla docker container.

How do other instances get around this if it's a block on Google's end?

@omarroth
Copy link
Contributor

omarroth commented Oct 9, 2019

You will also want to check the output for /das_captcha which will also indicate rate-limiting:

$ curl 'https://www.youtube.com/watch?v=ukDht4jRcT0&gl=US&hl=en&disable_polymer=1&has_verified=1&bpctr=9999999999' -s | grep '/das_captcha
<form action="/das_captcha" method="POST">

Testing with youtube-dl should return a 429 as well if this is the case.

As @Perflyst mentioned, this is due to rate-limiting/IP banning by Google. This has been an issue for other projects as well, see fent/node-ytdl-core#444, Tyrrrz/YoutubeExplode#292, and ytdl-org/youtube-dl#21729.

If IPv6 is available for your box, I would recommend testing with force_resolve: ipv4 or force_resolve: ipv6 in your config. Otherwise, you'll unfortunately have to wait for Google to unban you or get a new IP.

@omarroth omarroth mentioned this issue Oct 9, 2019
Closed
@OdinGitDat
Copy link
Contributor

When I use a browser on my server to watch a video on youtube it asks for a captcha and then lets me watch videos for as long as the cookie stays in the browser.

Isn't there a way to relay this captcha to invidious webmasters and use the cookie for subsequent requests?

@omarroth
Copy link
Contributor

omarroth commented Oct 9, 2019

It's possible. It's just not a practical workaround IMO since the cookie will only be valid for ~3 hours.

@jaruiz59
Copy link

Yep getting the same message .Video unavailable.

@asorel1942
Copy link

I don't quite understand what everyone in this thread means by hosting invidio locally. Is there a way to utilize the service without a browser? If so, will this be able to circumvent the "video is unavailable" messages everyone seems to be getting?

@BurnyBoi
Copy link

Does this mean invidio.us has been IP blocked by Google?

@HumanG33k
Copy link

Doing that they provide bad experience and people want to move back to YT. It’s a better option that blocking IP which provide badbuzz. They have done some similar things for ie when they downgrade qualities based on user agent in order to push FF users to Chrome.

@OdinGitDat
Copy link
Contributor

OdinGitDat commented Oct 11, 2019 •
edited
Loading

You can always host it locally to avoid the IP banning.

@nchristensen Can you please stop spreading this as a solution? The OP specifically has this problem on his own instance and so do I. Yes I am the only user of my instance, I watch max 10 videos a day, and I've still been banned twice within 3 days. Both my bans lasted 12 hours and afaik they all do.

I don't like to be pessimistic, but I think this may be the end of invidious if relaying a captcha/cookie is impractical as omarroth said.

@Booteille Booteille mentioned this issue Oct 11, 2019
Closed
@ghost
Copy link

ghost commented Oct 11, 2019 •
edited by ghost
Loading

Why not redirect instance traffic to tor > towards youtube? The ip of the instance will be protected from being banned. Or use some open proxies and couple it all with proxychains and tor, it would be perfect !

@omarroth omarroth mentioned this issue Oct 11, 2019
Closed
This was referenced Oct 13, 2019
Closed
Closed
@simmstein
Copy link
Author

@coolcoderplus it could work but this is not a fix.

@Gert-dev
Copy link

@coolcoderplus I regularly use Tor and am often blocked by Google services due to "too much traffic". The list of endpoints is constantly changing, but likely the majority remains fairly constant. As also only a subset of Tor nodes function as actual endpoints, the pool of IP addresses is also said subset.

In a grossly simplifying sense, it's comparable to a dynamic list of proxies, of which the IP addresses end up flagged after a while.

@sentriz
Copy link

sentriz commented Oct 15, 2019

i am experiencing this on my home network too. youtube-dl doesn't work either. but surprisingly newpipe on my phone is working

@2secslater
Copy link
Contributor

2secslater commented Oct 15, 2019 •
edited
Loading

I have a somewhat public instance (other folk can access it) running on my home network, and has been for rougly three days now. I have had the same IP address, using the instance most of the time and I haven't been sniped by Google (yet). My instance is running commit 7b77f20

@atahanacar
Copy link

Invidious and youtube-dl are not working on my private instance running on a VPS. However, when I connect to the VPN hosted on the same VPS, I can access Youtube and youtube-dl works fine.

@m4teh
Copy link

m4teh commented Oct 17, 2019

I've also been hit by this (self hosting Invidious via outbound VPN provider).
They must have done a big VPN sweep recently. I get a CAPTCHA when I browse YouTube's website too.

Is work being put into adding an ability to perform the CAPTCHA from the Invidious interface? That's the easiest solution I can think of.

@inhji
Copy link

inhji commented Oct 17, 2019 •
edited
Loading

My solution workaround was to snapshot the whole instance and restore it to a different VPS with a new ip address.

@HumanG33k
Copy link

it s hard workaround not really easly setup for people with less money/skills.

I guess there is no way to contact google for help us … they probably say use the api and we loose some privacy improvement provide by invidious.

@atahanacar
Copy link

@HumanG33k This is kind of unrelated but I am curious. If I am using an off-site server as a proxy, does it matter if I am using the API or an extractor?

@sentriz
Copy link

sentriz commented Dec 3, 2019

I meant to say I'm selfhosting invidious so newpipe and invidious would be accessing youtube from my same home IP I think

@elypter
Copy link

elypter commented Dec 4, 2019

maybe ipv6 helps if you get a sizable subnet for your server. google can still block a whole range but they might be more hesitant, especially if they dont know the network topology

@gerroon
Copy link

gerroon commented Dec 4, 2019 •
edited
Loading

I think that Google is using Webrtc and Browser stuff to discover different computers inside the same network, the reason I am saying is that I am blocked by Youtube but my wife's computer access to Youtube is not blocked and we are on the same wifi.

They might be tagging the "good" ones and letting them have access and block everything else from that network.

@sentriz
Copy link

sentriz commented Dec 4, 2019

hmm @elypter using a static ipv4 address at home

@haizrul
Copy link

haizrul commented Dec 4, 2019

Problem happen again. Most instances are down. :(

@tleydxdy
Copy link
Contributor

tleydxdy commented Dec 4, 2019

it's not suprising that they can detect devices using the same IP, google basically live off of that.

@narcolepticinsomniac
Copy link

I only got blocked for a day. I assume this commit helped, because of the timing. Still having connectivity issues though.

I only use invidious as a fallback with a script which redirects on error. The majority of the time, if youtube's blocked an embed for whatever reason, a redirect to invidious pulls the vid. Ever since it started kinda working again, at least half the time, I'm getting waiting for invidio.us....

@elypter
Copy link

elypter commented Dec 4, 2019

They might be tagging the "good" ones and letting them have access and block everything else from that network.

this is all in theory preventable. its just a matter of how much effort you put in to mimic a normal browser. the problem is that solutions that work one day might not work anymore the next. so decentralization should always be of similar importance

@Vagmer
Copy link

Vagmer commented Dec 5, 2019

NewPipe is quite different from Invidious, even if self hosted. NewPipe is just an alternative YT player/browser. It's like using a different, specialized web browser to access youtube.com. Or like using youtube-dl to do so.

@omarroth omarroth mentioned this issue Dec 5, 2019
Closed
@simoniz0r
Copy link

simoniz0r commented Dec 6, 2019 •
edited
Loading

I've noticed that vid.wxzm.sx seems to be the most reliable instance for getting a list of available videos even though it is blocked from getting data for individual videos.

This has got me thinking a bit... what if you were to add an option to Invidious that would handle most of the calls to YouTube on the client side through JavaScript? You could still allow people to use the method of getting the info from the invidious server if they wish, but running the YouTube calls client side through JavaScript would allow an instance to keep working even if it has been blocked.

You could maybe also show application writers how they can scrape the info from YouTube's website relatively easily (basically the same JSON you return is in the HTML of the video on YouTube's website) which could allow them to use Invidious as a way to get a list of videos, but not rely on it for getting information for specific videos.

@elypter
Copy link

elypter commented Dec 7, 2019

sounds good as a last resort but it should only be applied if distributing the work and caching is not enough.

@Vagmer
Copy link

Vagmer commented Dec 8, 2019

Yeah... as I discussed in a previous comment #811 (comment) that would only be good as a last ditch effort to just keep the site(s) working in name only, basically, if forsaking most of their original purpose. At that point there's little difference from viewing the site/running the JavaScript from a local HTML file instead.

@elypter
Copy link

elypter commented Dec 8, 2019

there is still a great difference. there is no code made by google running on your system. and the amount of requests to google servers is still vastly reduced.

@Perflyst
Copy link
Contributor

Perflyst commented Dec 8, 2019

but running the YouTube calls client side through JavaScript would allow an instance to keep working even if it has been blocked

-> "Does not require JS to play videos" in the readme

I am strongly against this feature, use youtube.com if your invidious instance is blocked or tell the admin to use anti-captcha.com (which currently solves all problems)

@simoniz0r
Copy link

simoniz0r commented Dec 8, 2019 •
edited
Loading

-> "Does not require JS to play videos" in the readme

I am strongly against this feature, use youtube.com if your invidious instance is blocked or tell the admin to use anti-captcha.com (which currently solves all problems)

This would be an option, not a requirement. You could still use Invidious without JavaScript with what I suggested above.

Also, using youtube.com if the instance is blocked rather than tolerating an option to have Invidious use a small amount of JavaScript to get the info about videos would be about the last thing I would want to do. YouTube's website is very laggy on my computer; it takes a good 30 seconds to even finish loading the webpage most times due to their heavy use of JavaScript. The amount of JavaScript required for what I suggested above would be pretty minimal and very unlikely to impact performance much at all.

@Vagmer
Copy link

Vagmer commented Dec 10, 2019

Kind of tired of repeating the same thing - maybe it's totally useless to, and I never said anything new to begin with - but again, that's basically asking Invidious to become, or add an option to be, a locally running YouTube client (viewer), which is almost a completely different kind of project than it is now, one which has little benefit of being hosted on a remote website at all, and doesn't even have to be running within a web browser. It would actually be a waste of any Invidious development time, unless that becomes the only way for the project to kind-of-live, I guess. If this ("connecting to and parsing YouTube in an alternative way") is what you want then you don't strictly need Invidious. Many alternative YT clients already exist for various platforms and have existed before Invidious, the uniqueness of which is that it's not really a client.

@Vagmer
Copy link

Vagmer commented Dec 10, 2019

@Perflyst

tell the admin to use anti-captcha.com (which currently solves all problems)

Good to hear. You know of instances currently employing such a solution? This one seems really really cheap, too. I'm sure I and others wouldn't mind donating to help make use of such.

@omarroth
Copy link
Contributor

omarroth commented Jan 20, 2020 •
edited
Loading

You know of instances currently employing such a solution?

The two that I can recall are https://invidio.us and https://invidious.snopyta.org, although I suspect others are also using this feature.

Closing since this has been addressed by d46b26e and 8236036. I would recommend taking a look at this comment in #957 for more explanation.

@Perflyst
Copy link
Contributor

Perflyst commented Jan 21, 2020 via email

@omarroth omarroth mentioned this issue Mar 26, 2020
Closed
@Perflyst Perflyst mentioned this issue May 16, 2020
Closed
@TheFrenchGhosty TheFrenchGhosty mentioned this issue Feb 21, 2021
Open
@tio-trom
Copy link

tio-trom commented Jun 5, 2021

I am too wondering how to try and fix this with our instance https://ytb.trom.tf/ - we have ipv6 enabled already. We keep the instance updated and all that. So basically what happens is youtube blocking the server's IP? That's the main issue? If so, is there any fix in sight for this?

@syeopite
Copy link
Member

syeopite commented Jun 5, 2021 •
edited
Loading

@TROMsite

I am too wondering how to try and fix this with our instance https://ytb.trom.tf/ - we have ipv6 enabled already. We keep the instance updated and all that. So basically what happens is youtube blocking the server's IP? That's the main issue? If so, is there any fix in sight for this?

Yes. It's been implemented for quite awhile now. Please checkout the documentation page on how to setup anti-captcha.

And for the record, please don't comment on such an ancient thread!

@tio-trom
Copy link

tio-trom commented Jun 5, 2021

Ah I see. So you pay for an anticaptcha to get around this... Shouldn't some proxy solutions be better? Maybe I'll open another issue so we don't revive such zombie threads. Thanks!

@unixfox
Copy link
Member

unixfox commented Jun 5, 2021 •
edited
Loading

Ah I see. So you pay for an anticaptcha to get around this... Shouldn't some proxy solutions be better? Maybe I'll open another issue so we don't revive such zombie threads. Thanks!

Proxy support is tracked in #1036. No real need to open another issue, anticaptcha is unfortunately the only way for the moment. Blame Google for implementing recaptcha, we cannot do anything about it.

@syeopite
Copy link
Member

syeopite commented Jun 5, 2021

Alternatively #879 could be a possible solution. Either way, I'm locking this issue.

@iv-org iv-org locked as resolved and limited conversation to collaborators Jun 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests