Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Circumventing geo-restrictions #92

Open
narcolepticinsomniac opened this issue Aug 6, 2018 · 25 comments

Comments

@narcolepticinsomniac
Copy link

commented Aug 6, 2018

I thought we were already circumventing geo-blocking, but we're not.

Blocked in the U.S.:
https://invidio.us/watch?v=xpWu4_hp00M

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2018

It would be good to start building a list of videos that are "Blocked in US, Blocked in DE, etc".
Do you happen to know in which regions the video linked above is allowed?

@narcolepticinsomniac

This comment has been minimized.

Copy link
Author

commented Aug 6, 2018

Nah, I can't get it to play using any VPN or proxy. Maybe not a great example, as I suppose it's possible that the link is just broken and the geo-restriction aspect isn't really the issue.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2018

Here's an example of a video that is allowed in DE but not US: https://invidio.us/watch?v=rV4Mrl2hQro
It appears that in /videoplayback URLs, for example medium quality for the above video YouTube now returns 403 when the ip parameter does not match the requesting IP.

Helpful link: https://www.youtube.com/get_video_info?video_id=rV4Mrl2hQro

@narcolepticinsomniac

This comment has been minimized.

Copy link
Author

commented Aug 6, 2018

Haven't I seen something like bypassip= params for googlevideo URLs?

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2018

You probably saw ratebypass, I don't happen to know what it does.
A few quick tests:

  • I misspoke, YouTube only cares about ip= when the IP is for a video that is not in one of the allowedRegions.
  • Attempting to access geo-blocked videos through Tor is fine, as long as the exit node is in the correct region.
@narcolepticinsomniac

This comment has been minimized.

Copy link
Author

commented Aug 6, 2018

No, I knew I've seen it. Went through my clipboard manager and found one. IDK if it actually works, but:

https://r3---sn-ab5szn7s.googlevideo.com/videoplayback?fvip=3&source=youtube&signature=230E3DC2DF22B86A40021B84ACAD8839AB515422.0D14359F1455503A0BE6096B7B8B1EFB4CD16927&ipbits=0&pl=18&id=o-AIfZC-oGvCCpSym9jRjuh5PzQ3mlPwK2nLKOpIQb6UFj&dur=180.419&mime=video%2Fmp4&key=cms1&ip=2604%3Aa880%3A400%3Ad1%3A%3A180%3Ad001&ei=fGFjW4_5I5SBhwaVtInwDg&c=WEB&requiressl=yes&sparams=dur,ei,expire,id,ip,ipbits,ipbypass,itag,lmt,mime,mip,mm,mn,ms,mv,pl,ratebypass,requiressl,source&itag=22&lmt=1532956145274494&expire=1533261276&ratebypass=yes&redirect_counter=1&rm=sn-ab5eel7s&req_id=ef8047609366a3ee&cms_redirect=yes&ipbypass=yes&mip=00.000.000.000&mm=31&mn=sn-ab5szn7s&ms=au&mt=1533239832&mv=m

(towards the end)

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 6, 2018

Doesn't appear to do anything. A while back I put some work into decrypting/resigning /videoplayback URLs but never got anywhere. Notice that in the above URL there is akey and signature, which prevents tampering.

@narcolepticinsomniac

This comment has been minimized.

Copy link
Author

commented Aug 6, 2018

ipbypass=yes seemed like it may be relevant. I was skeptical, but figured it might be worth mentioning.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 13, 2018

Working on this right now, currently can get info on the videos linked above. https://invidio.us/watch?v=xpWu4_hp00M appears to be allowed in Germany, Japan, Russia, Thailand (of the regions I've tested). The last thing that needs to be done is to proxy the files themselves, although obviously that will be more bandwidth intensive.

@dimqua

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2018

I suggest you to just add this information to the video page, so users can use VPN or proxy to watch videos that are allowed in specific countries.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 13, 2018

That's fair enough. I'm already managing downloads for livestreams, and will do the same for #34 and #90, so I thought I might do that here as well. I do plan on adding info to the video page, something like blacklisted countries or whitelisted countries (depending on which is bigger), and that info is already available though the API.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 13, 2018

Just pushed fada57a, which provides blacklisted/whitelisted countries. I'd take a look at the regions Invidious tries to bypass. It's a pretty small list, but it seems to be able to bypass most of the videos that are geo-restricted. If there are other regions that are allowed by a significant amount of geo-restricted videos, I'll consider adding those regions.
Region codes correspond to those in ISO 3166.

@dimqua

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2018

regions that make up a significant amount of traffic

How can you measure that?

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 13, 2018

I think what I mean is the regions that allow a user to watch the largest amount of content. So Germany, US, JP, etc. I can't measure it very well, but I think there is a metric where some percentage of videos are exclusive to region x, so it should be added to the list that Invidious tries to bypass.

I also just clarified my position a bit: the regions that are allowed by the largest number of geo-restricted videos should be added.

@Zero3K

This comment has been minimized.

Copy link

commented Aug 15, 2018

I wonder if its possible to get it to work without having to proxy the video file itself. If it is, then that would be nice.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Aug 15, 2018

Attempting to access a geo-blocked file returns a 403 as well, so it appears the only solution is to proxy that as well. This is also an issue for #90: Google will block access to the file itself in some instances, so this is likely the only solution unfortunately.

@narcolepticinsomniac

This comment has been minimized.

Copy link
Author

commented Dec 2, 2018

Not sure if you ever successfully implemented a geo-restriction circumvention. I kinda figured you had bc I hadn't run into any problems in quite a while, but this issue's still open, so perhaps not. Just ran into one though, and besides being geo-blocked, the page takes forever just to load an error message.

https://youtu.be/d9sApXSDtnE

@omarroth

This comment has been minimized.

Copy link
Owner

commented Dec 2, 2018

Currently the system in place is to cycle through proxies in specific regions, so most of the videos mentioned work (with dash enabled so videos are also proxied). The reason it's still open is because there's still a lot of room for improvement.. I expect it to be closed alongside #90.

The reason geo-blocked videos can take a long time to load is if Invidious can't find a proxy or if none of the regions Invidious tries return a valid response. I would like to fix that before I close this.

@Perflyst

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

Would it be possible to use Tor for a country from which you dont have a proxy? You can set the exit nodes country if you use tor.

@omarroth

This comment has been minimized.

Copy link
Owner

commented Apr 17, 2019

This was suggested in the Matrix server. The issue is that it is not possible to determine where a video is unrestricted without being in an unrestricted region.

The way the current system works is by cycling through proxies in different regions, and finding one where the video is unblocked. Doing this through Tor would put an enormous amount of strain on the network, since it would require building a circuit for every region that is tried.

Also important to note is that video files still have to be proxied from somewhere the video is unblocked. Doing this through Tor is something I'd also rather not do, since it similarly puts strain on the network.

@elypter

This comment has been minimized.

Copy link

commented Apr 18, 2019

you could reduce cycling if you create statistics about how many video requests are blocked in which country

@jeroen7s

This comment has been minimized.

Copy link

commented Apr 30, 2019

This is a really annoying issue, lots of video's still don't work for me, especially with music video's like from Vevo, the proxy doesn't seem to fix it for me.
even trying alternate instances located closer to me don't work.
i'm from Belgium, checking the German, Norwegian, Finnish instance under "alternative instances" doesn't work
the one from the Netherlands seems offline, and there's no Belgian instance
EDIT: seems like the proxy DOES fix the issue for me, just thought it worked automatically, but seems like you have to enable it in the settings :)

@jeroen7s

This comment has been minimized.

Copy link

commented May 5, 2019

proxy-ing does really cause bandwidth constraints though
can't you just catch the video information/url using a proxy, instead of proxy-ing the entire video through invidious servers? that would at least solve the issue that video's don't play, even though they are available in your country when visiting in youtube directly

@omarroth

This comment has been minimized.

Copy link
Owner

commented May 26, 2019

Sure, it just means you have to track or store the user's region in order to provide correct links. I believe there was also some discussion on this in #460, and I can see it being reasonable to allow a user to "prefer" a different region than where the instance is hosted.

Generally, it's preferable to proxy videos instead of trying to get native links. Mainly because you don't need to manage and retrieve the same video multiple times in different regions which allows you to cut down on the number of requests as well as a couple other optimizations, and since video streams for 1080p (#43) and region-blocked videos need to be proxied through the instance anyway it also makes sense to use the same solution here.

Although I do very much agree that proxying has constraints I believe it is the best solution in this case.

@tleydxdy

This comment has been minimized.

Copy link
Contributor

commented May 29, 2019

perhaps an opt-in "proxy server" config so that my instance can proxy requests for other invidious instance? and pair that with a introduction system so servers can gossip about proxy servers they know, and which region are they.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.