New 403 errors with Google

[raRaRa]

Hey all.

Until recently, many Google Drive photos have been returning 403 error code when being accessed through images.weserv.nl.

Example URL: https://images.weserv.nl/?url=ssl:drive.google.com%2Fuc%3Fid%3D1ZuBjHMRlBfoPh4NB6F-mpmc9GoEB78CX%26export%3Dview&w=2048&h=2048

Here's the original URL which works: https://drive.google.com/uc?id=1ZuBjHMRlBfoPh4NB6F-mpmc9GoEB78CX&export=view

Could it be that they have recently started blocking this service due to increased traffic?

Thanks!

[raRaRa]

I've also created a bug report on the Google end about this: https://issuetracker.google.com/issues/157584226

[kleisauke]

Thanks for reporting this! It seems that Google is indeed blocking us if too many requests are made. For reference; this is the response we got:

We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
See Google Help for more information.

Besides that, we also get a text/html response occasionally:

Google Drive can't scan this file for viruses.
We are experiencing technical difficulties. Would you still like to download this file? test123.png (2.5M)

Unfortunately, there's not much we can do for this. I starred the above mentioned issue, maybe that could help.

[andrieslouw]

Unfortunately, there is not much we can do about this.

[abubelinha]

I have just tried with the gdrive file linked in the way #145 (year 2018), and it still worked.
So does still work the file linked above (as of today, march 2022).

But as for this issue, I understand this can change at any time, so we shouldn't trust gdrive as a good storage system to combine with weserv?
@raRaRa what did you end up doing?
Did you (or @kleisauke or @andrieslouw) get any answer from Google, whitelisting weserv.nl to avoid future blockings?

I was planning to use gdrive/weserv combination for some museum collections' images.
Thanks in advance for your advice
@abubelinha

[kleisauke]

I'm not sure if Google is interested in whitelisting our project. If you are able to host your own solution, it might be more appropriate to use the weserv filter; nginx directive in combination with static content, for example:
https://images.weserv.nl/static/lichtenstein.jpg?w=200

This is handled by default in the Docker image (just place your images inside the /var/www/imagesweserv/public directory):

images/ngx_conf/imagesweserv.conf

Lines 61 to 65 in d9b0cef

	location /static {
	weserv filter;
	alias /var/www/imagesweserv/public;
	}

This way you could ensure that it will never cause upstream errors. You can also use something like #314 (comment) to process images from only one domain.

[abubelinha]

Thanks but no, we are not able to self host this service.

So the choices of weserv beign blacklisted by Google depend on the number of recent requests of images, which might be low by the time I tried but can change at any time.

What if I provide a custom php access point for my google images?
(I guess the number of Google requests from my site would be much lower)
So I pass to weserv my own site url which contains the google file id, then my site downloads it from Gdrive on the fly and serves it to weserv for caching?
(of course it would use a limited whitelist of google ids, so it only passes our own files to weserv)

But I wonder what happens if I don't have an ssl certificate.
I can only provide an http url right now ... would weserv still serve the images through https?

Thanks

[andrieslouw]

We can and will serve through HTTPS, no matter the original URL. There is a large group of users using our service just to do so. And indeed, your proposed additional "PHP-proxy" may work as suggested, but the fastest (and most "in-control") solution would be to host this project yourself. If you only have access to PHP and have libvips installed, there is an older version of our project still here (but it is not supported or updated): https://github.com/weserv/images/tree/3.x

If you have a really old PHP environment without libvips, you could even use the very first version, 1.0: https://github.com/weserv/images/tree/1.x

But be reminded that all PHP versions are much slower than the current C++ 5.x branch of our project. As always: feel free to branch them and modify as needed for your project.

I'm not entirely sure when Google stops users from abusing their service, or when it flags abuse, you could try to ask Google about your specific use case, it may be that they have a special program to support museums and such. If not: AWS S3 is used by many users to store their images, and we hardly see issues with that solution nowadays.

Be reminded that while bandwidth is dirt cheap nowadays, storage is never free, and can become quite expensive. I'm amazed by how much free solutions there are for hosting images today, but I fully understand that the companies and/or people behind such services have some limits in place.

[abubelinha]

I'll try the gDrive php-proxy idea first to see if it works (it's the easiest way for me).
Thanks for all the alternatives suggested @andrieslouw