unable to download sunlight map

[oxygala]

Hi,
When I try to fetch the world sunlight map from die.net, Variety gives the following error on command line:
503 Server Error: Service Temporarily Unavailable for url: https://static.die.net/earth/mercator/1600.jpg

The URL is accessible on a browser, albeit after a DDOS check. I wonder if this is fixable on Variety (eg. by giving a few seconds before timeout)
Maybe irrelevant, but it is a Manjaro/Arch system with Gnome 3.30.

Thanks

[telsub]

I have the same issue on Fedora 29 / Gnome 3.30.2 / Python 3.7 / variety 0,7.1.

Browser gets the https://static.die.net/earth/mercator/1600.jpg URL after DDOS check but variety reports this trying to change the wallpaper:

DEBUG: 2019-01-12 08:21:10,187: profiler() '[Thread-4] -> Entering function: download_one (line 37 in /usr/lib/python3.7/site-packages/variety/EarthDownloader.py) - not tracing further because MAX_NONTARGET_DEPTH=1'
INFO: 2019-01-12 08:21:10,187: download_one() 'Downloading world sunlight map from https://www.die.net/earth/'
INFO: 2019-01-12 08:21:10,188: save_locally() 'Origin URL: https://www.die.net/earth/'
INFO: 2019-01-12 08:21:10,190: save_locally() 'Image URL: https://static.die.net/earth/mercator/1600.jpg'
INFO: 2019-01-12 08:21:10,191: save_locally() 'Local name: /home/anv/.config/variety/Downloaded/Earth/1600.jpg'
INFO: 2019-01-12 08:21:10,645: save_locally() 'Download failed from image URL: https://static.die.net/earth/mercator/1600.jpg (source location: https://www.die.net/earth/) '
DEBUG: 2019-01-12 08:21:10,646: profiler() '[Thread-4] -> Leaving function: download_one_from (line 1018 in /usr/lib/python3.7/site-packages/variety/VarietyWindow.py)'
DEBUG: 2019-01-12 08:21:10,646: profiler() '[Thread-4] -> Entering function: exception (line 1364 in /usr/lib64/python3.7/logging/init.py) - not tracing further because MAX_NONTARGET_DEPTH=1'
DEBUG: 2019-01-12 08:21:10,646: profiler() '[Thread-4] -> Entering function: (line 994 in /usr/lib/python3.7/site-packages/variety/VarietyWindow.py)'
DEBUG: 2019-01-12 08:21:10,646: profiler() '[Thread-4] -> Leaving function: (line 994 in /usr/lib/python3.7/site-packages/variety/VarietyWindow.py)'
DEBUG: 2019-01-12 08:21:10,647: profiler() '[Thread-4] -> Entering function: makeRecord (line 1431 in /usr/lib64/python3.7/logging/init.py) - not tracing further because MAX_NONTARGET_DEPTH=1'
ERROR: 2019-01-12 08:21:10,647: download_thread() 'Could not download wallpaper:'
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/variety/VarietyWindow.py", line 986, in download_thread
self.download_one_from(downloader)
File "/usr/lib/python3.7/site-packages/variety/VarietyWindow.py", line 1018, in download_one_from
file = downloader.download_one()
File "/usr/lib/python3.7/site-packages/variety/EarthDownloader.py", line 39, in download_one
downloaded = self.save_locally(self.location, EARTH_IMAGE_URL, force_download=True, extra_metadata={'headline': 'World Sunlight Map'})
File "/usr/lib/python3.7/site-packages/variety/Downloader.py", line 117, in save_locally
raise e
File "/usr/lib/python3.7/site-packages/variety/Downloader.py", line 112, in save_locally
r = Util.request(image_url, stream=True)
File "/usr/lib/python3.7/site-packages/variety/Util.py", line 569, in request
r.raise_for_status()
File "/usr/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Temporarily Unavailable for url: https://static.die.net/earth/mercator/1600.jpg

[telsub]

Workaround...
in /usr/lib/python3.7/site-packages/variety/EarthDownloader.py
i have replaced the line with the picture:

#EARTH_IMAGE_URL = "https://static.die.net/earth/mercator/1600.jpg"
EARTH_IMAGE_URL = "http://www.opentopia.com/images/data/sunlight/world_sunlight_map_rectangular.jpg"

it's not gentlemen like, but works.

[jlu5]

Looking at this report I believe the problem is that die.net sometimes enables Cloudflare DDoS protection, which essentially blocks all automated clients from downloading from it. Unless we switch to a different backend entirely, I'm not sure there's much we can do about this.

[peterlevi]

I used to use Opentopia before, then I switched to Die.net as being the actual source.
I have moved back to Opentopia as part of the changes in this PR: e958e46

However I am contemplating removing this feature entirely - apart from the issue with Cloudflare (or the alternative of using a "referring" site like Opentopia), it also adds quite a lot of complexity via special cases in the code just to support the non-standard refreshing, does not fit great into the UI either, and generally is quite much different from what Variety generally does.

The only thing that stops me is that this does seem to be quite a popular feature, despite its clumsiness.

Alternatively, we may consider just providing a generic configurable "auto-refreshing" downloader plugin where user specifies an image URL. Thus there are other options apart from the world sunlight map by die.net - e.g. the images from http://lunaf.com/space/daylight/. This would be better for power users, but is pretty bad for feature discovery - most users will not even think to add such a source, unless the UI points them to it (which would also mean we need some form of non-trivial dedicated UI around this).
Also, the current logic does a crop on the image so that the labels at the bottom of the image from Opentopia are cut out, which would not be applicable for a generic refreshing plugin.

[sebalis]

I can download the URL using wget and curl, at a time when I see CloudFlare protection active in the browser. In the browser, the original response is an HTML document sent with status 503. It contains JavaScript that when executed will cause another request that eventually succeeds. WIth a different user agent such as 'wget', the protection is not active. So just change the user agent and it will work again. I’ve done this (in a simplistic way) and would be happy if this could be used.

[jlu5]

Fixed by #163 in Git master