scripts/get-external-data.py provides little user feedback

[SomeoneElseOSM]

Expected behavior

Some idea that it is doing something

Actual behavior

Some idea that it is doing something

Links and screenshots illustrating the problem

renderaccount@ubuntuvm66:~/src/openstreetmap-carto$ time scripts/get-external-data.py
INFO:root:Checking table simplified_water_polygons
INFO:root:Checking table water_polygons
(that's been sat like that for about 10 minutes)

At least optionally "get-external-data.py" should provide feedback on what it is doing and how far it has got through the process.

[kocio-pl]

True. I have been struck myself with this lately, since I was not even aware that something went bad with this import. It should at least indicate that (a) it's processing data and (b) if the result is OK or was there some error.

[SomeoneElseOSM]

or was there some error.

That last bit's important because Natural Earth have been suggesting that they'll move their CDN, Currently https://github.com/gravitystorm/openstreetmap-carto/blob/master/external-data.yml points at https://naciscdn.org and the first thing we'll know of any change there is when this script fails.

[StyXman]

It's going to be quite complex. You have to combine requests with something else to get asynchronicity (!!!) and calculate and display progress based on that. More info here.

[kocio-pl]

Adding informations like "now downloading...", "now applying data into database..." and "data added correctly!/there was an error while trying to add data!" should not be that hard.

[SomeoneElseOSM]

For info this has just raised it's head again because Natural Earth data is unavailable again (see nvkelso/natural-earth-vector#581 et al).

[jgruca]

I've added some extra logging in #4472 to address this. Feedback welcome.

[SomeoneElseOSM]

For info, even with the extra logging, this script still managed to defeat a potential user yesterday (according to a conversation on IRC). The old version (see e.g. in an old fork here was much better.

[StyXman]

Yeah, I tried to see how to implement any kind of progress feedback and it would be hard because the way the data is downloaded in a .... hmmm... in a a single gulp, and that thing (the whole zip in memory) gets passed around several functions until it's written to disk. I just noticed we could download chunks and accumulate them in the same download function. I'll give it another try... sometime.