GitHub Actions workflow fails (HTTP 404) on downloading ek.json - Add A Retry?

[eUgEntOptIc44]

Hi @geraldb

as the title of this issue reads. The update GitHub Actions workflow fails on ek.json

==> [55/260] ek Equatorial Guinea / Africa -- Countries:
  sleep 0.5 sec(s)...
GET https://www.cia.gov/the-world-factbook/geos/ek.json...
Error: Process completed with exit code 1.

The url https://www.cia.gov/the-world-factbook/geos/ek.json throws the error code 404.

I'll add further insights on this in the comments below.

Regards, Jean-Luc

[eUgEntOptIc44]

The html page for the country at https://www.cia.gov/the-world-factbook/countries/equatorial-guinea/ as well as the json content of Germany are still accessible. It might be just a human error.

However integrating execption handling per url in the code of the factbook script might be a good idea. As at the time of writing (to my personal understanding) the occurrence of an error in the parsing of one json url aborts the whole script.

[eUgEntOptIc44]

I'd be very grateful if @geraldb could look into this any time soon. Please do keep up the great work of keeping this project up and running!

[eUgEntOptIc44]

Sorry looks like the situation changed faster than expected. I just managed to successfully call the previously faulty url. Maybe somebody at the CIA has starred the repo. You never know... 😉

But I'd like to highlight that (more) exception handling would be nice to have.

[eUgEntOptIc44]

I'd expect an exception handling in the loop of download.rb to prevent it from crashing when a single download fails

[geraldb]

Thanks for reporting the error. I was offline for some days. I rerun the workflow / actions and luckily got all completed. I keep the issue open for now as a reminder to (maybe) add a retry or such to the download in the future. Cheers. Prost.

[eUgEntOptIc44]

Prost @geraldb 🍻
Thumbs up 👍 for the fast reaction!
Take care, Regards Jean-Luc

[eUgEntOptIc44]

Hi @geraldb

sorry for bringing this up another time but ... it happened again. Maybe their DDOS protection kicks in at some point. Just a random idea. To be honest I got no idea. I'd be so happy if you had the time to dig into this issue.

[geraldb]

Thanks for the notice / reminder. I kicked-off a manual (re)run and luckily this time the run / dataset update completed successfully. For sure - next time it might be time for trying to add a retry or such in the download script.

[eUgEntOptIc44]

as noted above I tried my best to integrate exception handling myself to reduce the workload on your side.

I managed to get download.rb successfully running (after modification). However double checking for typos and flaws might be a good idea. My knowledge of ruby is close to non existing...