Service unavailable errors (cache too!)

[vsoch]

hey @Danger89 ! It looks like we are hitting some previous errors with the server, you can see here: https://github.com/rocky-linux/documentation/actions/runs/3713744520/jobs/6296764864

WARNING:fake_useragent:Error occurred during loading data. Trying to use cache server file https://useragent.melroy.org/cache.json
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/fake_useragent/utils.py", line 64, in get
    urlopen(
  File "/opt/conda/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/opt/conda/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/fake_useragent/utils.py", line 139, in load
    browsers_dict[browser_name] = get_browser_user_agents(
  File "/opt/conda/lib/python3.9/site-packages/fake_useragent/utils.py", line 100, in get_browser_user_agents
    html = get(
  File "/opt/conda/lib/python3.9/site-packages/fake_useragent/utils.py", line 87, in get
    raise FakeUserAgentError("Maximum amount of retries reached")
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached
WARNING:fake_useragent:Error occurred during loading data. Trying to use cache server file https://useragent.melroy.org/cache.json
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/fake_useragent/utils.py", line 64, in get
    urlopen(
  File "/opt/conda/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/opt/conda/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/opt/conda/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

and reported here to link the issues. urlstechie/urlchecker-action#99

Is this ephemeral (should they try again?) and I wanted to ask - should we explore ways to provide the data that do not need a server? E.g., I'm an ORAS Python developer, and I use oras to store random blobs of data in GitHub packages. I could help with this idea if you are interested!

[vsoch]

And there is a very kind offer from a Rocky developer:

[melroy89]

So Yea, you were hitting my server too hard. I needed to change the code again to avoid my server from being ddos (mostly not on purpose but still).

Please update the pip package once again, it should now have the json file locally installed. Meaning the user agent strings are now part of the pypi package. Update using:

pip install --upgrade fake-useragent

[vsoch]

Thanks! I'll pass this on. So question - why do we need a server for the data? Doesn't that make it harder to manage, long term?

[melroy89]

We don't use a server for data anymore. Again it's stored locally within the pypi package.

[vsoch]

Woot! That's great :)