GitHub: Random 429 Client Error: too many requests

[stanislav-brabec]

I am experiencing this error e-mail randomly, often several similar mails at once:

Broken repository browser URL

This component is set up incorrectly.

The repository browser URL points to non-existing locations:

https://github.com/bbidulock/icewm/blob/icewm-1-4-BRANCH/src/aapm.cc#L154 (429 Client Error: too many requests for url: https://github.com/bbidulock/icewm/blob/icewm-1-4-BRANCH/src/aapm.cc#L154)

Looking at the log, it happened i the time, when several robots were crawling Weblate in parallel and several users were browsing, and one user did complicated searches.

It seems that GitHub introduces on decreased the limit. Weblate should reflect it and decrease frequency of source browser URL checks.

Looking into the localmessages, I see:

2021-09-21T17:13:23.338344+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 404 Client Error: Not Found for url: https://github.com/libyui/libyui-qt/blob/master/src/QY2DiskUsageList.cc#L189
2021-09-21T17:13:40.999848+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni-docs/blob/manager-4.2/modules/upgrade/nav-upgrade-guide.adoc#L1
2021-09-21T17:13:41.329890+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni/blob/master-weblate/susemanager/yast/susemanager_ask.rb#L38
2021-09-21T17:13:41.640259+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni-docs/blob/manager-4.1/modules/installation/nav-installation-guide.adoc#L1
2021-09-21T17:13:41.900937+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/openSUSE/scout/blob/master/scout/autoconf.py#L14

The first error is correct, the rest is unexpected.

To Reproduce the issue

It is not easy to reproduce. It appears randomly, once per several days or weeks.

Server configuration and status

Weblate installation: openSUSE Leap 15.3 + dedicated repositories.

weblate list_versions

* Weblate: 4.7.1
* Django: 3.2.5
* siphashc: 1.3
* translate-toolkit: 3.3.6
* lxml: 4.6.3
* Pillow: 8.3.1
* bleach: 3.1.5
* python-dateutil: 2.8.1
* social-auth-core: 4.1.0
* social-auth-app-django: 4.0.0
* django-crispy-forms: 1.9.0
* oauthlib: 2.0.6
* django-compressor: 2.4
* djangorestframework: 3.11.2
* django-filter: 2.4.0
* django-appconf: 1.0.4
* user-agents: 2.1
* filelock: 3.0.12
* setuptools: 40.5.0
* jellyfish: 0.7.2
* openpyxl: 3.0.3
* celery: 5.0.5
* kombu: 5.1.0
* translation-finder: 2.9
* weblate-language-data: 2021.5
* html2text: 2019.8.11
* pycairo: 1.20.0
* pygobject: 3.34.0
* diff-match-patch: 20181111
* requests: 2.25.1
* django-redis: 4.11.0
* hiredis: 1.0.1
* sentry_sdk: 0.14.4
* Cython: 0.29.14
* misaka: 2.1.1
* GitPython: 3.0.5
* borgbackup: 1.1.15
* pyparsing: 2.4.7
* pyahocorasick: 1.4.1
* python-redis-lock: 3.6.0
* Python: 3.6.13
* Git: 2.31.1
* psycopg2: 2.8.5
* chardet: 3.0.4
* ruamel.yaml: 0.17.4
* Redis server: 6.0.14
* PostgreSQL server: 12.8
* Database backends: django.db.backends.postgresql
* Cache backends: default:RedisCache, avatar:FileBasedCache
* Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
* OS encoding: filesystem=utf-8, default=utf-8
* Celery: redis://localhost:6379, redis://localhost:6379, regular
* Platform: Linux 5.3.18-59.19-default (x86_64)

Weblate deploy checks

weblate check --deploy

WARNINGS:
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
 (1)
	HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg': 'hg'
	HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: git: 'svn' is not a git command. See 'git --help'.

The most similar commands are
	fsck
	mv
	show
 (1)
	HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/backup.html
?: (weblate.I031) New Weblate version is available, please upgrade to 4.8.1.
	HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/upgrade.html

System check identified 14 issues (1 silenced).

Additional context

I am not sure, whether GitHub counts only direct accesses, or also accesses with Referrer.

Analyzing the web crawler logs, I suggest to enhance robots.txt with:

/api/
/checks/
/engage/
/languages/*/*/search/

and maybe

/widgets/

[stanislav-brabec]

And a minor suggestion: remote HTTP responses like 429, 500, 503, 504 and similar possibly temporary replies should not trigger immediate mails to all project/component subscribers. It makes sense to report it after several failed attempts.

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.