Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Got 'too many 502 error responses' again #158

Closed
yarikoptic opened this issue Dec 4, 2022 · 3 comments
Closed

Got 'too many 502 error responses' again #158

yarikoptic opened this issue Dec 4, 2022 · 3 comments

Comments

@yarikoptic
Copy link
Member

Previously I found only one mentioning of it while trying #151 (comment) to overcome problem which was presumably fixed on github, and thus we reverted that PR in #157 . And running that version we got a fresh fail

Date: Fri, 02 Dec 2022 05:00:03 -0500
...
Subject: Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-file-windows.lock /mnt/datasets/datalad/ci/file-windows/tools/cron_job

{
...
  "rate": {
    "limit": 5000,
    "used": 125,
    "remaining": 4875,
    "reset": 1669975836
  }
}
2022-12-02T04:45:02-0500 [INFO    ] tinuous tinuous 0.6.0+2.ga073f29
2022-12-02T04:45:02-0500 [INFO    ] tinuous Fetching resources from github
2022-12-02T04:45:02-0500 [INFO    ] tinuous Fetching runs newer than 2022-11-02 09:45:02.675964+00:00
2022-12-02T04:45:03-0500 [INFO    ] tinuous Fetching runs for workflow .github/workflows/build.yml (Build)
Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  [Previous line repeated 9 more times]
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 836, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/datalad/file-windows/actions/workflows/3075786/runs (Caused by ResponseError('too many 502 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module>
    sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')())
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
...
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 507, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/datalad/file-windows/actions/workflows/3075786/runs (Caused by ResponseError('too many 502 error responses'))

@jwodder -- any idea on what/why and what could be done?
@yarikoptic
Copy link
Member Author

oh, apparently it was not a singular case -- we got similar fails across many runs, but they are all from Dec 02, so nothing from 03 or current 04 -- so may be the issue was self-healed...

full list of recent emails matching "too many 502..." 
   1 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job
   2 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-deprecated.lock /mnt/datasets/datalad/ci/datalad-deprecated/tools/cron_job
   3 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
   4 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job
   5 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-metalad.lock /mnt/datasets/datalad/ci/datalad-metalad/tools/cron_job
   6 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandischema && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandischema.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
   7 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-catalog.lock /mnt/datasets/datalad/ci/datalad-catalog/tools/cron_job
   8 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
   9 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-heudiconv.lock /mnt/datasets/datalad/ci/heudiconv/tools/cron_job
  10 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-cli && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-cli.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  11 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex.lock /mnt/datasets/datalad/ci/git-annex/tools/cron_job
  12 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-file-windows.lock /mnt/datasets/datalad/ci/file-windows/tools/cron_job
  13 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job
  14 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-deprecated.lock /mnt/datasets/datalad/ci/datalad-deprecated/tools/cron_job
  15 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  16 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-metalad.lock /mnt/datasets/datalad/ci/datalad-metalad/tools/cron_job
  17 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job
  18 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandischema && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandischema.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  19 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad.lock /mnt/datasets/datalad/ci/logs/tools/cron_job
  20 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-catalog.lock /mnt/datasets/datalad/ci/datalad-catalog/tools/cron_job
  21 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  22 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-bids-specification.lock /mnt/datasets/datalad/ci/bids-specification/tools/cron_job
  23 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex.lock /mnt/datasets/datalad/ci/git-annex/tools/cron_job
  24 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-file-windows.lock /mnt/datasets/datalad/ci/file-windows/tools/cron_job
  25 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job
  26 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  27 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-deprecated.lock /mnt/datasets/datalad/ci/datalad-deprecated/tools/cron_job
  28 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-metalad.lock /mnt/datasets/datalad/ci/datalad-metalad/tools/cron_job
  29 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job
  30 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandischema && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandischema.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch
  31 O   Dec 02 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-catalog.lock /mnt/datasets/datalad/ci/datalad-catalog/tools/cron_job
  32 O   Dec 02 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch

@jwodder
Copy link
Member

jwodder commented Dec 5, 2022

@yarikoptic I can access the URL given in the error message just fine, so this seems to have been a transitory error on GitHub's end.

@yarikoptic
Copy link
Member Author

didn't see those for a while, let's consider indeed transient

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yarikoptic @jwodder