Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn on & skip workflow runs for certain "broken" GitHub workflows #151

Merged
merged 2 commits into from
Oct 11, 2022

Conversation

jwodder
Copy link
Member

@jwodder jwodder commented Oct 10, 2022

Closes #150.

@jwodder jwodder added the patch Increment the patch version when merged label Oct 10, 2022
@jwodder jwodder changed the title Warn on & skip workflow runs for certain "broken" workflows Warn on & skip workflow runs for certain "broken" GitHub workflows Oct 10, 2022
@yarikoptic
Copy link
Member

looks good. updated in tinuous-dev conda env of datalad@smaug which is used by cron jobs and testing on datalad-extensions. Since needs to wait for all those retries -- I will just check it later.

@yarikoptic
Copy link
Member

ha -- it crashed with 502 not 500

2022-10-10T22:23:11-0400 [INFO    ] tinuous Fetching runs for workflow .github/workflows/test_extensions.yml (Extensions)
Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 846, in urlopen
    return self.urlopen(
  [Previous line repeated 9 more times]
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 836, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/datalad/datalad-extensions/actions/workflows/605374/runs (Caused by ResponseError('too many 502 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module>
    sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')())
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/__main__.py", line 112, in fetch
    for obj in ci.get_build_assets(
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 147, in get_build_assets
    for run in self.get_runs(wf, self.since):
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 44, in wrapped
    return func(gha, *args, **kwargs)
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 103, in get_runs
    for r in wf.get_runs():
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/PaginatedList.py", line 56, in __iter__
    newElements = self._grow()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/PaginatedList.py", line 67, in _grow
    newElements = self._fetchNextPage()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/PaginatedList.py", line 199, in _fetchNextPage
    headers, data = self.__requester.requestJsonAndCheck(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/Requester.py", line 354, in requestJsonAndCheck
    *self.requestJson(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/Requester.py", line 454, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/Requester.py", line 528, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/Requester.py", line 555, in __requestRaw
    response = cnx.getresponse()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/github/Requester.py", line 127, in getresponse
    r = verb(
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 507, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/datalad/datalad-extensions/actions/workflows/605374/runs (Caused by ResponseError('too many 502 error responses'))

I have checked prior emails -- they were about 500s. I guess we should adjust condition so it could be 500 or 502. I will try with that.

Comment on lines 112 to 113
and reason.args[0]
== ResponseError.SPECIFIC_ERROR.format(status_code=500)
Copy link
Member

@yarikoptic yarikoptic Oct 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try now

Suggested change
and reason.args[0]
== ResponseError.SPECIFIC_ERROR.format(status_code=500)
and (reason.args[0] in
[ResponseError.SPECIFIC_ERROR.format(status_code=500),
ResponseError.SPECIFIC_ERROR.format(status_code=502)])

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirming that smth like that (I reentered manually) worked but needs adjustment of the log message below to log actual status_code - not hard-coded 500 -- please adjust accordingly and let's get it merged/released (adding a label).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yarikoptic Should the code just catch 500 and 502 here, or also 503 and 504 (the same as the status codes that are already retried)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally up to you - feel free to add those 5xx into the mix here as well .

  • Originally I thought about it and because we are dealing with specific workaround, thought we better keep to 500 and 502 until we run into the case where any other 5xx shows up for this particular scenario since we do not want to skip any other similar case I guess.
  • Since we do test for the since and presence of workflow file I think it would be ok if we skip on any 5xx.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated: 74066c7

@yarikoptic yarikoptic added the release Create a release when this pr is merged label Oct 11, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #151 (74066c7) into master (655bb1c) will increase coverage by 0.72%.
The diff coverage is 47.82%.

@@            Coverage Diff             @@
##           master     #151      +/-   ##
==========================================
+ Coverage   69.15%   69.88%   +0.72%     
==========================================
  Files          10       10              
  Lines         976      993      +17     
  Branches      183      152      -31     
==========================================
+ Hits          675      694      +19     
+ Misses        244      242       -2     
  Partials       57       57              
Impacted Files Coverage Δ
src/tinuous/github.py 51.06% <47.82%> (-0.32%) ⬇️
src/tinuous/travis.py 60.36% <0.00%> (+4.50%) ⬆️
src/tinuous/appveyor.py 75.29% <0.00%> (+7.05%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@@ -34,6 +35,12 @@
sanitize_pathname,
)

RETRY_STATUSES = [500, 502, 503, 504]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eh, I was trying to avoid "equating" the RETRY_STATUSES to all the statues we might want to skip on (only 5xxs). ATM they are indeed identically but in general not necessarily the same, so IMHO it mixes the semantics abit ... but since functionally is ok ATM, let's proceed. Thank you @jwodder ! Merging

@yarikoptic yarikoptic merged commit ac3116c into master Oct 11, 2022
@yarikoptic yarikoptic deleted the gh-150 branch October 11, 2022 20:36
jwodder added a commit that referenced this pull request Nov 2, 2022
This reverts commit ac3116c, reversing
changes made to 655bb1c.
yarikoptic added a commit that referenced this pull request Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch Increment the patch version when merged release Create a release when this pr is merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

run started to error out
3 participants