Ensure GitHub status checks have passed #289

sonicdoe · 2018-07-26T13:17:58Z

Ensures GitHub status checks have passed using the Statuses API. Resolves #53.

This is a really quick implementation to gather feedback early. Some things that are missing or need to be discussed:

sindresorhus · 2018-08-20T04:41:51Z

Thanks for starting this 🙌

sindresorhus · 2018-08-20T04:42:21Z

I agree with your list of missing things.

Link to the failed status check

Use terminal-link.

sindresorhus · 2018-08-20T04:43:51Z

Fail if commit hasn’t been pushed to GitHub

I often do some local commits. Usually some nitpicks before doing a release. And I then run np and let np push the commits to GitHub. Would be nice if this was handled and it would just wait for the status check of these new commits.

sindresorhus · 2018-08-20T04:44:23Z

Reorganize task (perhaps this should be a top-level task?)

👍

lib/prerequisite.js

sonicdoe · 2018-10-27T09:45:40Z

It’s getting somewhere 🙌

The checklist is now done but I still need to thoroughly test every case. Nevertheless, I think this is ready for another code review. I hope the commits along with my intentions are easy to follow. If there’s anything unclear, let me know.

Edit: I’ve now manually tested quite a few cases, such as commits with pending checks, commits with multiple failed checks, rate limit exceptions, and timeouts.

- If there’s only one check, GitHub shows “Failure: ” and the check’s description - If there are multiple checks and only some failed, GitHub shows “Some checks were not successful” and a list of all checks - If there are multiple checks and all failed, GitHub shows “All checks have failed” and a list of all checks

cli.js

lib/gh-checks.js

index.js

sindresorhus · 2018-12-15T01:16:33Z

index.js

+			title: 'Ensure GitHub checks have passed',
+			skip: () => {
+				if (!pkg.repository) {
+					return 'No repository specified';


No repo specified in package.json

?

What do you think about “No repository specified in package.json”? The package.json field is also called repository so I’m not sure whether we should abbreviate here.

“No repository specified in package.json”? The package.json field is also called repository so I’m not sure whether we should abbreviate here.

👍

Done: c12e272.

Should np fallback to getting the repo URL from git itself? I feel like requiring pkg.repository is just needless busy work.

I’ve had the same thought at one point. I don’t have particularly strong opinions in either direction.

On one hand, requiring pkg.repository encourages best practices (for example, npmjs.com will display a link to the repository if the field is specified) and gives us higher assurance that it actually specifies the package’s repository (whereas the Git remote might be a fork). On the other hand, this task already kind of expects the Git remote to be the upstream repository (since we git push before querying the GitHub API).

That's an interesting thought. If I'm on a fork and running np, I'm probably intentionally trying to publish some hacky thing I did, perhaps to a scoped package under my own personal namespace. I'm more likely to have a PR open on the pkg.repository (which I probably haven't updated and where CI checks are probably running) than to have CI setup on my fork. I don't think I have ever set up CI on a fork, though major projects obviously would. 🤔

That being said, I still think if pkg.repository isn't defined at all, it might be worth having some kind of fallback. Just "thinking out loud" here.

source/github-checks.js

itaisteinherz · 2020-03-04T22:29:08Z

source/github-checks.js

+		// It might take a bit for the status checks to be created.
+		// Even if, it probably takes at least a few seconds for them to pass.
+		task.title = `${title} ${chalk.yellow('(waiting for checks to start…)')}`;
+		await delay(THIRTY_SECONDS);


Wouldn't 10-15s suffice in this case?
Also, is this really needed? Won't pWaitFor do this for us anyways?

Wouldn't 10-15s suffice in this case?

Probably. I just sticked to the existing delay to keep it simple. If there are actually any status checks, I’d expect them to take more than 30 seconds to complete for most projects anyway.

Also, is this really needed? Won't pWaitFor do this for us anyways?

If we immediately call GitHub’s status API, total_count might still be zero and we’d skip the task with “No status checks found”. On the flip side, we can’t retry in that case because commits without any status checks would be caught in a loop then (since the state will always be pending).

Wouldn't 10-15s suffice in this case?

I agree. 30 seconds is too long.

It might take a bit for the status checks to be created.

This sounds like a GitHub bug. Why wouldn't they be created right away?

I agree. 30 seconds is too long.

Changed to 10 seconds in c704b6c.

This sounds like a GitHub bug. Why wouldn't they be created right away?

As far as I understand, GitHub can’t know whether a commit will result in status checks being created. For example, GitHub only notifies Travis CI of Git pushes and it’s then Travis CI’s call whether to create any status checks (which it might not need to, depending on your settings).

Good point. What about GitHub Actions? I assume GitHub would know then?

GitHub could but I still think there would be a slight delay. According to GitHub Help, GitHub Actions uses the Checks API to output statuses. Therefore, I think the Statuses API would still return zero status checks in the time frame between the commit having been pushed and the workflow starting to run.

source/util.js

sindresorhus · 2020-06-06T07:25:53Z

@sonicdoe Is this done? If so, I think it would be beneficial to merge it and get it out there so people can try it out and report any issues.

sonicdoe · 2020-06-07T17:55:56Z

@sindresorhus Sounds good, I’m fine with giving it a go 🙂

sindresorhus · 2020-06-29T07:40:32Z

I just tried this on https://github.com/sindresorhus/is-interactive after pushing a commit, but it skipped the task:

  ↓ Ensuring GitHub status checks have passed (waiting for checks to start…) [skipped]
    → No status checks found

However, there clearly is a status check:

sonicdoe · 2020-06-29T18:14:39Z

It looks like it took over ten seconds for the status check to be created. I don’t think GitHub exposes the push time so it’s difficult to say for sure. I can only see that the commit was committed at 07:37:04 and the Travis CI build started at 07:39:06.

If that’s the case, I’m not sure how to best resolve this. We don’t know when a check will be created and we can’t know in advance whether any checks will be created at all 😕 If it’s any improvement, we could check whether the latest commit (before pushing new ones) has checks and use that as an indicator.

sholladay · 2020-06-30T03:00:55Z

If I understand the code correctly, it currently delays 10 seconds, then tries to check the status, and if totalCount happens to be zero at that time, it gives up and skips, without retrying. That seems pretty fragile.

Here's how I would approach this:

Delay ten seconds
Predict whether a status check is likely to eventually exist, by:
- Fetching status checks of recently pushed commits on the remote prior to those pushed by np (as @sonicdoe mentioned), but also...
- Fetching the branch protection rules using one of these APIs:
Check the commit status and if totalCount === 0, then:
- If step 2 found previous status checks / branch protection contexts, retry every ten seconds, up to a limit of five minutes
- If step 2 errored or found no status checks / branch protection contexts, retry every ten seconds, up to a limit of one minute

sonicdoe · 2020-06-30T17:45:00Z

Interestingly, Get status checks protection is only available to authenticated users (see this topic). Fortunately, Get a branch provides enough information in the protection key. For example, /repos/sholladay/pogo/branches/master lists two status checks: “Travis CI - Branch” and “Travis CI - Pull Request”.

In any case, I’m not sure how widespread protected branches are. For example, neither lodash/lodash, nor facebook/react, nor chalk/chalk list required status checks. Most importantly, sindresorhus/is-interactive doesn’t have any.

sholladay · 2020-06-30T18:00:49Z

That's why retries are so important. It doesn't currently retry when totalCount is 0, but it should. My suggestion is to retry up to one minute when there are no branch protection rules.

I do think that more people would enable branch protection if np gave them a good reason to do so.

sonicdoe · 2020-07-01T18:49:15Z

We’ve actually had a delay of 30 seconds before (see this conversation). I guess it will always be a balance between not missing any status checks and annoying people who don’t use them at all.

I do think that more people would enable branch protection if np gave them a good reason to do so.

How would retrying up to one minute when there are no branch protection rules motivate people to enable them? People who don’t use status checks couldn’t enable protection and therefore would always have to wait a whole minute. On the flip side, people who do use status checks wouldn’t really gain much from enabling protection because they’d have to wait for their status checks to appear in both cases. Granted, the second group would have the minor benefit of np failing in case their status checks don’t appear after a minute.

sholladay · 2020-07-02T02:19:27Z

How would retrying up to one minute when there are no branch protection rules motivate people to enable them?

It's about reliability. Take Sindre, for example. As you mentioned, he doesn't currently use branch protection. With the algorithm I proposed, it'll still work for his repos as-is, but if there's an unusually high delay, as happens occasionally (on bad days I've seen it take a couple of minutes to create the status), then np would more eagerly skip this step compared to if he enabled branch protection. The docs can mention this as a means to improve reliability.

Additionally, I suspect that a lot of people may just not be aware of the branch protection feature and its ability to guarantee a status check has passed. It's not exactly front and center in the GitHub UI. There are good reasons to enable it that have nothing to do with np, so mentioning it in the docs would be a good thing for the community.

People who don’t use status checks couldn’t enable protection and therefore would always have to wait a whole minute.

No, not always. I'm not proposing a one minute delay. It's a timeout. It's only a whole minute in the worst case.

sonicdoe · 2020-07-03T15:51:19Z

Thanks for elaborating. I’d even go so far as to fail if np can’t find any status checks and branch protection is enabled. In some sense, this matches the behavior on GitHub where pull requests can’t be merged if status checks haven’t reported any status.

No, not always. I'm not proposing a one minute delay. It's a timeout. It's only a whole minute in the worst case.

I don’t think I follow. If you don’t use status checks at all, in which case would your proposed algorithm skip sooner than one minute?

sindresorhus · 2021-03-07T11:09:29Z

Is anyone interested in pushing this forward? This is not something I personally need, so I don't really have a lot of energy to make decisions and push this forward.

sindresorhus · 2021-06-03T19:30:21Z

I'm unfortunately going to close this as it doesn't seem to go anywhere: #53 (comment)

sonicdoe · 2021-06-04T15:52:22Z

No worries, thanks for everything so far! The fact that status checks are created asynchronously, which I think is the biggest unresolved issue, is certainly unfortunate.

sindresorhus reviewed Aug 20, 2018

View reviewed changes

lib/prerequisite.js Outdated Show resolved Hide resolved

sindresorhus reviewed Aug 20, 2018

View reviewed changes

lib/prerequisite.js Outdated Show resolved Hide resolved

sonicdoe force-pushed the github-status-check branch from 870da36 to 4f938f5 Compare September 18, 2018 19:33

sonicdoe force-pushed the github-status-check branch from 4f938f5 to 4650f5a Compare October 24, 2018 19:57

sonicdoe changed the title ~~[WIP] Check GitHub status checks~~ [WIP] Ensure GitHub status checks have passed Oct 24, 2018

sonicdoe force-pushed the github-status-check branch 2 times, most recently from 4a29c15 to d5c0f80 Compare October 24, 2018 20:29

sonicdoe added 2 commits October 27, 2018 10:46

Ensure GitHub checks have passed

7518b1b

Wait if checks are pending

3480906

sonicdoe force-pushed the github-status-check branch 2 times, most recently from ef74c8d to bc8d011 Compare October 27, 2018 09:33

sonicdoe added 10 commits October 29, 2018 20:29

Skip if the repository is not hosted on GitHub

ac217e5

Skip if no repository is specified

76f01e9

Link to check’s target URL

aa90c92

Time out after 30 minutes

393020a

Change title while waiting for pending checks

5d3f571

Improve timeout error message

84a88c6

Skip checks with --no-checks

643b79f

Push unpushed commits first

dbd4588

Improve rate limit error message

f6e3290

sonicdoe force-pushed the github-status-check branch from bc8d011 to f6e3290 Compare October 29, 2018 20:18

sindresorhus added 2 commits December 15, 2018 02:05

Update index.js

a234662

Update gh-checks.js

9836dc6

sindresorhus requested changes Dec 15, 2018

View reviewed changes

Remove blank lines

e3ecee0

itaisteinherz reviewed Mar 4, 2020

View reviewed changes

source/github-checks.js Outdated Show resolved Hide resolved

itaisteinherz reviewed Mar 4, 2020

View reviewed changes

sonicdoe added 5 commits April 4, 2020 12:12

Use listr’s skip() to skip task

0b54e4e

Merge branch 'master' into github-status-check

dfc7950

Refactor error handling

945c64c

Add statusChecks flag to readme

8d6ec89

Merge branch 'master' into github-status-check

3ff1db0

sindresorhus reviewed May 24, 2020

View reviewed changes

source/util.js Outdated Show resolved Hide resolved

sonicdoe added 4 commits May 26, 2020 19:58

Rename githubChecks to verifyCommitStatusChecks

4cfcb43

Merge branch 'master' into github-status-check

f0bb068

Use one-line arrow function for hostedGitInfo()

1c255d8

Shorten wait time for checks to start

c704b6c

Merge branch 'master' into github-status-check

819bbb3

Base automatically changed from master to main January 23, 2021 17:55

sindresorhus mentioned this pull request Jun 3, 2021

Ensure CI has passed #53

Open

sindresorhus closed this Jun 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure GitHub status checks have passed #289

Ensure GitHub status checks have passed #289

sonicdoe commented Jul 26, 2018 •

edited

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sonicdoe commented Oct 27, 2018 •

edited

sindresorhus Dec 15, 2018

sonicdoe Dec 30, 2018

sindresorhus Dec 31, 2018

sonicdoe Jan 2, 2019

sholladay Jan 3, 2019

sonicdoe Jan 4, 2019

sholladay Jan 4, 2019

itaisteinherz Mar 4, 2020

sonicdoe Mar 5, 2020

sindresorhus May 24, 2020

sonicdoe May 26, 2020

sindresorhus Jun 6, 2020

sonicdoe Jun 7, 2020

sindresorhus commented Jun 6, 2020

sonicdoe commented Jun 7, 2020

sindresorhus commented Jun 29, 2020

sonicdoe commented Jun 29, 2020

sholladay commented Jun 30, 2020

sonicdoe commented Jun 30, 2020

sholladay commented Jun 30, 2020

sonicdoe commented Jul 1, 2020

sholladay commented Jul 2, 2020

sonicdoe commented Jul 3, 2020

sindresorhus commented Mar 7, 2021

sindresorhus commented Jun 3, 2021

sonicdoe commented Jun 4, 2021

Ensure GitHub status checks have passed #289

Ensure GitHub status checks have passed #289

Conversation

sonicdoe commented Jul 26, 2018 • edited

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sindresorhus commented Aug 20, 2018

sonicdoe commented Oct 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sindresorhus commented Jun 6, 2020

sonicdoe commented Jun 7, 2020

sindresorhus commented Jun 29, 2020

sonicdoe commented Jun 29, 2020

sholladay commented Jun 30, 2020

sonicdoe commented Jun 30, 2020

sholladay commented Jun 30, 2020

sonicdoe commented Jul 1, 2020

sholladay commented Jul 2, 2020

sonicdoe commented Jul 3, 2020

sindresorhus commented Mar 7, 2021

sindresorhus commented Jun 3, 2021

sonicdoe commented Jun 4, 2021

sonicdoe commented Jul 26, 2018 •

edited

sonicdoe commented Oct 27, 2018 •

edited