Circle CI not triggering builds on some pull requests

[chris48s]

📋 Description

This is absolutely doing my head in and we seem to be seeing it more frequently.

I don't know what the problem is but I have spent the time to write up an issue on the Circle CI support forum. Lets see where it goes.. https://discuss.circleci.com/t/circle-ci-builds-failing-to-trigger-on-some-pull-requests/44472

[chris48s]

I raised this as a support request. Here's the response I got

Thank you for reaching out to CircleCI Support. From the information you've provided, we've determined that this could be hitting a known issue related to permissions on open projects at this time. This scenario has open work in flight and we've linked your case to this internal tracker for further updates in case this does fit after further investigation.

In the meantime, will you please ask the user to sign in to the CircleCI application and authorize the app so that it appears in the personal account settings in case this helps: https://github.com/settings/applications

Can you let us know if that particular user has this under those settings and if not, did that help?

Another scenario where this could happen is where the user is following the project on the fork and not on the source of the project. Has this user's builds ever run? See https://circleci.com/docs/2.0/oss under "Note" or Why aren't pull requests triggering jobs on my organization?

I can't find the PR but having read this, I seem to remember someone submitting a PR and saying they had to create an account/log in to CircleCI to get the build to start.. does that ring a bell?

@sunny0826 there might be some advice above or in https://support.circleci.com/hc/en-us/articles/360008097173-Why-aren-t-pull-requests-triggering-jobs-on-my-organization- that might be worth a try next time you open a PR (although you're not the only contributor we've seen have this issue).

There is a part of me also wondering if the time has come to migrate everything off CircleCI and on to Github Actions. We're already using GHA for quite a lot of things. It might improve everyone's contributor experience if we moved the CI builds there too.

[calebcartwright]

We'd seen this before with people not being able to view the Circle logs if they didn't have an account, but nothing like what we've seen recently. There was a non-brief period when the jobs weren't triggering against a PR I opened, in this-repo, so at a minimum I think there are (or at least have been) multiple issues at play.

There is a part of me also wondering if the time has come to migrate everything off CircleCI and on to Github Actions. We're already using GHA for quite a lot of things. It might improve everyone's contributor experience if we moved the CI builds there too.

I'm not opposed to this in spirit, and have largely made the same move in other projects. However, GitHub Actions is still quite feature nascent on several fronts, notably including rich test execution and reporting. Don't anticipate that being a monumental effort to overcome, but for certain jobs, e.g. daily tests, it may be a trickier than expected exercise

[chris48s]

We'd seen this before with people not being able to view the Circle logs if they didn't have an account

Aah yes - that sounds right. Well remembered

GitHub Actions is still quite feature nascent on several fronts, notably including rich test execution and reporting. Don't anticipate that being a monumental effort to overcome, but for certain jobs, e.g. daily tests, it may be a trickier than expected exercise

This is another good point. One nice thing Circle CI does is pull out the bits you care about from a giant pile of text. I reckon this might be a good thing to look in to if we want to replicate that: https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

In some ways daily tests is less critical because its only really us that use it - not external contributors working on forks but equally I wouldn't want to be in a situation where the main repo and daily tests use a totally different test environment for any length of time.

[chris48s]

@sunny0826 - how did you manage to get this working?

There was a period where every PR you submitted wasn't running the builds, but they have been running on all your recent contributions. Did you change something, or did it just start working one day?

[sunny0826]

@sunny0826 - how did you manage to get this working?

There was a period where every PR you submitted wasn't running the builds, but they have been running on all your recent contributions. Did you change something, or did it just start working one day?

You may have missed my previous feedback #8108 (comment). This issue was caused by me not registering for Circle CI, and it went away after I completed the registration and bound GItHub.

[chris48s]

Thanks - did you have to grant Circle write access to private repos to do that?

[sunny0826]

yep. The permissions granted were actually full control! which I hadn't even noticed before.

[chris48s]

OK thanks for confirming.

Its kind of good that we know how to "fix" this. I'm still not 100% sure if this is required or if there is something else less invasive that users could do in order to submit PRs where the builds run.

Obviously us as maintainers have to give Circle CI a certain amount of access/trust, but if we also have to say to some subset of users: "step 1 of contributing to this open source project is give this app you may or may not trust write access to your private repos", that's not a position I'm really comfortable having to adopt and the complaint in #8275 (comment) is justified IMO.

Given that, I think we should probably plan to start gradually migrating stuff from Circle to GHA (although I probably don't have time to start this until ~next week). It doesn't have to be a 'big bang'. I think we can take each of the builds (main, frontend, package, integration, services, etc) one-by-one in its own PR and do it gradually. I'm sure each will throw up its own challenges 😄 I am quite keen to have an experiment with https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/ For the services, there is a WIP at #5129 already, although we can probably reign the old scope in on that on a bit.

[chris48s]

I have made a start on this.

You're right that there is going to be a bit of work to generate decent outputs to allow people to see what failed and why, but certainly not impossible.

One thing that I have realised since starting on it is that there are a number of things we kind of hacked one way or another with Circle to reduce the number of concurrent builds we are running (e.g: not having separate builds for tests and linters, running the package builds one-by-one using NVM), but migrating to GHA should allow us to just run as many as we want.

[calebcartwright]

Obviously us as maintainers have to give Circle CI a certain amount of access/trust, but if we also have to say to some subset of users: "step 1 of contributing to this open source project is give this app you may or may not trust write access to your private repos", that's not a position I'm really comfortable having to adopt and the complaint in #8275 (comment) is justified IMO.

Completely agree with this 👍

[chris48s]

The core of this is done.

✔️ Last build migrated from Circle to GHA.
✔️ Branch protection rules updated
✔️ Circle CI disabled
✔️ Env vars deleted from Circle

There's a few bits of follow up needed which I have noted in

• Implement check for .only() in service tests #8819 (danger)
• Find a way to run service tests for services requiring auth on PRs from forks #8820 (services)
• Migrate daily service tests to GitHub actions #8821 (daily tests)

If there is any more mop-up I have failed to capture, shout.