Skip to content

add a job that check all other job status #3230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

cbeauchesne
Copy link
Collaborator

@cbeauchesne cbeauchesne commented May 5, 2025

Description

Add a job in the CI that runs in PR. The job will be a success if all other jobs are skipped/success, and fails otherwise

Motivation

While it's possible to enforce a green CI policy using GitHub's native "required status checks" feature, doing so requires explicitly listing all job names under branch protection rules. This approach has two key drawbacks:

  • It does not support optional jobs
  • It introduces ongoing maintenance overhead as the job list evolves

This jobs will check ALL other job, and we'll be able to set this as a requirement. The action used offers a ignored-name-patterns parameters`, I added few job that failed on that PR, let me know if others should be added.

the plan is to merge this PR, wait few days to be sure that everything is fine. Then add it as a requirement.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@cbeauchesne cbeauchesne requested a review from a team as a code owner May 5, 2025 10:22
@cbeauchesne cbeauchesne marked this pull request as draft May 5, 2025 10:22
@codecov-commenter
Copy link

codecov-commenter commented May 5, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.41%. Comparing base (0697a7f) to head (2f10573).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #3230   +/-   ##
=========================================
  Coverage     71.41%   71.41%           
  Complexity     2948     2948           
=========================================
  Files           118      118           
  Lines         11633    11633           
=========================================
  Hits           8308     8308           
  Misses         3325     3325           
Flag Coverage Δ
tracer-php 71.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0697a7f...2f10573. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cbeauchesne cbeauchesne force-pushed the cbeauchesne/all-jobs-are-green branch from 28f9055 to 2f10573 Compare May 5, 2025 11:15
runs-on: ubuntu-latest
steps:
- name: Run Ensure CI Success
uses: DataDog/ensure-ci-success@v1

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've built something similar to replace https://github.com/upsidr/merge-gatekeeper in our CI - https://github.com/lendable/sloth - maybe you would be interested in joining the effort? 😊

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gosh, I crawl the web to find an action that fits my needs, and I missed yours 🤕 .

Unfortunately, I won't be able to join, as go is chineese to me.

Though, I'll put some suggestion easy to tackle that would have been a blocker for me. If at some point it's a good fit, I'll consider using yours!

Thank's a lot for the head's up!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, ours is the latter one though (sloth), which is in TypeScript :)

@cbeauchesne cbeauchesne marked this pull request as ready for review May 5, 2025 18:02
Copy link
Collaborator

@bwoebi bwoebi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not going to merge this currently.

This request changes is not a request for changes today, but for changes in the future, around ignored-name-patterns.

The reason is that we're in the middle of migration to gitlab CI, which currently has more flaky and non-working tests, which we might have to merge within the next two weeks, even with not all issues resolved.
This requirement will then be ugly and with possibly many exceptions.
I'd like to revisit this PR again within a month though.

@cbeauchesne
Copy link
Collaborator Author

cbeauchesne commented May 5, 2025

Hi @bwoebi ,

ignored-name-patterns is a list of pattern (regex), so it's quite easy to ignore entirely gitlab by using gitlab/.*.

At least, other jobs will be in required list. My point is that if we wait to have enough healthy jobs to move forward, we'll probably never move forward, as some jobs may become unhealthy while we're fixing other.

I totally understand the burden of maintaining a big list, that's why I used regexes for this parameters, please let me know if it can be a good fit.

@bwoebi
Copy link
Collaborator

bwoebi commented May 5, 2025

@cbeauchesne This is pointless though as within two weeks all ci/circleci will disappear, leaving this check effectively checking nothing then.
I'd rather consider this PR when the time comes to do it properly.

It's not "wait for enough healthy jobs", but "let's merge gitlab and wait 1-2 weeks to see where it's actually unstable and then apply this".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants