Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Replace current GitHub Actions test reporting solution with less verbose solution #16736

Closed
lhotari opened this issue Jul 22, 2022 · 6 comments · Fixed by #17319
Closed
Labels

Comments

@lhotari
Copy link
Member

lhotari commented Jul 22, 2022

Is your enhancement request related to a problem? Please describe.

We are currently using a test reporting solution based on https://github.com/dorny/test-reporter and there's a copy of this action in https://github.com/apache/pulsar-test-infra/tree/master/test-reporter .

The problem with this solution is that the test reporting is very verbose since it creates new check runs for the PR.
When there are failed tests, the PR will be marked failing even in cases where the tests pass in a later retry.

Describe the solution you'd like

There's an alternative solution in https://github.com/mikepenz/action-junit-report which is actively maintained.
It's possible to use this action with annotate_only: 'true' so that check runs aren't created at all. This is to mitigate the gap in GitHub Actions solution that check results cannot target a specific workflow when there are multiple active ones (more details in mikepenz/action-junit-report#40 (comment)).
The action will create a report for each build job using a new method available in GitHub Actions. example: https://github.com/mikepenz/action-junit-report/actions/runs/2717236074#summary-7464569272

@lhotari
Copy link
Member Author

lhotari commented Jul 22, 2022

The change could be tested in your own fork (some instructions in https://pulsar.apache.org/contributing/#ci-testing-in-your-fork) before submitting a PR.
The copy of https://github.com/mikepenz/action-junit-report would have to be added to https://github.com/apache/pulsar-test-infra since only whitelisted or trusted locations can be used in GH actions workflows for apache/pulsar.

@lhotari
Copy link
Member Author

lhotari commented Jul 22, 2022

https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/ explains the new feature available in GitHub Actions which is used in action-junit-report .

@github-actions
Copy link

The issue had no activity for 30 days, mark with Stale label.

@github-actions github-actions bot added the Stale label Aug 22, 2022
@lhotari
Copy link
Member Author

lhotari commented Aug 28, 2022

running an experiment in lhotari#78

@eolivelli
Copy link
Contributor

The experiments looks promising

@tisonkun
Copy link
Member

https://github.com/apache/pulsar/pull/17282/checks?check_run_id=8047947991 I notice another issue on test reports that the report can be assigned to multiple jobs.

lhotari added a commit that referenced this issue Aug 30, 2022
Fixes #16736

* [improve][CI] Replace test reporting with less verbose solution

- uses https://github.com/mikepenz/action-junit-report which has been
  copied to https://github.com/apache/pulsar-test-infra/tree/master/action-junit-report

* Address review comment: remove merge-test-reports action
nicoloboschi pushed a commit that referenced this issue Sep 9, 2022
Fixes #16736

* [improve][CI] Replace test reporting with less verbose solution

- uses https://github.com/mikepenz/action-junit-report which has been
  copied to https://github.com/apache/pulsar-test-infra/tree/master/action-junit-report

* Address review comment: remove merge-test-reports action

(cherry picked from commit cb881d4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants