Collect and tag flaky tests

[srbarrios]

What does this PR change?

Related to:

• https://github.com/SUSE/spacewalk/issues/22377
• https://hackweek.opensuse.org/23/projects/collect-flaky-test-cases-identified-by-the-team-in-a-github-board-and-highlight-them-in-the-test-report

Flaky tests: Those that have not a consistent result, and sometimes are failing and others are passing.

When we review a Test Report in a Pull Request, we would like to identify which of the test features or scenarios we are reviewing has been failing intermittently in the past, which will facilitate a comparison of results between our PR and the main branch.

To address that we want to:

• Have a list of flaky test cases
• We can use the cards in the view https://github.com/orgs/SUSE/projects/23/views/7 (We can do API calls with graphql for that)
• Inject Cucumber Tags into the .feature file (feature or scenario levels)

GUI diff

No difference.

• DONE

Documentation

• No documentation needed

• DONE

Test coverage

• No tests

• DONE

Links

Ports:

• Manager-4.3 https://github.com/SUSE/spacewalk/pull/22963

• DONE

Changelogs

Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository

If you don't need a changelog check, please mark this checkbox:

• No changelog needed

If you uncheck the checkbox after the PR is created, you will need to re-run changelog_test (see below)

Re-run a test

If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run:

• Re-run test "changelog_test"
• Re-run test "backend_unittests_pgsql"
• Re-run test "java_pgsql_tests"
• Re-run test "schema_migration_test_pgsql"
• Re-run test "susemanager_unittests"
• Re-run test "javascript_lint"
• Re-run test "spacecmd_unittests"

[github-actions]

👋 Hello! Thanks for contributing to our project.
Acceptance tests will take some time (aprox. 1h), please be patient ☕
You can see the progress at the end of this page and at https://github.com/uyuni-project/uyuni/pull/7832/checks
Once tests finish, if they fail, you can check 👀 the cucumber report. See the link at the output of the action.
You can also check the artifacts section, which contains the logs at https://github.com/uyuni-project/uyuni/pull/7832/checks.
See the troubleshooting guide if you need any help.

Happy hacking!
⚠️ You should not merge if acceptance tests fail to pass. ⚠️

[jordimassaguerpla]

Hi @srbarrios ! In general I like the whole idea! IIUC we do not need any additional hardware or service, but we use the github project ... great idea!

I have a few comments.

what is the GH_TOKEN_GALAXY_CI in GITHUB_TOKEN: ${{ secrets.GH_TOKEN_GALAXY_CI }} ?

I would not add the call to the new rake task into testsuite/podman_runner/05_install_gems_in_controller.sh . I think this is confusing. What about creating a new script? 06_update_flaky_tests.sh? or rename the 05 to 05_install_gems_in_controller_and_update_flaky_tests ...

[srbarrios]

what is the GH_TOKEN_GALAXY_CI in GITHUB_TOKEN: ${{ secrets.GH_TOKEN_GALAXY_CI }} ?

This is a secret that probably will only work in branches, but not in forks.
So for now.. I think this will not work on forks, we will need to figure out a different approach on GH Validation for PR of forks.
An option could be to upload daily the flaky tests of our private GH Board to our Redis instance with public read access in cloud, and get the information from there.

I would not add the call to the new rake task into testsuite/podman_runner/05_install_gems_in_controller.sh . I think this is confusing. What about creating a new script? 06_update_flaky_tests.sh? or rename the 05 to 05_install_gems_in_controller_and_update_flaky_tests ...

Indeed.
I was worry about renumbering the current scripts, but if you don't mind, then I will separate the new actions in different scripts.