Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate stable and unstable E2E tests for CI run #12059

Closed
wkurniawan07 opened this issue Feb 6, 2023 · 2 comments · Fixed by #12515
Closed

Separate stable and unstable E2E tests for CI run #12059

wkurniawan07 opened this issue Feb 6, 2023 · 2 comments · Fixed by #12515
Labels
a-Build CI, task automation, build pipeline a-Testing Testing-related traits such as efficiency, robustness, coverage c.Task Other non-user-facing works, e.g. refactoring, adding tests help wanted Moderate difficulty, small localized change; suitable for novice contributors

Comments

@wkurniawan07
Copy link
Member

As mentioned in #12035, E2E tests instability is a perpetual problem that has plagued us.

However, it is noteworthy that only a small number of E2E tests are actually unstable enough to warrant rerunning up to 5 times (and sometimes still failing after that). This means that re-running all E2E tests just to have those few unstable tests passing is often time wasted.

Proposal

To have a separate GitHub Actions job (not a separate workflow) for unstable E2E tests.
@wkurniawan07 wkurniawan07 added help wanted Moderate difficulty, small localized change; suitable for novice contributors a-Testing Testing-related traits such as efficiency, robustness, coverage a-Build CI, task automation, build pipeline c.Task Other non-user-facing works, e.g. refactoring, adding tests labels Feb 6, 2023
@domoberzin
Copy link
Contributor

Hi, may I work on this issue? Also is there any info on which tests are stable/unstable at the moment?

@github-actions github-actions bot mentioned this issue Jul 1, 2023
Open
@weiquu
Copy link
Contributor

weiquu commented Jul 2, 2023

Hi @wkurniawan07, do you have any examples of unstable issues? I vaguely recall the Instructor Home test being quite unstable

@domoberzin I think you can also check which tests usually have to be ran multiple times by going into a PR and checking E2E test details, then seeing which need multiple runs before passing. Will take a look at this also

@domoberzin domoberzin mentioned this issue Jul 14, 2023
Merged
weiquu pushed a commit that referenced this issue Jul 27, 2023
@domoberzin @wkurniawan07
[#12059] Separate stable and unstable E2E tests (#12515)
5e58bfa 
* fix: separate stable and unstable E2E tests

* fix: separate unstable tests into a separate job

* fix: revert name change

* fix: reduce duplication for unstable e2e tests

* fix: lint errors

---------

Co-authored-by: Wilson Kurniawan <wkurniawan.92@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
a-Build CI, task automation, build pipeline a-Testing Testing-related traits such as efficiency, robustness, coverage c.Task Other non-user-facing works, e.g. refactoring, adding tests help wanted Moderate difficulty, small localized change; suitable for novice contributors
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#12059] Separate stable and unstable E2E tests domoberzin/teammates
3 participants
@wkurniawan07 @weiquu @domoberzin