[Testing:Developer] Manual parallelizing in Github Actions

[IDzyre]

Please check if the PR fulfills these requirements:

• Tests for the changes have been added/updated (if possible)
• Documentation has been updated/added if relevant
• Screenshots are attached to Github PR if visual/UI changes were made

What is the current behavior?

Currently, Cypress Cloud parallelizes the runs, however this only works for Submitty members, not external contributors. It also has a limit of 100,000 different test runs (Specific It('should test...', ()=>{}); runs, not spec files or describe instances). With increasing development, and the creation of new spec files that add to the tests, this will most likely not be enought.

What is the new behavior?

The runs are now split up into five different categories manually, and Cypress Cloud has been removed from CI:

• Admin -- Testing admin features, or website options
• Feature -- Testing user features, polls, forums, late day caching, etc.
• Gradeable -- Testing creation, editing, submission of gradeables
• System -- Testing things that need other system functions to run, and accessibility testing.
• UI -- Testing misc user interfaces

[codecov]

Codecov Report

Merging #9737 (3124099) into main (ae74ecc) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #9737   +/-   ##
=========================================
  Coverage     23.25%   23.25%           
  Complexity     8232     8232           
=========================================
  Files           227      227           
  Lines         29362    29362           
  Branches         73       73           
=========================================
  Hits           6828     6828           
  Misses        22464    22464           
  Partials         70       70           

Flag	Coverage Δ
autograder	22.15% <ø> (ø)
js	27.51% <ø> (ø)
migrator	100.00% <ø> (ø)
php	20.15% <ø> (ø)
python_submitty_utils	71.65% <ø> (ø)
submitty_daemon_jobs	91.01% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jason490]

You can try using the github matrix to get rid of doing similar operations
ex:

Cypress-Test:
    ...
   strategy:
      matrix:
           containers: ["gradeable","admin","UI",etc...]
   ...
      ...
           spec: cypress/e2e/${{matrix.containers}}/*

[williamjallen]

Changes seem reasonable to me. I do worry about needing to rebalance the sets of tests occasionally, but that's certainly manageable, and better than the alternative of just running all of the tests sequentially.

Is there any parallelization you can do in one or more of the test sets? We'd have to make sure none of the tests potentially interfere with one another and the benefits of parallelization would be hampered by the limited resources available on the CI runners, but it may reduce the time slightly at least.

[IDzyre]

Changes seem reasonable to me. I do worry about needing to rebalance the sets of tests occasionally, but that's certainly manageable, and better than the alternative of just running all of the tests sequentially.

Is there any parallelization you can do in one or more of the test sets? We'd have to make sure none of the tests potentially interfere with one another and the benefits of parallelization would be hampered by the limited resources available on the CI runners, but it may reduce the time slightly at least.

I think its probably possible, but to make the CI run more smoothly as it is, I think that can be a future PR.

[williamjallen]

Good work on this!