Our tests should run in a certain order to save resources

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

Create a mode in pytest that runs the tests in a certain order and fails the first time any of the tests fails.

We want to use this test to run our challenges. This way we don't waste time on tests that don't even pass unit tests.

Here is a suggestion for the order.
1- UNIT TESTS
2- BAR RAISERS: Challenges that already pass in master but we know are hard to beat.
3- Others.

We can start with this ordering. We could use this test as the bar raiser, since it doesn't look easy to achieve.

Examples 🌈

No response

Motivation 🔦

No response

[Boostrix]

We could use this #3901 as the bar raiser, since it doesn't look easy to achieve.

FWIW, I believe we can work out a way to run an outer agent to mutate those specs in project.txt to create several different versions of the project using different "styles" (think CSS, colors for now) and other attributes (file names), that should then also work with the unit testing portion we're currently drafting to "test" the result afterwards (file names, validate HTML, run XPath on the DOM of the HTML and do a back-end test using an actuall HTTPServer instance) - so if that works out well enough, the same project.txt /COULD/ be used to create different variations of the same project in a loop - such as adding different HTML elements/widgets, which would obviously a good stress-test while increasing code coverage.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.