Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added --run-tests open in pytest #2085

Merged
merged 6 commits into from
Mar 7, 2024
Merged

Conversation

MikeSullivan7
Copy link
Collaborator

@MikeSullivan7 MikeSullivan7 commented Mar 1, 2024

Issue

Closes #2082.
Closes #1906.

Description

On Windows, running python -m pytest -n auto was running the system tests and the screenshot eyes tests instead of just the "normal" tests.

A --run-unit-tests option is now available with pytest to explicitly only run the unittests by not collecting the eyes_tests and the system-tests. This functionality should be robust against future added tests.
You can also exclude the unit tests from pytest by not using the --run-unit-tests option.

--run-unit-tests has been included into the makefile such that running make test should only run the unittest and not the system tests or screenshot eyes tests. These can be run with make test-system and make test-screenshots (or on windows: make test-screenshots-win) respectively. The --ignore=<Path> can now be used along with custom flags to explicitly exclude certain paths or tests in the collection phase of pytest.

Acceptance Criteria

run the following and make sure that the appropriate tests ran as expected:

  1. make test
  2. make test-verbose
  3. make test-system
  4. make test-screenshots or make test-screenshots-win
  5. make check
  6. use the --ignore=<path> flag along with a custom flag to check it works as expected, e.g. python -m pytest --run-unit-tests --ignore=mantidimaging/core

NOTE: running --run-eyes-tests will only work if the API local key is correctly set up. This can be done via the command line on linux but is done via the environment variables in Windows. The environment variables are taken care of in the makefile so running make test-screenshots-win should run the eyes tests locally on windows.

Documentation

Will add to Developer release notes

@coveralls
Copy link

coveralls commented Mar 1, 2024

Coverage Status

coverage: 74.392% (+0.03%) from 74.359%
when pulling 32c8796 on 2082_pytest_separate_tests
into e82455a on main.

conftest.py Outdated Show resolved Hide resolved
conftest.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@samtygier-stfc samtygier-stfc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works well, thanks.

Possible small tweak. If you could add the path to the system tests to the end of the line in make file rule, so that it it only collects those and does not print out so many skip messages.

	${XVFBRUN} python -m pytest -vs -rs -p no:xdist -p no:randomly -p no:repeat -p no:cov -o log_cli=true --run-system-tests mantidimaging/gui/test/

@samtygier-stfc samtygier-stfc added this pull request to the merge queue Mar 7, 2024
Merged via the queue into main with commit d82757f Mar 7, 2024
8 checks passed
@samtygier-stfc samtygier-stfc deleted the 2082_pytest_separate_tests branch March 7, 2024 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
3 participants