Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run a nightly Github Actions workflow to monitor tests memory per test (configurable for other metrics too) and lists the slowest 100 tests #2449

Merged
merged 8 commits into from
Jan 10, 2022

Conversation

valeriupredoi
Copy link
Contributor

@valeriupredoi valeriupredoi commented Jan 5, 2022

Description

Porting the test performance monitor we have in ESMValCore. Github Actions tests OK (py3.8 died of segfaulted worker, common case) 馃嵑

Small nightly Github Actions workflow that installs and uses pytest-monitor as pytest plugin to output a .pymon sqlite database where performance metrics have been recorded for each pytest test we run (memory, CPU time, usage etc); the wflow runs a wee script I wrote that currently outputs the name of the test and amount of RES memory it used if more than 1000 MB (this can be extended to output a whole lot other metrics). Note that this works only for pytests and not for unittests; also note that I wouldn't turn it on as a mainstream pytest plugin since it slows down the tests (not sure by how much but it's quite visible). The GH Action parses the output via the script and lists the 100 slowest pytest tests from the slowest at the top - this is CPU time.


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 馃洜 Technical or 馃И Scientific review.

@zklaus
Copy link
Contributor

zklaus commented Jan 10, 2022

It is not clear to me why this is being added. @valeriupredoi, could you please improve the description and possibly add an issue or reference one?

While it is ok to remove items from the checklist that are not applicable, this mechanism is not intended to skip all housekeeping that is deemed cumbersome.

@valeriupredoi
Copy link
Contributor Author

fair do's, Klaus, I will 馃嵑

@valeriupredoi valeriupredoi changed the title added tests monitor Github Action workflow Run a nightly Github Actions workflow to monitor tests memory per test (configurable for other metrics too) and lists the slowest 100 tests Jan 10, 2022
@valeriupredoi
Copy link
Contributor Author

done, @bouweandela had previously put this to good use to optimize a few Core tests, I will have a look at the Tool tests too - you OK with this, @zklaus ?

@zklaus
Copy link
Contributor

zklaus commented Jan 10, 2022

Looking good, thanks!

@zklaus zklaus merged commit 012cdd2 into main Jan 10, 2022
@zklaus zklaus deleted the test-monitor branch January 10, 2022 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants