[MAINT] add script to check duration of test suite #4231

Remi-Gau · 2024-01-17T17:04:37Z

Relates to speed up test for reporting #4220

Changes proposed in this pull request:

add script to collect duration of runs of the test suite in github action and plot the results it over time
mostly for monitoring
does not go too far back as it as it tracks the workflow that uses tox that is fairly new

github-actions · 2024-01-17T17:22:19Z

👋 @Remi-Gau Thanks for creating a PR!

Until this PR is ready for review, you can include the [WIP] tag in its title, or leave it as a github draft.

Please make sure it is compliant with our contributing guidelines. In particular, be sure it checks the boxes listed below.

PR has an interpretable title.
PR links to Github issue with mention Closes #XXXX (see our documentation on PR structure)
Code is PEP8-compliant (see our documentation on coding style)
Changelog or what's new entry in doc/changes/latest.rst (see our documentation on PR structure)

For new features:

There is at least one unit test per new function / class (see our documentation on testing)
The new feature is demoed in at least one relevant example.

For bug fixes:

There is at least one test that would fail under the original bug conditions.

We will review it as quick as possible, feel free to ping us with questions if needed.

codecov · 2024-01-17T17:57:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.57%. Comparing base (29ae5f1) to head (28148bc).
Report is 24 commits behind head on main.

❗ Current head 28148bc differs from pull request most recent head 5f4b746. Consider uploading reports for the commit 5f4b746 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4231      +/-   ##
==========================================
+ Coverage   92.06%   92.57%   +0.51%     
==========================================
  Files         144      144              
  Lines       16419    18145    +1726     
  Branches     3434     4025     +591     
==========================================
+ Hits        15116    16798    +1682     
- Misses        761      782      +21     
- Partials      542      565      +23

Flag	Coverage Δ
macos-latest_3.10_test_plotting	`92.33% <ø> (?)`
macos-latest_3.11_test_plotting	`92.33% <ø> (?)`
macos-latest_3.12_test_plotting	`92.33% <ø> (?)`
macos-latest_3.8_test_plotting	`92.28% <ø> (?)`
macos-latest_3.9_test_plotting	`92.29% <ø> (?)`
ubuntu-latest_3.10_test_plotting	`92.33% <ø> (+0.47%)`	⬆️
ubuntu-latest_3.11_test_plotting	`92.33% <ø> (+0.47%)`	⬆️
ubuntu-latest_3.12_test_plotting	`92.33% <ø> (?)`
ubuntu-latest_3.12_test_pre	`92.33% <ø> (?)`
ubuntu-latest_3.8_test_min	`66.02% <ø> (-2.93%)`	⬇️
ubuntu-latest_3.8_test_plot_min	`92.08% <ø> (+0.51%)`	⬆️
ubuntu-latest_3.8_test_plotting	`92.28% <ø> (+0.46%)`	⬆️
ubuntu-latest_3.9_test_plotting	`92.29% <ø> (+0.47%)`	⬆️
windows-latest_3.10_test_plotting	`92.31% <ø> (?)`
windows-latest_3.11_test_plotting	`92.31% <ø> (+0.47%)`	⬆️
windows-latest_3.12_test_plotting	`92.31% <ø> (?)`
windows-latest_3.8_test_plotting	`92.26% <ø> (+0.47%)`	⬆️
windows-latest_3.9_test_plotting	`92.27% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Remi-Gau · 2024-01-18T07:57:36Z

output example

Remi-Gau · 2024-01-18T07:58:19Z

does not have to go into this repo but handy to have around

bthirion · 2024-01-18T21:56:17Z

output example

Thx. Not that the use of a unqiue color for different curves makes it barely usable.

bthirion · 2024-01-18T22:20:56Z

OK, you will then create the Github action, I guess ?

bthirion

Could not spot any issue.

Remi-Gau · 2024-01-19T07:06:46Z

OK, you will then create the Github action, I guess ?

yeah to run on command and every month or so.

Remi-Gau · 2024-01-30T13:25:09Z

will update graph and add this to CI to help automate it

maint_tools/requirements.txt

Remi-Gau · 2024-01-31T12:17:30Z

Not that the use of a unqiue color for different curves makes it barely usable.

improved this a bit:

color = python version
dash = OS
marker = dependencies

Remi-Gau · 2024-01-31T12:22:20Z

OK, you will then create the Github action, I guess ?

This is done

bthirion

LGTM, thx.

bthirion · 2024-01-31T21:49:46Z

Happy to hear about other's opinion on this feature.

Remi-Gau · 2024-02-01T06:54:23Z

Happy to hear about other's opinion on this feature.

As this is not urgent I will wait for a couple of weeks before merging.

Remi-Gau · 2024-02-26T12:52:58Z

Unless I hear anything otherwise I will merge this next week.

maint_tools/check_gha_workflow.py

emdupre · 2024-03-01T22:03:00Z

maint_tools/check_gha_workflow.py

+    return next(
+        (
+            version
+            for version in ["3.8", "3.9", "3.10", "3.11", "3.12"]


Is there a way we could keep this in-sync with the Python versions we support, without having to remember to update it here ?

Cannot think of an easy way... Yet.

Could not find an easy way to do this, so for now I at least extract this list into a constant at the top of the script that should make it easier to find and modify.

maint_tools/check_gha_workflow.py

Co-authored-by: Elizabeth DuPre <emd222@cornell.edu>

Remi-Gau · 2024-03-04T08:54:34Z

maint_tools/check_gha_workflow.py

+This may require a github token to work
+(if you need to make a lot of call to the GitHub API).
+
+You can get a github token at:
+https://github.com/settings/tokens
+
+You can either:
+
+- save the github token in a file and modify the script
+  (see the variable USERNAME and TOKEN_FILE below).
+  This is can be useful if you want to run the script locally.
+
+- pass the token directly to the script as an argument.
+  This is the way to do it when using this script in continuous integration.
+
+USAGE
+-----
+
+.. code-block:: bash
+
+    python maint_tools/check_gha_workflow.py $GITHUB_TOKEN


@emdupre
does this seem more clear?

Yes, thanks !

* add script to check duration of test suite * add comment * comment and refactor * typo * improve paging * add workflow * Update maint_tools/requirements.txt * fail on bad request * upload artefacts * upload artefacts * remove running on PR * Apply suggestions from code review Co-authored-by: Elizabeth DuPre <emd222@cornell.edu> * update doc string * update doc string * refactor --------- Co-authored-by: Elizabeth DuPre <emd222@cornell.edu>

…ap with masker report images (#4308) * extract table folding and overlapping * update changelog * Update doc/changes/latest.rst Co-authored-by: bthirion <bertrand.thirion@inria.fr> * FIX] fixes several plotting details in glm reports (#4266) * fix several issues in glm reports * further fixes * change cmap for glass brain * fix remaining issues * [full doc] request full build * update changelog * [full doc] request full build * [FIX] force views in surface plotting to be pairs of `int` or `float` when not a `string` (#4297) * for views to be pairs of int or float when not a string * update changelog * [FIX] fix cut position in nifti maps maskers (#4304) * fix cut position in nifti maps maskers * update changelog * [FIX] remove conftest for externals (#4306) * remove conftest tempita * ignore tempita in pyproject.toml * [MAINT] add script to check duration of test suite (#4231) * add script to check duration of test suite * add comment * comment and refactor * typo * improve paging * add workflow * Update maint_tools/requirements.txt * fail on bad request * upload artefacts * upload artefacts * remove running on PR * Apply suggestions from code review Co-authored-by: Elizabeth DuPre <emd222@cornell.edu> * update doc string * update doc string * refactor --------- Co-authored-by: Elizabeth DuPre <emd222@cornell.edu> * [DOC] display all masker reports in the doc (#4295) * display masker reports in doc * update changelog * refactor * add notebooks * clear cells * [full doc] request full build * add link to examples * [full doc] request full build * fix doc build * [full doc] request full build * [full doc] request full build --------- Co-authored-by: bthirion <bertrand.thirion@inria.fr> Co-authored-by: Elizabeth DuPre <emd222@cornell.edu>

Remi-Gau added 2 commits January 17, 2024 18:01

add script to check duration of test suite

0c1cb06

add comment

ed8fbb2

Remi-Gau added 2 commits January 18, 2024 08:52

comment and refactor

2413208

typo

3904d32

Remi-Gau marked this pull request as ready for review January 18, 2024 07:58

Remi-Gau changed the title ~~[WIP][MAINT] add script to check duration of test suite~~ [MAINT] add script to check duration of test suite Jan 18, 2024

bthirion approved these changes Jan 18, 2024

View reviewed changes

Remi-Gau added 3 commits January 30, 2024 14:25

Merge remote-tracking branch 'upstream/main' into test_speed

82184ae

improve paging

c076a85

add workflow

18a3fcb

Remi-Gau commented Jan 31, 2024

View reviewed changes

maint_tools/requirements.txt Outdated Show resolved Hide resolved

Remi-Gau added 4 commits January 31, 2024 11:42

Update maint_tools/requirements.txt

7063e0c

fail on bad request

ce3a131

upload artefacts

9ba1531

upload artefacts

88241f9

remove running on PR

28148bc

Remi-Gau requested a review from bthirion January 31, 2024 12:21

bthirion approved these changes Jan 31, 2024

View reviewed changes

Remi-Gau marked this pull request as draft February 1, 2024 06:54

Remi-Gau marked this pull request as ready for review February 26, 2024 12:52

Remi-Gau closed this Feb 29, 2024

Remi-Gau deleted the test_speed branch February 29, 2024 07:56

Remi-Gau restored the test_speed branch February 29, 2024 07:58

Remi-Gau reopened this Feb 29, 2024

emdupre reviewed Mar 1, 2024

View reviewed changes

maint_tools/check_gha_workflow.py Show resolved Hide resolved

emdupre reviewed Mar 1, 2024

View reviewed changes

maint_tools/check_gha_workflow.py Outdated Show resolved Hide resolved

emdupre reviewed Mar 1, 2024

View reviewed changes

maint_tools/check_gha_workflow.py Show resolved Hide resolved

emdupre reviewed Mar 4, 2024

View reviewed changes

maint_tools/check_gha_workflow.py Outdated Show resolved Hide resolved

Remi-Gau and others added 4 commits March 4, 2024 09:27

Apply suggestions from code review

5ce3160

Co-authored-by: Elizabeth DuPre <emd222@cornell.edu>

update doc string

02db34a

update doc string

e21a2f9

refactor

5f4b746

Remi-Gau commented Mar 4, 2024

View reviewed changes

Remi-Gau merged commit 9d2090f into nilearn:main Mar 5, 2024
12 of 31 checks passed

Remi-Gau deleted the test_speed branch March 5, 2024 08:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAINT] add script to check duration of test suite #4231

[MAINT] add script to check duration of test suite #4231

Remi-Gau commented Jan 17, 2024 •

edited

github-actions bot commented Jan 17, 2024

codecov bot commented Jan 17, 2024 •

edited

Remi-Gau commented Jan 18, 2024

Remi-Gau commented Jan 18, 2024

bthirion commented Jan 18, 2024

bthirion commented Jan 18, 2024

bthirion left a comment

Remi-Gau commented Jan 19, 2024

Remi-Gau commented Jan 30, 2024

Remi-Gau commented Jan 31, 2024

Remi-Gau commented Jan 31, 2024

bthirion left a comment

bthirion commented Jan 31, 2024

Remi-Gau commented Feb 1, 2024

Remi-Gau commented Feb 26, 2024

emdupre Mar 1, 2024

Remi-Gau Mar 2, 2024

Remi-Gau Mar 4, 2024

Remi-Gau Mar 4, 2024

emdupre Mar 4, 2024

[MAINT] add script to check duration of test suite #4231

[MAINT] add script to check duration of test suite #4231

Conversation

Remi-Gau commented Jan 17, 2024 • edited

github-actions bot commented Jan 17, 2024

codecov bot commented Jan 17, 2024 • edited

Codecov Report

Remi-Gau commented Jan 18, 2024

Remi-Gau commented Jan 18, 2024

bthirion commented Jan 18, 2024

bthirion commented Jan 18, 2024

bthirion left a comment

Choose a reason for hiding this comment

Remi-Gau commented Jan 19, 2024

Remi-Gau commented Jan 30, 2024

Remi-Gau commented Jan 31, 2024

Remi-Gau commented Jan 31, 2024

bthirion left a comment

Choose a reason for hiding this comment

bthirion commented Jan 31, 2024

Remi-Gau commented Feb 1, 2024

Remi-Gau commented Feb 26, 2024

emdupre Mar 1, 2024

Choose a reason for hiding this comment

Remi-Gau Mar 2, 2024

Choose a reason for hiding this comment

Remi-Gau Mar 4, 2024

Choose a reason for hiding this comment

Remi-Gau Mar 4, 2024

Choose a reason for hiding this comment

emdupre Mar 4, 2024

Choose a reason for hiding this comment

Remi-Gau commented Jan 17, 2024 •

edited

codecov bot commented Jan 17, 2024 •

edited