Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: Speed up test_batch_run #2039

Merged
merged 2 commits into from
Feb 16, 2024
Merged

tests: Speed up test_batch_run #2039

merged 2 commits into from
Feb 16, 2024

Conversation

rht
Copy link
Contributor

@rht rht commented Feb 16, 2024

This is an attempt to speed up overall CI testing time, as much as possible without reducing comprehensiveness. Currently, it takes 12s on Python 3.12 on Ubuntu.

Copy link

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
Schelling small 🔵 +0.7% [+0.5%, +0.9%] 🔵 -0.1% [-0.2%, +0.1%]
Schelling large 🔵 +0.0% [-1.7%, +1.9%] 🔵 +0.5% [-0.5%, +1.6%]
WolfSheep small 🔵 +0.7% [+0.4%, +1.0%] 🔵 +0.1% [-0.0%, +0.2%]
WolfSheep large 🔵 +0.4% [-1.2%, +2.0%] 🔵 +2.4% [+1.0%, +4.0%]
BoidFlockers small 🔵 +0.5% [-0.1%, +1.1%] 🔵 +0.3% [-0.4%, +0.9%]
BoidFlockers large 🔵 +0.1% [-0.3%, +0.5%] 🔵 -0.1% [-0.6%, +0.4%]

@quaquel
Copy link
Contributor

quaquel commented Feb 16, 2024

Can you explain why you are concerned about CI testing time?

@rht
Copy link
Contributor Author

rht commented Feb 16, 2024

  1. It speeds up the review feedback loop between the reviewers and PR (that has been marked as ready) authors. The sub 1 min vs 2 min+ speedup benefit compounds for a single PR that requires several testing such as Add cell-centric discrete spaces (experimental) #1994. The Ubuntu CI runs used to be ~1 min, now about 2x faster at 30s.
  2. More immediate feedback to contributors.

@quaquel
Copy link
Contributor

quaquel commented Feb 16, 2024

Fair enough, although this does not make a massive difference for me. I always think of this in a more categorical manner: can I wait behind my desk, can I go for a cup of coffee, can it run over lunch, or does it take even longer? The difference between 30 seconds and 2 minutes is still very much in the behind-my-desk category.

Also, I see CI stuff as stuff that runs after having tested everything locally anyway, so I don't really see the feedback loop argument.

To be clear: speedups are still nice.

Copy link
Contributor

@EwoutH EwoutH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see significant degradation in test functionality, so looks good. Faster CI is always nice, thanks!

@rht rht merged commit ac9fdd1 into projectmesa:main Feb 16, 2024
13 checks passed
@rht rht deleted the test-faster branch February 16, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants