Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Balancing benchmark scheduling #2

Open
falkecarlsen opened this issue Aug 28, 2023 · 0 comments
Open

Balancing benchmark scheduling #2

falkecarlsen opened this issue Aug 28, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@falkecarlsen
Copy link
Member

Most suites have a natural increase in complexity and compute-time needed to finish, e.g. Robot Coordination; as you increase the grid-size, the computation grows exponentially.

Sorting the tests in a suite is a good first step to balance where time is spent checking performance. There is no need to start at the most expensive tests. However, when comparing search strategies, we don't know when to switch.

Currently, each strategy runs to completion on a suite, before progressing to the next strategy. Running each tests for all strategies before advancing could be good. This is just switching the order of iteration. However, then we need to load intermediate CSVs for strategies, when these resume benching. Indexing into and saving upon test completionpds[STRATEGIES.length] is probably also fine.

We also have a timeout set per bench. Since we iterate over six different thread-counts, this has an upper bound of taking 12 hours. While we have taken pains to avoid noise in results, for longer running benchmarks allowing parallelism might be fine.

@falkecarlsen falkecarlsen added the enhancement New feature or request label Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant