Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use test-queue for parallel testing #1570

Merged
merged 1 commit into from
May 12, 2023
Merged

Conversation

koic
Copy link
Contributor

@koic koic commented May 7, 2023

This PR makes developer tests parallel using test-queue: https://github.com/tmm1/test-queue

Behavior

Before

Around 55 seconds:

$ bundle exec rake spec
(snip)

Finished in 55.03 seconds (files took 0.77671 seconds to load)
865 examples, 0 failures

Randomized with seed 6253

After

Around 11 seconds (e.g. 8 CPUs with hyper threading) :

$ bundle exec rake spec
rspec-queue spec
Starting test-queue master (/tmp/test_queue_12223_3620.sock)

==> Summary (16 workers in 11.6051s)

    [ 1]                                      65 examples, 0 failures         1 suites in 7.1969s      (pid 12226 exit 0 )
    [ 2]                                      19 examples, 0 failures         1 suites in 7.3070s      (pid 12227 exit 0 )
    [ 3]                                      20 examples, 0 failures         1 suites in 11.5977s      (pid 12228 exit 0 )
    [ 4]                                      80 examples, 0 failures         3 suites in 11.5978s      (pid 12229 exit 0 )
    [ 5]                                     101 examples, 0 failures         3 suites in 11.5977s      (pid 12230 exit 0 )
    [ 6]                                      55 examples, 0 failures         5 suites in 11.5976s      (pid 12231 exit 0 )
    [ 7]                                      49 examples, 0 failures         4 suites in 11.5975s      (pid 12232 exit 0 )
    [ 8]                                      57 examples, 0 failures         6 suites in 11.5974s      (pid 12233 exit 0 )
    [ 9]                                      31 examples, 0 failures         6 suites in 11.5973s      (pid 12234 exit 0 )
    [10]                                      46 examples, 0 failures         4 suites in 11.5972s      (pid 12235 exit 0 )
    [11]                                      55 examples, 0 failures         5 suites in 11.5971s      (pid 12236 exit 0 )
    [12]                                      57 examples, 0 failures         6 suites in 11.5968s      (pid 12237 exit 0 )
    [13]                                      70 examples, 0 failures         6 suites in 11.5967s      (pid 12238 exit 0 )
    [14]                                      57 examples, 0 failures         5 suites in 11.5962s      (pid 12239 exit 0 )
    [15]                                      47 examples, 0 failures         6 suites in 11.5951s      (pid 12240 exit 0 )
    [16]                                      56 examples, 0 failures         6 suites in 11.5933s      (pid 12241 exit 0 )

It can be up to 5x faster depending on the environment.

Other information

If fork is not supported, parallelization is not possible.


Additional info

Pull request checklist

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been reviewed and added / updated if needed (for bug fixes / features)
  • Added the appropriate label for the given change

Does this introduce a breaking change?

Please see our docs on breaking changes to help!

  • Yes (Please add the Type: Breaking change label)
  • No

If Yes, what's the impact:

  • N/A

Pull request type

Please add the corresponding label for change this PR introduces:

  • Bugfix: Type: Bug
  • Feature/model/API additions: Type: Feature
  • Updates to docs or samples: Type: Documentation
  • Dependencies/code cleanup: Type: Maintenance

This PR makes developer tests parallel using test-queue:
https://github.com/tmm1/test-queue

## Before

Around 55 seconds:

```console
$ bundle exec rake spec
(snip)

Finished in 55.03 seconds (files took 0.77671 seconds to load)
865 examples, 0 failures

Randomized with seed 6253
```

## After

Around 11 seconds (e.g. 8 CPUs with hyper threading) :

```console
$ bundle exec rake spec
rspec-queue spec
Starting test-queue master (/tmp/test_queue_12223_3620.sock)

==> Summary (16 workers in 11.6051s)

    [ 1]                                      65 examples, 0 failures         1 suites in 7.1969s      (pid 12226 exit 0 )
    [ 2]                                      19 examples, 0 failures         1 suites in 7.3070s      (pid 12227 exit 0 )
    [ 3]                                      20 examples, 0 failures         1 suites in 11.5977s      (pid 12228 exit 0 )
    [ 4]                                      80 examples, 0 failures         3 suites in 11.5978s      (pid 12229 exit 0 )
    [ 5]                                     101 examples, 0 failures         3 suites in 11.5977s      (pid 12230 exit 0 )
    [ 6]                                      55 examples, 0 failures         5 suites in 11.5976s      (pid 12231 exit 0 )
    [ 7]                                      49 examples, 0 failures         4 suites in 11.5975s      (pid 12232 exit 0 )
    [ 8]                                      57 examples, 0 failures         6 suites in 11.5974s      (pid 12233 exit 0 )
    [ 9]                                      31 examples, 0 failures         6 suites in 11.5973s      (pid 12234 exit 0 )
    [10]                                      46 examples, 0 failures         4 suites in 11.5972s      (pid 12235 exit 0 )
    [11]                                      55 examples, 0 failures         5 suites in 11.5971s      (pid 12236 exit 0 )
    [12]                                      57 examples, 0 failures         6 suites in 11.5968s      (pid 12237 exit 0 )
    [13]                                      70 examples, 0 failures         6 suites in 11.5967s      (pid 12238 exit 0 )
    [14]                                      57 examples, 0 failures         5 suites in 11.5962s      (pid 12239 exit 0 )
    [15]                                      47 examples, 0 failures         6 suites in 11.5951s      (pid 12240 exit 0 )
    [16]                                      56 examples, 0 failures         6 suites in 11.5933s      (pid 12241 exit 0 )
```

It can be up to 5x faster depending on the environment.
@koic
Copy link
Contributor Author

koic commented May 7, 2023

Here is the result from CI:

Before (27.87 seconds)

Run bundle exec rspec -w
(snip)

Finished in 27.87 seconds (files took 0.63688 seconds to load)
875 examples, 0 failures

https://github.com/octokit/octokit.rb/actions/runs/4725471255/jobs/8383924167

After (12.2976 seconds)

Run bundle exec rspec-queue -w
(snip)

==> Summary (2 workers in 12.2976s)

    [ 1]                                     634 examples, 0 failures        53 suites in 12.2939s      (pid 2000 exit 0 )
    [ 2]                                     241 examples, 0 failures        17 suites in 12.2940s      (pid 2001 exit 0 )

https://github.com/octokit/octokit.rb/actions/runs/4906312889/jobs/8760680945?pr=1570

On 2 CPUs GitHub Actions, it's around 2x faster.

@nickfloyd nickfloyd added the Type: Feature New feature or request label May 12, 2023
Copy link
Contributor

@nickfloyd nickfloyd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid change @koic, thanks for doing this ❤️

@nickfloyd nickfloyd merged commit d5eec6e into octokit:main May 12, 2023
13 checks passed
@koic koic deleted the use_test_queue branch May 13, 2023 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Feature New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants