Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev-cmd/tests: Add profile option #16441

Merged

Conversation

apainintheneck
Copy link
Contributor

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

This will allow us to more easily measure test performance. The only downside here is that we can't use it with parallel rspec because it will show the n slowest tests for each parallel rspec run which is not what we want.

Valid Profile Count

$ brew tests --only=utils/ --profile=5
Randomized with seed 41310
Run options: exclude {:needs_linux=>true, :needs_network=>true, :needs_ci=>true, :needs_svn=>true}

Randomized with seed 41310
............................................................................................................................................................................................................................................................................................................................................................

Top 5 slowest examples (6.4 seconds, 18.2% of total time):
  Utils::Svn::available? returns svn version if svn available
    1.86 seconds ./test/utils/svn_spec.rb:11
  Utils::Git#file_at_commit returns empty when file doesn't exist
    1.31 seconds ./test/utils/git_spec.rb:91
  Utils::Git#last_revision_commit_of_files when before_commit is not nil gives last revision commit
    1.13 seconds ./test/utils/git_spec.rb:107
  Utils::Git#cherry_pick! aborts when cherry picking an existing hash
    1.05 seconds ./test/utils/git_spec.rb:62
  Utils::Git::path returns path of git when git is available
    1.04 seconds ./test/utils/git_spec.rb:150

Top 5 slowest example groups:
  Utils::Git
    1.01 seconds average (22.2 seconds / 22 examples) ./test/utils/git_spec.rb:5
  Utils::Svn
    0.52233 seconds average (2.09 seconds / 4 examples) ./test/utils/svn_spec.rb:5
  Utils
    0.30937 seconds average (6.19 seconds / 20 examples) ./test/utils/git_repository_spec.rb:5
  Utils
    0.15552 seconds average (0.62206 seconds / 4 examples) ./test/utils/ruby_check_version_script_spec.rb:3
  Utils::Bottles
    0.10359 seconds average (0.10359 seconds / 1 example) ./test/utils/bottles/bottles_spec.rb:5

Finished in 35.2 seconds (files took 4.55 seconds to load)
348 examples, 0 failures

Randomized with seed 41310

Invalid Profile Count

In this case, it prints a warning and defaults to 10.

$ brew tests --only=utils/ --profile=sdklfjs
Randomized with seed 18207
WARNING: Non integer specified as profile count, separate your path from options with -- e.g. `rspec --profile -- sdklfjs`.
Run options: exclude {:needs_linux=>true, :needs_network=>true, :needs_ci=>true, :needs_svn=>true}

Randomized with seed 18207
............................................................................................................................................................................................................................................................................................................................................................

Top 10 slowest examples (10.36 seconds, 31.0% of total time):
  Utils::Git#cherry_pick! aborts when cherry picking an existing hash
    1.08 seconds ./test/utils/git_spec.rb:62
  Utils::Git::ensure_installed! when git is not already available raises error if can't install git
    1.05 seconds ./test/utils/git_spec.rb:181
  Utils::Git::remote_exists? when git is available returns false when git remote does not exist
    1.05 seconds ./test/utils/git_spec.rb:219
  Utils::Git#last_revision_of_file returns last revision of file
    1.05 seconds ./test/utils/git_spec.rb:118
  Utils::Git#file_at_commit returns empty when file doesn't exist
    1.05 seconds ./test/utils/git_spec.rb:91
  Utils::Git::path returns path of git when git is available
    1.04 seconds ./test/utils/git_spec.rb:150
  Utils::Git#last_revision_of_file returns last revision of file based on before_commit
    1.04 seconds ./test/utils/git_spec.rb:125
  Utils::Git::ensure_installed! returns nil if git already available
    1 seconds ./test/utils/git_spec.rb:167
  Utils::Git#last_revision_commit_of_files when before_commit is nil gives last revision commit
    1 seconds ./test/utils/git_spec.rb:99
  Utils::Git#cherry_pick! can cherry pick a commit
    0.99866 seconds ./test/utils/git_spec.rb:58

Top 10 slowest example groups:
  Utils::Git
    1 seconds average (22.05 seconds / 22 examples) ./test/utils/git_spec.rb:5
  Utils
    0.31344 seconds average (6.27 seconds / 20 examples) ./test/utils/git_repository_spec.rb:5
  Utils
    0.15897 seconds average (0.63588 seconds / 4 examples) ./test/utils/ruby_check_version_script_spec.rb:3
  Utils::Bottles
    0.10109 seconds average (0.10109 seconds / 1 example) ./test/utils/bottles/bottles_spec.rb:5
  Utils::Svn
    0.09407 seconds average (0.37628 seconds / 4 examples) ./test/utils/svn_spec.rb:5
  Utils::Bottles::Collector
    0.05015 seconds average (0.30092 seconds / 6 examples) ./test/utils/bottles/collector_spec.rb:5
  Utils
    0.03146 seconds average (0.06292 seconds / 2 examples) ./test/utils/fork_spec.rb:5
  Utils::Gzip
    0.0224 seconds average (0.112 seconds / 5 examples) ./test/utils/gzip_spec.rb:5
  Utils
    0.02039 seconds average (0.2243 seconds / 11 examples) ./test/utils/popen_spec.rb:5
  Utils::Tar
    0.01971 seconds average (0.09857 seconds / 5 examples) ./test/utils/tar_spec.rb:5

Finished in 33.45 seconds (files took 3.95 seconds to load)
348 examples, 0 failures

Randomized with seed 18207

This will allow us to more easily measure test performance. The only
downside here is that we can't use it with parallel rspec because
it will show the n slowest tests for each parallel rspec run which
is not what we want.
Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this, thanks!

@MikeMcQuaid MikeMcQuaid merged commit 9cdfcd3 into Homebrew:master Jan 8, 2024
25 checks passed
@github-actions github-actions bot added the outdated PR was locked due to age label Feb 8, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 8, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated PR was locked due to age
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants