Skip to content

Commit

Permalink
Enhance test suite to emit JUnit XML test reports
Browse files Browse the repository at this point in the history
In preparation for detecting flaky tests with BuildPulse, this commit
sets up the rspec_junit_formatter gem to output JUnit XML reports of the
test suite, which is the format used by BuildPulse and various other
tooling that interprets test results.

Because the test suite uses the parallel_tests gem, this commit
incorporates some related changes to make all the parallel_tests gem and
the rspec_junit_formatter gem to cooperate with each other.

rspec_junit_formatter writes everything to a single XML file. That works
fine when there's only one process writing to the file. By default,
whatever process finishes last will write to the file and clobber the
output of all the other processes that wrote to the file. 馃檲

To prevent this issue, the parallel_tests wiki recommends adding a
`.rspec_parallel` file to specify its RSpec options
(https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber),
then the project can specify different files for each process to write
to like so:

  --format RspecJunitFormatter
  --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml

However, prior to this commit, the Homebrew/brew test suite specified
its RSpec options via the command line. Unfortunately though, there's no
way (AFAICT) to set the equivalent of these options via the command
line:

  --format RspecJunitFormatter
  --out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml

So, we need to use a `.rspec_parallel` file to specify these options 鈽濓笍.

However, it appears that RSpec allows you to specify formatters _either_
in an options file (like `.rspec_parallel`) _or_ via command-line args.
But if you specify any formatters via command-line args, then all
formatters in the options file are ignored.  (I suspect that's somehow
related to this bit of code in rspec-core:
https://github.com/rspec/rspec-core/blob/v3.10.0/lib/rspec/core/configuration_options.rb#L64.)

With that in mind, in order to have the RspecJunitFormatter configured
 in `.rspec_parallel`, we need to move the other formatters into
 `.rpsec_parallel` as well, instead of passing them as command-line
 args. Therefore, this commit moves all the formatters into a
 `.rspec_parallel` file.
  • Loading branch information
jasonrudolph committed Jun 21, 2021
1 parent 09f7bc2 commit e163eb8
Show file tree
Hide file tree
Showing 5 changed files with 13 additions and 5 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Expand Up @@ -14,6 +14,7 @@
/Library/Homebrew/test/.gem
/Library/Homebrew/test/.subversion
/Library/Homebrew/test/coverage
/Library/Homebrew/test/junit
/Library/Homebrew/test/fs_leak_log
/Library/Homebrew/vendor/portable-ruby
/Library/Taps
Expand Down Expand Up @@ -133,6 +134,7 @@
**/vendor/bundle/ruby/*/gems/rspec-*/
**/vendor/bundle/ruby/*/gems/rspec-core-*/
**/vendor/bundle/ruby/*/gems/rspec-expectations-*/
**/vendor/bundle/ruby/*/gems/rspec_junit_formatter-*/
**/vendor/bundle/ruby/*/gems/rspec-its-*/
**/vendor/bundle/ruby/*/gems/rspec-mocks-*/
**/vendor/bundle/ruby/*/gems/rspec-retry-*/
Expand Down
6 changes: 6 additions & 0 deletions Library/Homebrew/.rspec_parallel
@@ -0,0 +1,6 @@
--format NoSeedProgressFormatter
--format ParallelTests::RSpec::RuntimeLogger
--out <%= ENV["PARALLEL_RSPEC_LOG_PATH"] %>
--format RspecJunitFormatter
--out test/junit/rspec<%= ENV["TEST_ENV_NUMBER"] %>.xml
<%= "--format RSpec::Github::Formatter" if ENV["GITHUB_ACTIONS"] %>
1 change: 1 addition & 0 deletions Library/Homebrew/Gemfile
Expand Up @@ -12,6 +12,7 @@ gem "ronn", require: false
gem "rspec", require: false
gem "rspec-github", require: false
gem "rspec-its", require: false
gem "rspec_junit_formatter", require: false
gem "rspec-retry", require: false
gem "rspec-wait", require: false
gem "rubocop", require: false
Expand Down
3 changes: 3 additions & 0 deletions Library/Homebrew/Gemfile.lock
Expand Up @@ -109,6 +109,8 @@ GEM
rspec-support (3.10.2)
rspec-wait (0.0.9)
rspec (>= 3, < 4)
rspec_junit_formatter (0.4.1)
rspec-core (>= 2, < 4, != 2.12.0)
rubocop (1.17.0)
parallel (~> 1.10)
parser (>= 3.0.0.0)
Expand Down Expand Up @@ -194,6 +196,7 @@ DEPENDENCIES
rspec-retry
rspec-sorbet
rspec-wait
rspec_junit_formatter
rubocop
rubocop-ast
rubocop-performance
Expand Down
6 changes: 1 addition & 5 deletions Library/Homebrew/dev-cmd/tests.rb
Expand Up @@ -111,6 +111,7 @@ def tests
else
"#{HOMEBREW_CACHE}/#{parallel_rspec_log_name}"
end
ENV["PARALLEL_RSPEC_LOG_PATH"] = parallel_rspec_log_path

parallel_args = if ENV["CI"]
%W[
Expand All @@ -133,13 +134,8 @@ def tests
--seed #{seed}
--color
--require spec_helper
--format NoSeedProgressFormatter
--format ParallelTests::RSpec::RuntimeLogger
--out #{parallel_rspec_log_path}
]

bundle_args << "--format" << "RSpec::Github::Formatter" if ENV["GITHUB_ACTIONS"]

unless OS.mac?
bundle_args << "--tag" << "~needs_macos" << "--tag" << "~cask"
files = files.reject { |p| p =~ %r{^test/(os/mac|cask)(/.*|_spec\.rb)$} }
Expand Down

0 comments on commit e163eb8

Please sign in to comment.