Need to wait until all tests suite complete before failing because of minimum_coverage

[ronwsmith]

I have three test suites and use the use_merging flag to combine coverage, but the report is generated after each test suite completes as well.

As my test suites are running, the coverage goes up like this:
Unit tests: 84%
Functional tests: 91%
Features: 95%

Currently, I have to have minimum_coverage set to 84%, but I'd like to be able to set it to 95% since that's our total combined coverage.

Is there any way to have simplecov wait to make a run as failed based on minimum_coverage until after the final test suite completes?

[jshraibman-mdsol]

Do something like this:

  running_in_parallel = ENV['PARALLEL_TEST_GROUPS'] && ENV['TEST_ENV_NUMBER']

  SimpleCov.at_exit do
    result = SimpleCov.result
    # Count the number of commas in the command name to figure out how many result groups were combined into this
    # result.  We only want to do the processing below when the last parallel worker finishes.
    if !running_in_parallel || ENV['PARALLEL_TEST_GROUPS'].to_i <= result.command_name.scan(/,/).size + 1
      result.format!                                                 # Print out coverage result
      exit(1) if result.covered_percent < 100.0
    end
  end

[ronwsmith]

I'm not using parallel_tests across suites (yet), but your suggestion is definitely giving me ideas on how to implement a workaround for my issue. Thanks!

[colszowka]

It would be cool to have some standardized solution to this kind of thing built into simplecov, without having to resort to defining custom at_exit behaviour.

@ronwsmith What did you end up doing?

[ronwsmith]

Here's what I did:

SimpleCov.at_exit do
  result = SimpleCov.result
  result.format!
  simplecov_test_suites = ['minitest', 'rspec']
  parallel_offset       = ENV['PARALLEL_TEST_GROUPS'] ? ENV['PARALLEL_TEST_GROUPS'].to_i - 1 : 0
  minimum_coverage      = 97.25
  # Count the number of commas in the command name to figure out how many result groups were combined into this result
  if result.command_name.scan(/,/).size + 1 >= simplecov_test_suites.size + (parallel_offset * 2) # two parallel suites
    # We only want to enforce minimum coverage after all test suites finish
    if result.covered_percent.round(2) < minimum_coverage
      puts "Coverage percentage has dropped below the minimum threshold of #{minimum_coverage}%"
      exit(1)
    end
  end
end

I also have these defined in the SimpleCov.start since I'm running two different test suites back-to-back:

  # Merge coverage reports for multiple test types
  use_merging true
  # Merge results within the last 60 minutes.  If the tests take longer, this must be extended
  merge_timeout 3600

Edit: Note, this assumes I'm always running in parallel, so for an official SimpleCov feature, we should add in what @jshraibman-mdsol recommended above.

[bf4]

@colszowka Would you mind enabling the wiki so people can share solutions such as these?