question: getting started with collate & refuse_coverage_drop on CI

[madisonsites]

Hey Howdy Hey!

I'm super appreciative of the refuse_coverage_drop option, but have two issues:

1. I cannot seem to get the last_run file generated.
2. Even if I could, is there a way to change the coverage path only when reading the last_run_path so I don't have to give our CI a write access token to commit these?

Missing last_run file

I suspect this is coming from the fact that we are collating. I generally used this tutorial from Knapsack Pro to help knapsack, simplecov, and semaphore get along nicely through these steps...

1. Knapsack parallelizes a full rspec run (bundle exec rake knapsack_pro:queue:rspec['--no-color --format progress --format RspecJunitFormatter --out tmp/rspec-junit/rspec.xml']) across 6 jobs. After each job completes (not all), this is run to push up the results into a semaphore artifact (will be used in the next step):

#!/bin/bash
set -eu

if [ -d "tmp/rspec-junit" ]
then
  echo "Pushing rspec junit results"
  artifact push job tmp/rspec-junit --destination semaphore/test-results/
fi

if [ -d "coverage" ]
then
  echo "Pushing simplecov results"
  tar czf coverage_${SEMAPHORE_JOB_INDEX}.tgz -C coverage .
  artifact push workflow coverage_${SEMAPHORE_JOB_INDEX}.tgz
fi

2. Once all of the rspec jobs are complete, a temporary directory is made from the coverage files that were pushed to the workflow's artifacts:

#!/bin/bash
set -euo pipefail

for i in $(eval echo "{1..$SEMAPHORE_RAILS_JOB_COUNT}"); do
  artifact pull workflow coverage_$i.tgz;
  mkdir coverage_$i
  tar -xzf coverage_$i.tgz -C coverage_$i
done

3. A rake task is run that collates all those files together to output a total_coverage directory. This is where I attempted to add refuse_coverage_drop as I want to compare across everything together, not within the individual runs themselves (what is in each of these changes all the time and I presumed this would interfere with things).

None of these attempts generated a last_run file.

attempt one (setting refuse_coverage_drop IN the collate block:

  task report: :environment do
    require 'simplecov'
    require 'simplecov-json'

    SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
      SimpleCov::Formatter::HTMLFormatter,
      SimpleCov::Formatter::JSONFormatter,
    ])
    SimpleCov.collate Dir['coverage_*/.resultset.json'] do
      refuse_coverage_drop
    end
  end

attempt two (setting refuse_coverage_drop AFTER the collate block):

  task report: :environment do
    require 'simplecov'
    require 'simplecov-json'

    SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
      SimpleCov::Formatter::HTMLFormatter,
      SimpleCov::Formatter::JSONFormatter,
    ])
    SimpleCov.collate Dir['coverage_*/.resultset.json']
    SimpleCov.refuse_coverage_drop
  end

attempt three (setting refuse_coverage_drop BEFORE the collate block):

  task report: :environment do
    require 'simplecov'
    require 'simplecov-json'

    SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
      SimpleCov::Formatter::HTMLFormatter,
      SimpleCov::Formatter::JSONFormatter,
    ])
    SimpleCov.refuse_coverage_drop
    SimpleCov.collate Dir['coverage_*/.resultset.json']
  end

attempt four (setting refuse_coverage_drop in .simplecov):

require 'knapsack_pro'

SimpleCov.start 'rails' do
  add_filter [
    "lib/analytics/",
    "lib/data_migrations/",
    "lib/data_restoration/",
    "lib/tasks/turnstone.rake",
    "spec/",
  ]

  refuse_coverage_drop
end

KnapsackPro::Hooks::Queue.before_queue do
  SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
end

The artifacts always output as such:

For each of the above attempts, I checked both...

total_coverage

coverage_1

Reading/writing last_run in CI

More of a general "how is everyone else doing this?" than anything. I'd liked to dynamically generate the last_run file as part of our CI workflows as a step in the deploy process (meaning only when the branch being tested is master) without giving write access to Semaphore (so it could commit the file each time).

It seems plausible that if I can figure out the first issue, then I can easily push the last_run file to our project's artifacts on Semaphore (overwriting when it is the master branch only).

I'm more confused about reading the last run when checking the differences for maximum coverage drop.

I see that LastRun uses the coverage_path to fetch the file. I'm aware that coverage_path is configurable, but that's an all-or-nothing when I only want to "look elsewhere" for the last_run file - outside the standard coverage_path and instead to the last_run file in project's artifact.

simplecov (0.21.2)
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin19]
rails (~> 5.1)

spec/spec_helper.rb if it is helpful:

# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require 'simplecov'
require File.expand_path("../../config/environment", __FILE__)
require 'knapsack_pro'
require 'rspec/rails'
require 'shoulda/matchers'
require 'vcr'
require 'stripe_mock'
require 'sidekiq/testing'

KnapsackPro::Adapters::RSpecAdapter.bind

[davidwparker]

@madisonsites - were you able to make progress on this? I'm running into the same issue.

[ArturT]

I would expect that refuse_coverage_drop should generate "coverage/.last_run.json" file as stated here https://coderwall.com/p/ki2sdq/simplecov-prevent-coverage-drop

require 'knapsack_pro'

SimpleCov.start 'rails' do
  add_filter [
    "lib/analytics/",
    "lib/data_migrations/",
    "lib/data_restoration/",
    "lib/tasks/turnstone.rake",
    "spec/",
  ]

  refuse_coverage_drop # this line here I guess should generate  "coverage/.last_run.json" file.
end

KnapsackPro::Hooks::Queue.before_queue do
  SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
end

I'm not sure thou if "coverage/.last_run.json" file would contain combined tests for each test suite run on a given CI node index when running in Knapsack Pro Queue Mode.
In the Queue Mode, we run multiple times set of tests fetched from Queue API (there are multiple test suites executed from RSpec perspective). More about the Queue Mode here.

This means RSpec hooks like after/before :suite are called multiple times. I'm not sure if this is relevant from the simplecov perspective. I don't know how it works internally.

I see @madisonsites uses also JUnit formatter. This is how it can be done in Queue Mode:
https://knapsackpro.com/faq/question/how-to-use-junit-formatter
We use hook KnapsackPro::Hooks::Queue.after_subset_queue to move xml file to a different location. Not sure if a similar approach could be useful for simplecov.

[hannahramadan]

@madisonsites @davidwparker Did either of you find a solution to this?

[madisonsites]

@hannahramadan for the life of me, I can't remember how I got this working. I think it was something a bit silly - like having something in .gitignore or something I added or removed that I had forgotten to change. I just remember looking at all my changes together in the pull request and either realizing it or playing with something that got it working.

I'm so sorry that past me failed to document and follow up ☹️

[hannahramadan]

Thanks for responding @madisonsites :) It's working now! Here's the rake task for those interested:

namespace :coverage do
  desc "Collates all result sets generated by the different test runners"
  task :report do
    require 'simplecov'
    require 'simplecov_json_formatter'
    SimpleCov.collate Dir["coverage*/.resultset.json"] do
      formatter SimpleCov::Formatter::MultiFormatter.new([
        SimpleCov::Formatter::JSONFormatter,
        SimpleCov::Formatter::HTMLFormatter
      ])
      refuse_coverage_drop
    end
  end
end

Needed to fiddle with our CI results path, which ended up being resultPath: lib/coverage/.last_run.json.