Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix simplecov rcov calculation #226

Merged
merged 9 commits into from
May 4, 2014
Merged

Conversation

bf4
Copy link
Member

@bf4 bf4 commented May 2, 2014

Turns out #222 didn't really fix #153

Basically,

  • SimpleCov divides lines into three states: covered, missed, and ignored.
  • RCov divides lines into two states: covered, missed

The RCov formatter, being faithful to the RCov format, didn't annotate 'missed' lines, which #222 did not address, and hence #153 was not fixed.

In this PR, I introduce a syntax '--' to the RCov report generator that we then parse as a 'missed' line. And all is good in the world. Now, our RCov output is no longer really RCov, and in fact true RCov docs may be processed incorrectly as RCov covered lines were identified by the absence of '!!'. See logic in RCovFormatCoverage#assemble_files

All of this leads to the need to make a stronger break from pretending to use RCov, and to, at least

  1. Output the simplecov data in _data/ as "#{MetricFu.report_id}_coverage.json" or some-such that
  2. Is automatically picked up by metric_fu when running a report with that id. (The report_id is really just YYYYMMDD)

That is, to no longer output an 'rcov.txt' to 'coverage/rcov/rcov.txt'

I'd love to change the metric namespace to TestCoverage.. but that's more work to do without breaking the existing contracted behavior.

It also turns out that the JRuby Coverage module produces different results from MRI. See

@bf4
Copy link
Member Author

bf4 commented May 2, 2014

JRuby failed with

home/travis/build/metricfu/metric_fu/gemfiles/vendor/bundle/jruby/1.9/gems/simplecov-0.8.2/lib/simplecov.rb:31 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag

SimpleCov::Formatter::MetricFu calculates the same coverage from an RCov report as from SimpleCov
     Failure/Error: expect(source_file.coverage.count).to eq(covered_lines_from_rcov_text.count)

       expected: 11
            got: 8

       (compared using ==)
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/rcov/simplecov_formatter_spec.rb:52:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'

::MetricFu::Calculate.integer_percent(numerator, denominator)
end

def self.percent_run(lines)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be refactored, perhaps much of it go into ::MetricFu::Calculate. It's certainly pretty noisy code in here.

bf4 added 6 commits May 2, 2014 14:16
Otherwise, metric_fu code will be loaded before SimpleCov turns
Coverage on and it will not be tracked
Will break old RCov behavior where a file is that is loaded, but
no lines executed, will have a coverage percentage.  Under the
new rubric, it will be 0.0, which really is more correct.

Since we've already removed support for running via RCov, this
won't be a problem.
bf4 added a commit that referenced this pull request May 2, 2014
 see #226
     jruby/jruby#1196
     https://jira.codehaus.org/browse/JRUBY-6106
     simplecov-ruby/simplecov#86

Travis ENV modified per
http://docs.travis-ci.com/user/ci-environment/#Environment-variables

@enebo https://twitter.com/tom_enebo/status/462289024107814914
recommended JRUBY_OPTS="-Xcli.debug=true"
but that didn't work and caused other failures
JRUBY_OPTS="--debug" Still produced the below error, as did
JRUBY_OPTS="--debug -Xcli.debug=true"

Failure message was
home/travis/build/metricfu/metric_fu/gemfiles/vendor/bundle/jruby/1.9/gems/simplecov-0.8.2/lib/simplecov.rb:31 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag

SimpleCov::Formatter::MetricFu calculates the same coverage from an RCov report as from SimpleCov
     Failure/Error: expect(source_file.coverage.count).to eq(covered_lines_from_rcov_text.count)

       expected: 11
            got: 8

       (compared using ==)
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/rcov/simplecov_formatter_spec.rb:52:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
bf4 added 3 commits May 4, 2014 10:58
 see #226
     jruby/jruby#1196
     https://jira.codehaus.org/browse/JRUBY-6106
     simplecov-ruby/simplecov#86

Travis ENV modified per
http://docs.travis-ci.com/user/ci-environment/#Environment-variables

@enebo https://twitter.com/tom_enebo/status/462289024107814914
recommended JRUBY_OPTS="-Xcli.debug=true"
but that didn't work and caused other failures
JRUBY_OPTS="--debug" Still produced the below error, as did
JRUBY_OPTS="--debug -Xcli.debug=true"

Failure message was
home/travis/build/metricfu/metric_fu/gemfiles/vendor/bundle/jruby/1.9/gems/simplecov-0.8.2/lib/simplecov.rb:31 warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag

SimpleCov::Formatter::MetricFu calculates the same coverage from an RCov report as from SimpleCov
     Failure/Error: expect(source_file.coverage.count).to eq(covered_lines_from_rcov_text.count)

       expected: 11
            got: 8

       (compared using ==)
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/rcov/simplecov_formatter_spec.rb:52:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
And we only turned that on for Coverage which
1) Didn't fix 87ffbac2162e0142fc4
2) We're not testing anymore for JRuby

https://travis-ci.org/metricfu/metric_fu/jobs/24305005

  1) MetricFu::ReekGenerator emit includes config file pattern into reek parameters when specified
     Failure/Error: reek.emit
     (anonymous error class):
       execution expired
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:35:in ``'
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:49:in ``'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:126:in `reek_version'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:118:in `turn_off_color'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:98:in `cli_options'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:16:in `emit'
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/reek/generator_spec.rb:17:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  2) MetricFu::ReekGenerator emit disables lines numbers from reek output
     Failure/Error: reek.emit
     (anonymous error class):
       execution expired
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:35:in ``'
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:49:in ``'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:126:in `reek_version'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:118:in `turn_off_color'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:98:in `cli_options'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:16:in `emit'
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/reek/generator_spec.rb:34:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  3) MetricFu::ReekGenerator emit includes files to analyze into reek parameters
     Failure/Error: reek.emit
     (anonymous error class):
       execution expired
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:35:in ``'
     # file:/home/travis/.rvm/rubies/jruby-1.7.11-d19/lib/jruby.jar!/jruby/kernel/jruby/process_manager.rb:49:in ``'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:126:in `reek_version'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:118:in `turn_off_color'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:98:in `cli_options'
     # /home/travis/build/metricfu/metric_fu/lib/metric_fu/metrics/reek/generator.rb:16:in `emit'
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/metrics/reek/generator_spec.rb:39:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  4) MetricFu given configured metrics, when run creates a data yaml file
     Failure/Error: expect { metric_fu }.to create_file("#{data_directory}/#{MetricFu.report_id}.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/_data/20140502.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:56:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  5) MetricFu given configured metrics, when run creates a report html file
     Failure/Error: expect { metric_fu }.to create_file("#{output_directory}/index.html")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/output/index.html" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:60:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  6) MetricFu given configured metrics, when run creates a report yaml file
     Failure/Error: expect { metric_fu }.to create_file("#{base_directory}/report.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/report.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:52:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  7) MetricFu given configured metrics, when run with command line specified out only outputs to the specified path
     Failure/Error: expect { metric_fu "--out customdir --no-open"}.to create_file("#{base_directory}/customdir/index.html")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/customdir/index.html" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:147:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  8) MetricFu given configured metrics, when run with configured and command line formatter outputs using command line formatter
     Failure/Error: expect { metric_fu "--format yaml"}.to create_file("#{base_directory}/report.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/report.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:104:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  9) MetricFu given configured metrics, when run with command line formatter outputs using command line formatter
     Failure/Error: expect { metric_fu "--format yaml"}.to create_file("#{base_directory}/report.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/report.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:86:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  10) MetricFu given configured metrics, when run with command line specified formatter + out outputs to the specified path
     Failure/Error: expect { metric_fu "--format yaml --out customreport.yml"}.to create_file("#{base_directory}/customreport.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/customreport.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:136:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  11) MetricFu given configured metrics, when run with configured formatter outputs using configured formatter
     Failure/Error: }.to create_file("#{base_directory}/report.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/report.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:70:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
  12) MetricFu given configured metrics, when run with configured specified out outputs using configured out
     Failure/Error: }.to create_file("#{base_directory}/customreport.yml")
       The file "/home/travis/build/metricfu/metric_fu/tmp/metric_fu/test/customreport.yml" was not created
     # /home/travis/build/metricfu/metric_fu/spec/metric_fu/run_spec.rb:120:in `(root)'
     # /home/travis/build/metricfu/metric_fu/spec/support/timeout.rb:6:in `(root)'
bf4 added a commit that referenced this pull request May 4, 2014
[Feature/Fix] Fix simplecov rcov calculation
@bf4 bf4 merged commit 68d51ec into master May 4, 2014
@bf4 bf4 deleted the fix_simplecov_rcov_calculation branch May 4, 2014 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Code coverage is different for text and html rcov formatter in metric_fu
1 participant