Decreased coverage with snapshot #4404

donv opened this Issue Dec 21, 2016 · 8 comments


None yet

4 participants

donv commented Dec 21, 2016

jruby (2.3.1) 2016-12-21 946befd Java HotSpot(TM) 64-Bit Server VM 25.112-b16 on 1.8.0_112-b16 +jit [darwin-x86_64]

When testing I get a very low coverage percentage using simplecov and JRuby

I get the expected coverage percentage of 86% when using any MRI version since 2.1 and JRuby,, and

To reproduce:

git clone
cd capybara-screenshot-diff
rake test
@donv donv added the regression label Dec 21, 2016
enebo commented Dec 21, 2016

@donv Not seeing this on macos at least:

11 files inspected, no offenses detected
Started with run options --seed 42427

  11/11: [=================================] 100% Time: 00:00:02, Time: 00:00:02

Finished in 2.32263s
11 tests, 25 assertions, 0 failures, 0 errors, 0 skips

Java was:

Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)
PragTob commented Dec 25, 2016 edited

Ha! Fun thing... I just talked to @colszowka (creator of simplecov) a while back about JRuby support etc. and he asked if I'd be willing to join the team and be the JRuby support person and officially support JRuby. We both agreed that chances of this requiring any work/of any regressions would be super low 😄

Anyhow, the simplecov specs also fail on jruby-head now and it seems to be exactly the integration tests with different reported coverage values.

I can also confirm the decreased coverage on the repo @donv mentioned:

tobi@comfy ~/github/capybara-screenshot-diff $ GEM_HOME=vendor JRUBY_OPTS=--debug ../jruby/bin/jruby -S vendor/bin/bundle exec rake test
Started with run options --seed 61202

  11/11: [=================================] 100% Time: 00:00:00, Time: 00:00:00

Finished in 0.77429s
11 tests, 25 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Unit Tests to /home/tobi/github/capybara-screenshot-diff/coverage. 11333 / 26516 LOC (42.74%) covered.
Coverage (42.74%) is below the expected minimum coverage (86.00%).
rake aborted!
Command failed with status (2)
#lots of complaining

@enebo there is a switch in the repo that checks if JRuby has --debug enabled and doesn't run them unless the switch is set --> with it it runs fine :)

Happy to help in any way to debug this and get it back working for as this is a critical bug/regression for me personally :D

Anyhow, happy holidays to all of you and thanks for all you do 🌟

enebo commented Dec 27, 2016

@PragTob ah heh...I definitely did not run with --debug so that is why I did not see the issue. Thanks for pointing that out.

@enebo enebo added this to the JRuby milestone Dec 27, 2016
enebo commented Jan 3, 2017

Bisected this to f9d704c ...

kares commented Jan 4, 2017

so JRuby must potentially dead-locks in a debugging session so that simplecov reports correct coverage ...
maybe there's a remedy of having simplecov improved or not - imagine it relies on an at_axit handler

PragTob commented Jan 4, 2017

at_exit seems to be used her - I'm not in the simplecov code base that much yet and time is somewhat limited, if you gave me general directions of what'd be the problem/potential fix I could try fixing that :)

also @enebo thanks for bisecting and all the work :) 💚

kares commented Jan 5, 2017 edited

@PragTob @enebo think its fine to revert and deal with the dead-lock later (might try a fix in ruby-debug)

kares commented Jan 5, 2017

@enebo see you already reverted on master at 57ba4fd ... thanks, closing

@kares kares closed this Jan 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment