Cucumber tests failing in main branch

[tycooon]

For some reason cucumber tests are failing for me when I run them locally in main branch. For example, I currently have a clean local copy on commit 9cd3a46 and here is what I get:

$ be cucumber features/old_version_json.feature:12
Using the default profile...
....F

(::) failed steps (::)

Ambiguous match, found 3 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/old_version_json.feature:18:in `I should see a line coverage summary of 8/9'

Failing Scenarios:
cucumber features/old_version_json.feature:12 # Scenario: Running the tests succeeds

1 scenario (1 failed)
5 steps (1 failed, 4 passed)
0m6.321s

It looks like I have wrong version of simplecov-html, but it's locked to 0.12.2 in the Gemfile.
Build fails on Github as well: https://github.com/umbrellio/simplecov/runs/1063083286 (I just pushed the main branch to the forked repo).

Any ideas how to fix that?

[PragTob]

GRML... I have no idea right now and not too much OSS time on my hand. I'lll take a look hopefully at the latest on Saturday.

It's just fun because this is a classic case of "works on my machine" which isn't too surprising (-CI) as that's probably how it got there in the first place.

Just some... extra fun for me. Yay.

tobi@speedy:~/github/simplecov(main)$ git pull
Already up to date.
tobi@speedy:~/github/simplecov(main)$ be cucumber
Using the default profile...
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

76 scenarios (76 passed)
478 steps (478 passed)
1m31.649s

[PragTob]

Also works on my other machine...

tobi@comfy:~/github/simplecov(main)$ be rake
Running RuboCop...
Inspecting 99 files
...................................................................................................

99 files inspected, no offenses detected
/home/tobi/.asdf/installs/ruby/2.7.0/bin/ruby -I/home/tobi/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/lib:/home/tobi/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/rspec-support-3.9.3/lib /home/tobi/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/rspec-core-3.9.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb

Randomized with seed 2220
................................................................................................................................................................................................................................................................................................................................................
Non-app warnings written to tmp/warnings.txt



Finished in 7.41 seconds (files took 0.2317 seconds to load)
336 examples, 0 failures

Randomized with seed 2220

/home/tobi/.asdf/installs/ruby/2.7.0/bin/ruby -S bundle exec cucumber
Warning: the running version of Bundler (2.1.3) is older than the version that created the lockfile (2.1.4). We suggest you to upgrade to the version that created the lockfile by running `gem install bundler:2.1.4`.
Using the default profile...
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

76 scenarios (76 passed)
478 steps (478 passed)
2m57.244s

SO MUCH FUN

[PragTob]

Looks like the visibility checking isn't working on the CI and on your machine...

summary_text = find("#{extra_class}.t-#{coverage_type}-summary", visible: true).text

I checked that in the HTML for this one:

Ambiguous match, found 8 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/branch_coverage.feature:21:in `I should see a line coverage summary of 56/61'

It's visible exactly once but somehow on CI and your machines it doesn't care about that/do the visibility checking correctly.

So... I mean it could be race condition or maybe the css doesn't work as it should or what not...

Also tried a fresh clone, all works for me.

[PragTob]

deactivating the visible: true leads to exactly the same errors (unsurprisingly) just no idea why or how:

Ambiguous match, found 8 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/branch_coverage.feature:21:in `I should see a line coverage summary of 56/61'

[PragTob]

I'm starting to think that the JS is probably not working and I'm not sure why....

 Unable to find button "cboxClose" that is not disabled (Capybara::ElementNotFound)
./features/step_definitions/html_steps.rb:69:in `"I close the detailed view"'
features/pagination.feature:20:in `I close the detailed view'

cbxClose is from the modal that should open when a link is clicked... so I guess it's not clicked.

At this point maybe the assets are messed up or the the driver doesn't execute JS... I'm not sure. I confirmed google chrome is installed and apparition is configured... let's see.

I think I'm taking a break for now

[tycooon]

Just a little note: I see that CI uses ubuntu-latest and I guess it got updated on October, 22 (see https://wiki.ubuntu.com/Releases).

[PragTob]

Good thinking but October 22 hasn't passed yet and ubuntu-latest refers to 18.04 apparently even 🤔 tried explicit 20.04 and 18.04, nothing works. At this point rewriting the class selectors to be unique among visible/not visible might be the safest bet to long term resolve this but that's quite some work I'm not quite sure I got the head space for right now :|

no idea why when or how it happened... there were a couple of failures on main but those are different :(

[tycooon]

Oops, I somehow messed October and August :( Sorry about that.

[PragTob]

you're on the right track though, theoretically it could have been only some package update but which one, why or how is... fun to debug with me being unable to reproduce.

I could try the work mac for another data point...

[tycooon]

Yeah, must be some update. Maybe Chrome?

[PragTob]

Yeah... I can't quite figure it out. It seems somewhat most likely as that's one of the places where we differ as I'm not using chrome but chromium, difference shouldn't be big but still... :|

[PragTob]

well at least i can reproduce now on a machine... no time to fix right now
.

the puma error is local, but the local duplications also happen... yayy macs

tobi@Tobiass-MBP simplecov % bundle exec cucumber
Using the default profile...
....F-------...............................................................................................................................................................................F-..........F--------------......................................................................................F.....F.....F--..............F....F...................................................F....F....F.F--.............................................F....F....F.....................

(::) failed steps (::)

Ambiguous match, found 8 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/branch_coverage.feature:21:in `I should see a line coverage summary of 56/61'

expected to find css "#cboxContent" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/simplecov_steps.rb:75:in `"the overlay should be open"'
features/config_tracked_files_relevant_lines.feature:34:in `the overlay should be open'

expected not to find text "�" in "Generated 2020-09-08T10:17:59+02:00\nAll Files ( 55.56% covered at 0.56 hits/line )\n4 files in total.\n18 relevant lines, 10 lines covered and 8 lines missed. ( 55.56% )\nFile\t% covered\tLines\tRelevant Lines\tLines covered\tLines missed\tAvg. Hits / Line\nlib/euc_jp.rb\t75.00 %\t10\t4\t3\t1\t0.75\nlib/euc_jp_not_declared.rb\t75.00 %\t11\t4\t3\t1\t0.75\nlib/euc_jp_not_declared_tracked.rb\t0.00 %\t11\t6\t0\t6\t0.00\nlib/utf8.rb\t100.00 %\t9\t4\t4\t0\t1.00\nGenerated by simplecov v0.19.0 and simplecov-html v0.12.2\nusing RSpec\nlib/euc_jp.rb\n75.0% lines covered\n4 relevant lines. 3 lines covered and 1 lines missed.\n    \n      \n        \n          \n            \n            \n            \n            # encoding: EUC-JP\n          \n        \n      \n        \n          \n            \n            \n            \n            # frozen_string_literal: true\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n            class Fun2\n          \n        \n      \n        \n          \n            1\n            \n            \n              MSG = \"おはよう\"\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n              def no_moji\n          \n        \n      \n        \n          \n            \n            \n            \n                \"tada!\"\n          \n        \n      \n        \n          \n            \n            \n            \n              end\n          \n        \n      \n        \n          \n            \n            \n            \n            end\n          \n        \n      \n    \n  \nlib/euc_jp_not_declared.rb\n75.0% lines covered\n4 relevant lines. 3 lines covered and 1 lines missed.\n    \n      \n        \n          \n            \n            \n            \n            # frozen_string_literal: true\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n            class Fun3\n          \n        \n      \n        \n          \n            \n            \n            \n              # ruby actually breaks on the not decalred multibyte characters,\n          \n        \n      \n        \n          \n            \n            \n            \n              # which is nice I guess\n          \n        \n      \n        \n          \n            1\n            \n            \n              MSG = \"Something Else\"\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n              def no_moji\n          \n        \n      \n        \n          \n            \n            \n            \n                \"tada!\"\n          \n        \n      \n        \n          \n            \n            \n            \n              end\n          \n        \n      \n        \n          \n            \n            \n            \n            end\n          \n        \n      \n    \n  \nlib/euc_jp_not_declared_tracked.rb\n0.0% lines covered\n6 relevant lines. 0 lines covered and 6 lines missed.\n    \n      \n        \n          \n            \n            \n            \n            # frozen_string_literal: true\n          \n        \n      \n        \n          \n            \n            \n            \n            # Ruby can't even execute this file but it might be added\n          \n        \n      \n        \n          \n            \n            \n            \n            # via track_files or similar means and we still don't wanna crash!\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            \n            \n            \n            class NoDeclare\n          \n        \n      \n        \n          \n            \n            \n            \n              MSG = \"���Ϥ褦\"\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            \n            \n            \n              def ??\n          \n        \n      \n        \n          \n            \n            \n            \n                \"tada!\"\n          \n        \n      \n        \n          \n            \n            \n            \n              end\n          \n        \n      \n        \n          \n            \n            \n            \n            end\n          \n        \n      \n    \n  \nlib/utf8.rb\n100.0% lines covered\n4 relevant lines. 4 lines covered and 0 lines missed.\n    \n      \n        \n          \n            \n            \n            \n            # frozen_string_literal: true\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n            class Fun\n          \n        \n      \n        \n          \n            1\n            \n            \n              MSG = \"おはよう\"\n          \n        \n      \n        \n          \n            \n            \n            \n            \n          \n        \n      \n        \n          \n            1\n            \n            \n              def 🇯🇵\n          \n        \n      \n        \n          \n            1\n            \n            \n                \"tada!\"\n          \n        \n      \n        \n          \n            \n            \n            \n              end\n          \n        \n      \n        \n          \n            \n            \n            \n            end" (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/web_steps.rb:68:in `"{string} should not be visible"'
features/encoding.feature:19:in `"�" should not be visible'

Ambiguous match, found 3 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/old_version_json.feature:18:in `I should see a line coverage summary of 8/9'

Ambiguous match, found 3 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
features/old_version_json.feature:27:in `I should see a line coverage summary of 8/9'

Unable to find button "cboxClose" that is not disabled (Capybara::ElementNotFound)
./features/step_definitions/html_steps.rb:67:in `"I close the detailed view"'
features/pagination.feature:20:in `I close the detailed view'

Ambiguous match, found 10 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
./features/step_definitions/parallel_tests_steps.rb:23:in `"I should see the branch coverage results for the parallel tests project"'
features/parallel_tests.feature:44:in `I should see the branch coverage results for the parallel tests project'

Ambiguous match, found 10 elements matching visible css " .t-line-summary" (Capybara::Ambiguous)
./features/step_definitions/html_steps.rb:55:in `/^I should see a (.+) coverage summary of (\d+)\/(\d+)( for the file)?$/'
./features/step_definitions/parallel_tests_steps.rb:23:in `"I should see the branch coverage results for the parallel tests project"'
features/parallel_tests.feature:57:in `I should see the branch coverage results for the parallel tests project'


expected: 8
     got: 4

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/rspec_groups_and_filters_basic.feature:27:in `I should see the source files:'


expected: 4
     got: 1

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/rspec_groups_and_filters_complex.feature:38:in `I should see the source files:'


expected: 14
     got: 7

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/rspec_groups_using_filter_class.feature:35:in `I should see the source files:'

Expected `bundle` to succeed but got non-zero exit status and the following output:

Fetching gem metadata from https://rubygems.org/.............
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using rake 13.0.1
Fetching concurrent-ruby 1.1.6
Installing concurrent-ruby 1.1.6
Fetching i18n 1.8.2
Installing i18n 1.8.2
Fetching minitest 5.14.0
Installing minitest 5.14.0
Using thread_safe 0.3.6
Fetching tzinfo 1.2.6
Installing tzinfo 1.2.6
Fetching zeitwerk 2.2.2
Installing zeitwerk 2.2.2
Fetching activesupport 6.0.2.1
Installing activesupport 6.0.2.1
Using builder 3.2.4
Using erubi 1.9.0
Using mini_portile2 2.4.0
Fetching nokogiri 1.10.8
Installing nokogiri 1.10.8 with native extensions
Using rails-dom-testing 2.0.3
Using crass 1.0.6
Fetching loofah 2.4.0
Installing loofah 2.4.0
Using rails-html-sanitizer 1.3.0
Fetching actionview 6.0.2.1
Installing actionview 6.0.2.1
Fetching rack 2.2.2
Installing rack 2.2.2
Using rack-test 1.1.0
Fetching actionpack 6.0.2.1
Installing actionpack 6.0.2.1
Using nio4r 2.5.2
Fetching websocket-extensions 0.1.4
Installing websocket-extensions 0.1.4
Fetching websocket-driver 0.7.1
Installing websocket-driver 0.7.1 with native extensions
Fetching actioncable 6.0.2.1
Installing actioncable 6.0.2.1
Using globalid 0.4.2
Fetching activejob 6.0.2.1
Installing activejob 6.0.2.1
Fetching activemodel 6.0.2.1
Installing activemodel 6.0.2.1
Fetching activerecord 6.0.2.1
Installing activerecord 6.0.2.1
Fetching mimemagic 0.3.4
Installing mimemagic 0.3.4
Using marcel 0.3.3
Fetching activestorage 6.0.2.1
Installing activestorage 6.0.2.1
Using mini_mime 1.0.2
Using mail 2.7.1
Fetching actionmailbox 6.0.2.1
Installing actionmailbox 6.0.2.1
Fetching actionmailer 6.0.2.1
Installing actionmailer 6.0.2.1
Fetching actiontext 6.0.2.1
Installing actiontext 6.0.2.1
Fetching public_suffix 4.0.3
Installing public_suffix 4.0.3
Using addressable 2.7.0
Using bindex 0.8.1
Using msgpack 1.3.3
Fetching bootsnap 1.4.5
Installing bootsnap 1.4.5 with native extensions
Using bundler 2.1.4
Fetching byebug 11.1.1
Installing byebug 11.1.1 with native extensions
Fetching regexp_parser 1.7.0
Installing regexp_parser 1.7.0
Using xpath 3.2.0
Fetching capybara 3.31.0
Installing capybara 3.31.0
Fetching childprocess 3.0.0
Installing childprocess 3.0.0
Using diff-lcs 1.3
Using docile 1.3.2
Fetching jbuilder 2.10.0
Installing jbuilder 2.10.0
Fetching method_source 0.9.2
Installing method_source 0.9.2
Fetching puma 4.3.1
Installing puma 4.3.1 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/Users/tobi/.gem/ruby/2.6.6/gems/puma-4.3.1/ext/puma_http11
/opt/rubies/2.6.6/bin/ruby -I /opt/rubies/2.6.6/lib/ruby/2.6.0 -r
./siteconf20200908-96645-ov1jr8.rb extconf.rb
checking for BIO_read() in -lcrypto... yes
checking for SSL_CTX_new() in -lssl... yes
checking for openssl/bio.h... yes
checking for DTLS_method() in openssl/ssl.h... yes
checking for TLS_server_method() in openssl/ssl.h... yes
checking for SSL_CTX_set_min_proto_version in openssl/ssl.h... yes
creating Makefile

current directory: /Users/tobi/.gem/ruby/2.6.6/gems/puma-4.3.1/ext/puma_http11
make "DESTDIR=" clean

current directory: /Users/tobi/.gem/ruby/2.6.6/gems/puma-4.3.1/ext/puma_http11
make "DESTDIR="
compiling http11_parser.c
ext/puma_http11/http11_parser.c:44:18: warning: unused variable
'puma_parser_en_main' [-Wunused-const-variable]
static const int puma_parser_en_main = 1;
                 ^
1 warning generated.
compiling io_buffer.c
compiling mini_ssl.c
mini_ssl.c:145:7: warning: unused variable 'min' [-Wunused-variable]
  int min, ssl_options;
      ^
mini_ssl.c:299:40: warning: function 'raise_error' could be declared with
attribute 'noreturn' [-Wmissing-noreturn]
void raise_error(SSL* ssl, int result) {
                                       ^
2 warnings generated.
compiling puma_http11.c
puma_http11.c:203:22: error: implicitly declaring library function 'isspace'
with type 'int (int)' [-Werror,-Wimplicit-function-declaration]
  while (vlen > 0 && isspace(value[vlen - 1])) vlen--;
                     ^
puma_http11.c:203:22: note: include the header <ctype.h> or explicitly provide a
declaration for 'isspace'
1 error generated.
make: *** [puma_http11.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/tobi/.gem/ruby/2.6.6/gems/puma-4.3.1
for inspection.
Results logged to
/Users/tobi/.gem/ruby/2.6.6/extensions/x86_64-darwin-19/2.6.0-static/puma-4.3.1/gem_make.out

An error occurred while installing puma (4.3.1), and Bundler cannot continue.
Make sure that `gem install puma -v '4.3.1' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
  puma

 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/simplecov_steps.rb:80:in `"I install dependencies"'
features/rspec_rails.feature:13:in `I install dependencies'


expected: 8
     got: 4

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/test_unit_groups_and_filters_basic.feature:27:in `I should see the source files:'


expected: 3
     got: 1

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/test_unit_groups_and_filters_complex.feature:36:in `I should see the source files:'


expected: 12
     got: 6

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/test_unit_groups_using_filter_class.feature:35:in `I should see the source files:'

Failing Scenarios:
cucumber features/branch_coverage.feature:9 # Scenario:
cucumber features/config_tracked_files_relevant_lines.feature:10 # Scenario:
cucumber features/encoding.feature:12 # Scenario: Running tests produces coverage and it's mostly legible
cucumber features/old_version_json.feature:12 # Scenario: Running the tests succeeds
cucumber features/old_version_json.feature:20 # Scenario: Running the tests succeeds even with a current time stamp
cucumber features/pagination.feature:11 # Scenario:
cucumber features/parallel_tests.feature:34 # Scenario: Running the project with normal rspec and branch coverage
cucumber features/parallel_tests.feature:47 # Scenario: Running the project with normal rspec and branch coverage
cucumber features/rspec_groups_and_filters_basic.feature:10 # Scenario:
cucumber features/rspec_groups_and_filters_complex.feature:12 # Scenario:
cucumber features/rspec_groups_using_filter_class.feature:12 # Scenario:
cucumber features/rspec_rails.feature:12 # Scenario: Running bundle exec rspec produces a coverage report
cucumber features/test_unit_groups_and_filters_basic.feature:10 # Scenario:
cucumber features/test_unit_groups_and_filters_complex.feature:12 # Scenario:
cucumber features/test_unit_groups_using_filter_class.feature:12 # Scenario:

76 scenarios (15 failed, 61 passed)
478 steps (15 failed, 26 skipped, 437 passed)
3m4.926s

[PragTob]

grml I thought I checked issues but now upon re checking I find twalpole/apparition#64 which is the likely cause

[deivid-rodriguez]

Ha! I realize I might've run into the same thing in other repository, but I reverted the base docker image to the last working version and didn't dig further.

I guess an easy test we can do is revert b0d8826 and upgrade to the latest cuprite to see if the issues I had with it when migrating to Github Actions are now fixed. That would be the third switch we make because of issues with the alternative 😆.

[PragTob]

😁
yeah my action item for today/tomorrow or so is to try and switch to cuprite that's one of the recommended workarounds for the time being.

[tycooon]

Hi, a little update on the issue. Currently things got better, but I still get 1 test randomly failing in main branch: https://github.com/umbrellio/simplecov/runs/1148966104?check_suite_focus=true

Note that this is not specific to jruby since it also sometimes fail in MRI as well :(

[PragTob]

This is... very weird. Unless the error message is wrong:

 expected: 0
     got: 4

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/merging_test_unit_and_rspec.feature:42:in `I should see the source files:'


expected: 0
     got: 5

(compared using ==)
 (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/html_steps.rb:28:in `/^I should see the source files:$/'
features/skipping_code_blocks_manually.feature:60:in `I should see the source files:'

If you look at the code:

simplecov/features/step_definitions/html_steps.rb

Lines 24 to 32 in c1a7e7d

	Then /^I should see the source files:$/ do |table|
	expected_files = table.hashes
	available_source_files = all(".t-file", visible: true)
	expect(expected_files.length).to eq(available_source_files.count)
	include_branch_coverage = table.column_names.include?("branch coverage")
	# Find all filenames and their coverage present in coverage report
	files = available_source_files.map do |file_row|

this means that expected_files.length is 0 to my interpretation. And that definitely should not be flaky as that's just the hash from the cucumber scenario...

[tycooon]

Actually expectation is on the wrong side there, it should look like this:

expect(available_source_files.count).to eq(expected_files.length)

I already fixed that in my branch since it's really confusing :)

[PragTob]

ah right the thing in expect is the actual not the expected... yeah that makes more sense. I hope I can give it another look this evening. It looks a bit scary as that might be further visibility shenanigans like I was facing with other browsers. Hopefully it's just a timing issue which I should be able to fix

[PragTob]

It's interesting... because we see the groups for one of the 2 examples but then don't see the source files... which is very strange.

    And I should see the groups:
      | name      | coverage | files |
      | All Files | 90.48%   | 4     |

    And I should see the source files:
      | name                                    | coverage |
      | lib/faked_project.rb                    | 100.00 %  |
     # ...

[PragTob]

In the mean time this seems to have been resolved.. mostly

[PragTob]

Some of the tests are still semi randomly failing like this one:

expected to find visible css ".t-file" 4 times but there were no matches. Also found "lib/faked_project/some_class.rb 80.00 % 30 15 12 3 1.73", "lib/faked_project/framework_specific.rb 87.50 % 20 8 7 1 1.50", "lib/faked_project.rb 100.00 % 13 6 6 0 4.00", "lib/faked_project/meta_magic.rb 100.00 % 26 13 13 0 2.00", which matched the selector but not all filters.  (Capybara::ExpectationNotMet)
./features/step_definitions/html_steps.rb:25:in `/^I should see the source files:$/'
features/merging_test_unit_and_rspec.feature:42:in `I should see the source files:'

Failing Scenarios:
cucumber features/merging_test_unit_and_rspec.feature:11 # Scenario: 

76 scenarios (1 failed, 7 skipped, 68 passed)
478 steps (1 failed, 40 skipped, 437 passed)
10m1.127s
Error: Process completed with exit code 1.
0s
0s

[PragTob]

#939 begrudgingly itnroduces a retry to ward this off.

[tycooon]

At least this makes it possible to submit a PR, thanks!